Constraint perception gradient projection-based world model and reinforcement learning collaborative optimization method

Abstract

The invention relates to the field of artificial intelligence control, in particular to a world model and reinforcement learning collaborative optimization method based on constraint perception gradient projection, which comprises the following steps: constructing a world model based on Transform architecture, performing unified modeling on an environment state, system dynamics and a reward function, and training the model based on reference model data to predict an environment future state; training a reinforcement learning strategy based on a virtual track and a reward signal generated by the world model, and calculating strategy gradient information; an explicit gradient projection operator of constraint perception is designed, a strategy gradient is corrected in real time according to future constraint conditions predicted by the world model, and it is ensured that the gradient updating direction meets the safety constraint and the optimization target at the same time; and synchronously updating world model parameters and reinforcement learning strategy parameters by using the gradient after projection correction to realize collaborative optimization of the world model parameters and the reinforcement learning strategy parameters. According to the method, the key problems of local optimal trap, low convergence speed, insufficient stability and the like in the collaborative optimization process of a traditional method are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence control technology, and in particular to a world model based on constraint-aware gradient projection and a reinforcement learning-based collaborative optimization method. Background Technology

[0002] With the rapid development of intelligent connected and autonomous driving technologies, energy-efficient driving has become a crucial optimization goal for autonomous vehicles. Existing energy-saving control methods are typically based on optimization models or reinforcement learning frameworks, learning the energy consumption characteristics of vehicles under different operating conditions to achieve the most energy-efficient driving strategy. However, these methods face numerous challenges in complex real-world scenarios: 1. Local Optimality Trap Problem: When using reinforcement learning methods to update policies in a high-dimensional continuous action space, it is easy to get trapped in local optima, resulting in unsatisfactory global performance.

[0003] 2. Unstable convergence: In gradient updates during joint training, inconsistent gradient directions can lead to slow convergence, vanishing gradients, or oscillating divergence.

[0004] Therefore, there is an urgent need for a technical method that can explicitly and collaboratively optimize model parameters and reinforcement learning strategies while maintaining consistency between optimization objectives and safety constraints, so as to improve the stability and global optimality of energy-saving control. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a constrained gradient projection-based co-optimization method using a world model and reinforcement learning. This invention is applicable to energy-saving driving control scenarios for autonomous vehicles. By introducing an explicit gradient projection operator into the joint optimization process of the world model and reinforcement learning, this method achieves constraint-consistent optimization of model parameters and control strategies, significantly improving the convergence speed and stability of the optimization.

[0006] The technical means employed in this invention are as follows: A constraint-aware gradient projection-based world model and reinforcement learning co-optimization method includes: collecting multimodal input data containing bird's-eye view BEV information; constructing a world model based on the Transformer architecture to uniformly model the environment state, system dynamics, and reward function, and training the model based on benchmark model data to predict the future state of the environment; training a reinforcement learning policy based on the virtual trajectory and reward signal generated by the world model, and calculating the policy gradient information; designing a constraint-aware explicit gradient projection operator to correct the policy gradient in real time according to the future constraints predicted by the world model, ensuring that the gradient update direction simultaneously satisfies safety constraints and optimization objectives; and synchronously updating the world model parameters and reinforcement learning policy parameters using the projection-corrected gradient to achieve co-optimization of the two.

[0007] Furthermore, the multimodal input data includes environmental conditions, vehicle location, speed, and energy consumption information; a system dynamic model is established based on historical data and the current state.

[0008] Furthermore, the world model is constructed based on a Transformer encoder and decoder. The encoder processes the current state information, and the decoder predicts the state evolution sequence for the next T steps. Model parameters are updated in real time to adapt to environmental changes. A joint loss function is designed, comprising state prediction loss, reward prediction loss, and constraint violation penalty. The world model is pre-trained using an offline dataset collected from a benchmark model to establish a dynamic representation of the environment. Cumulative gradient information is recorded during the world model training process. Fit the recorded gradient information into the gradient space to construct the entire gradient manifold. .

[0009] Furthermore, the reinforcement learning strategy utilizes a trained world model to generate virtual interaction trajectories, providing a large amount of simulation data for reinforcement learning; a multi-objective reward function is designed, incorporating energy-saving goals, safety constraints, and comfort requirements; and cumulative reward gradient information is recorded during policy gradient algorithm updates. Fit the recorded gradient information into the gradient space to obtain the complete manifold gradient. .

[0010] Furthermore, the explicit gradient projection operator is designed. Specifically, this includes: utilizing the manifold gradient projection operator The gradient directions of subsequent updates from the world model and reinforcement learning are respectively directed towards the fitted gradient manifold. Directional projection; The projection operation ensures that the gradient update does not violate the system constraints; the gradient direction is fine-tuned to avoid getting trapped in local optima.

[0011] Furthermore, the explicit gradient projection operator is corrected using the manifold gradient projection method to find a vector within the tangent space of the manifold. This makes it similar to the original gradient. In Riemannian metric The distance below is the minimum:

[0012] in, It represents the gradient manifold, which represents the geometric structure of the parameter evolution gradient in space; This represents the projection operator, which projects any vector onto the tangent space of a manifold; This represents the projection result, i.e., the vector that is closest to the original gradient in the tangent space of the manifold; This represents the original gradient at the current step, and the update direction before projection; This indicates that the manifold is at the current parameter point. The tangent space at a given point is locally linearly approximated. This indicates that the manifold is at the current parameter point. The tangent space at a given point is locally linearly approximated. Furthermore, the step of synchronously updating the world model parameters and reinforcement learning policy parameters using the projection-corrected gradient specifically includes: calculating the parameter update amount based on the corrected gradient; adjusting the update step size using an adaptive learning rate; updating the model parameters or control policy; and verifying the performance and stability of the updated model.

[0013] Furthermore, the convergence conditions for the collaborative optimization include: setting convergence conditions, including a gradient magnitude threshold and a maximum number of iterations; checking whether the convergence conditions are met after each iteration; if convergence is not achieved, returning to the training reinforcement learning policy stage to continue iteration; if convergence is achieved, outputting the optimal result.

[0014] Compared with the prior art, the present invention has the following advantages: This invention significantly accelerates the convergence speed of the optimization process by introducing an explicit gradient projection mechanism, effectively avoiding getting trapped in local optima; it ensures the stability and feasibility of parameter updates through manifold constraint projection, avoiding gradient vanishing and oscillating divergence problems; and it achieves synergistic optimization of the world model and reinforcement learning strategy, ensuring that the update process always meets safety constraints and optimization objectives. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the world model and reinforcement learning co-optimization method based on constraint-aware gradient projection in this invention.

[0017] Figure 2 This is a schematic diagram of the system architecture in this invention.

[0018] Figure 3 This is a schematic diagram of the explicit gradient update rule in this invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0022] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0023] like Figure 1 As shown, this invention provides a method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning, including: collecting multimodal input data containing bird's-eye view BEV information; in a preferred embodiment of this invention, the multimodal input data includes environmental state, vehicle position, speed and energy consumption information; and establishing a system dynamic model based on historical data and the current state.

[0024] A world model based on the Transformer architecture is constructed to uniformly model the environmental state, system dynamics, and reward function. The model is trained based on benchmark model data to predict the future state of the environment. In a preferred embodiment of the present invention, the world model is constructed based on a Transformer encoder and decoder. The encoder is used to process the current state information, and the decoder is used to predict the state evolution sequence in the next T steps. The model parameters are updated in real time to adapt to environmental changes. Design a joint loss function, which includes state prediction loss, reward prediction loss, and constraint violation penalty term; The world model is pre-trained using an offline dataset collected from a benchmark model to establish a dynamic representation of the environment; the cumulative gradient information during the world model training process is recorded. Fit the recorded gradient information into the gradient space to construct the entire gradient manifold. .

[0025] Based on the virtual trajectory and reward signal generated by the world model, a reinforcement learning policy is trained, and policy gradient information is calculated. In a preferred embodiment of this invention, the reinforcement learning policy uses the trained world model to generate virtual interactive trajectories, providing a large amount of simulation data for reinforcement learning. A multi-objective reward function is designed, incorporating energy-saving goals, safety constraints, and comfort requirements. Cumulative reward gradient information is recorded during policy gradient algorithm updates. Fit the recorded gradient information into the gradient space to obtain the complete manifold gradient. .

[0026] Design a constraint-aware explicit gradient projection operator to correct the policy gradient in real time based on the future constraints predicted by the world model, ensuring that the gradient update direction simultaneously satisfies safety constraints and optimization objectives; in a preferred embodiment of this invention, design the explicit gradient projection operator. Specifically, this includes: utilizing the manifold gradient projection operator The gradient directions of subsequent updates from the world model and reinforcement learning are respectively directed towards the fitted gradient manifold. Directional projection; ensuring gradient updates do not violate system constraints through projection operations; fine-tuning gradient directions to avoid getting trapped in local optima.

[0027] In a specific implementation, as a preferred embodiment of the present invention, the explicit gradient projection operator is corrected using the manifold gradient projection method to find a vector within the tangent space of the manifold. This makes it similar to the original gradient. In Riemannian metric The distance below is the minimum:

[0028] in, It represents the gradient manifold, which represents the geometric structure of the parameter evolution gradient in space and can be obtained by fitting the sample gradient; This represents the projection operator, which projects any vector onto the tangent space of a manifold; This represents the projection result, i.e., the vector that is closest to the original gradient in the tangent space of the manifold; This represents the original gradient at the current step, and the update direction before projection; This indicates that the manifold is at the current parameter point. The tangent space at a given point is locally linearly approximated. This indicates that the manifold is at the current parameter point. The tangent space at a given point is locally linearly approximated.

[0029] The world model parameters and reinforcement learning policy parameters are updated synchronously using the projection-corrected gradient to achieve collaborative optimization. In a preferred embodiment of this invention, the parameter update amount is calculated based on the corrected gradient; the update step size is adjusted using an adaptive learning rate; the model parameters or control policy are updated; and the performance and stability of the updated model are verified.

[0030] In a specific implementation, as a preferred embodiment of the present invention, the convergence condition of the collaborative optimization includes: setting convergence conditions, including a gradient magnitude threshold and a maximum number of iterations; checking whether the convergence condition is met after each iteration; if convergence is not achieved, returning to the training reinforcement learning strategy stage to continue iteration; if convergence is achieved, outputting the optimal result.

[0031] The method of this invention avoids local optima through the following technical means: (1) Introduce a random perturbation mechanism during gradient projection to enhance global search capability; (2) Adopt a multi-starting-point parallel optimization strategy to increase the probability of finding the global optimal solution; (3) Using the gradient projection operator While maintaining the gradient update direction, the projection magnitude is limited to ensure that the policy has sufficient exploration space; (4) Combine historical gradient information to make directional corrections to avoid oscillations and divergence.

[0032] The method of this invention is applicable to energy-saving driving scenarios for autonomous vehicles.

[0033] The present invention also includes a corresponding optimization system based on explicit gradient updates. Exemplarily, the system includes: a modeling module configured to construct a prediction model according to task requirements, modeling the system state and environmental information; a gradient calculation module configured to calculate the gradient of the objective function during iterative optimization; an explicit gradient update module configured to correct parameters based on explicit gradient update rules; a projection constraint module configured to apply projection constraints to the updated parameters to ensure that the optimization process remains within the feasible solution space; and an optimization execution module configured to drive the model to run based on the updated parameters, obtain the optimized result, and achieve rapid convergence and stable optimization.

[0034] Example This invention provides a complete flowchart of an optimization method based on explicit gradient updates. This method achieves the globally optimal solution for energy-saving driving control through collaborative optimization of a world model and reinforcement learning. The implementation flow structure is as follows: Figure 1 As shown, the specific implementation steps are as follows: Step 1: At the beginning of the optimization process, a world model based on the Transformer architecture is first constructed. This world model adopts an encoder-decoder structure and receives multimodal input data containing bird's-eye view (BEV) information.

[0035] Data Acquisition and Preprocessing: Environmental perception data of autonomous vehicles is collected in the CARLA autonomous driving simulator, including: road information from the BEV perspective, covering a range of 50 meters around the vehicle; vehicle status information (position, speed, acceleration, energy consumption, etc.).

[0036] World model architecture design: such as Figure 2 As shown, the encoder and decoder adopt the Transformer architecture, and the output layer directly outputs future state predictions, reward predictions, and constraint violation predictions.

[0037] Loss function design: The joint loss function consists of three parts, namely the state prediction error loss, the reward prediction loss, and the constraint violation penalty term loss.

[0038] Pre-training process: Use a baseline model to collect a certain amount of trajectory data and perform offline pre-training on the world model.

[0039] Step 2: Based on the trained world model, train the reinforcement learning policy network. For example... Figure 1 As shown, this step utilizes virtual trajectories generated by a world model, avoiding the costs and risks associated with extensive interactions in the real environment.

[0040] Virtual trajectory generation: Virtual trajectories are generated using a world model, including state sequences, action sequences, reward sequences, and constraint satisfaction.

[0041] Reinforcement learning algorithm: Proximal policy gradient algorithm (PPO) is adopted.

[0042] Multi-objective reward function design: includes rewards for safe driving and rewards for energy consumption.

[0043] Gradient information recording: During training, the gradient information of the policy network is recorded after each parameter update and stored in the gradient history buffer.

[0044] Step 3: As Figure 1 As shown, after completing the initial training, it is necessary to perform manifold fitting on the accumulated gradient information, which is a key step in realizing constraint-aware gradient projection.

[0045] Gradient data collection: Extract the gradient sequence of the world model and the gradient history of the policy network from the gradient history buffer.

[0046] Manifold fitting method: Kernel principal component analysis is used to fit the gradient manifold.

[0047] Tangent space calculation: Calculate the tangent space of the manifold at the current parameter point position.

[0048] Step 4: As Figure 1 As shown, the gradient projection operator with constraint awareness is used to correct the subsequent gradient update direction in real time.

[0049] Projection operator design: Defining the manifold gradient projection operator .

[0050] Projection calculation process: For the newly calculated original gradient First, find its tangent space. The gradient is then projected onto the image and then orthogonally projected to obtain the gradient after projection. It maintains the original gradient's optimization direction while also satisfying the manifold's geometric constraints.

[0051] Constraint checking: Check whether the projected gradient violates safety constraints and energy constraints. If a constraint violation exists, an additional penalty is introduced.

[0052] Adaptive adjustment: The projection intensity is adaptively adjusted based on the current optimization progress.

[0053] Step 5: As Figure 1 As shown, the world model parameters and reinforcement learning policy parameters are updated synchronously using the modified gradient.

[0054] Step 6: As Figure 1 As shown, after each iteration, it is checked whether the convergence condition is met.

[0055] The core innovation of this invention lies in the explicit gradient projection mechanism. For example... Figure 3 As shown, this mechanism ensures that the parameter update direction satisfies both the optimization objective and geometric constraints by projecting the original gradient onto the tangent space of the gradient manifold. (Gradient manifold) The geometric structure formed by all effective gradient directions in the parameter space is obtained by fitting the sample gradient; This represents the projection operator, which projects any vector onto the tangent space of a manifold; This represents the projection result, i.e., the vector that is closest to the original gradient in the tangent space of the manifold; This represents the original gradient at the current step, and the update direction before projection; This indicates that the manifold is at the current parameter point. Tangent space at a point, local linear approximation Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning, characterized in that, include: Collect multimodal input data containing BEV information from a bird's-eye view; Construct a world model based on the Transformer architecture to uniformly model the environmental state, system dynamics, and reward function, and train the model based on benchmark model data to predict the future state of the environment; Based on the virtual trajectory and reward signal generated by the world model, a reinforcement learning policy is trained, and policy gradient information is calculated. Design a constraint-aware explicit gradient projection operator to correct the policy gradient in real time based on the future constraints predicted by the world model, ensuring that the gradient update direction simultaneously satisfies safety constraints and optimization objectives. By using the gradient after projection correction to synchronously update the parameters of the world model and the reinforcement learning policy, the co-optimization of the two is achieved.

2. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 1, characterized in that, The multimodal input data includes environmental conditions, vehicle location, speed, and energy consumption information; a dynamic model of the system is established based on historical data and the current state.

3. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 1, characterized in that, The world model is constructed based on a Transformer encoder and decoder. The encoder processes the current state information, and the decoder predicts the state evolution sequence for the next T steps. The model parameters are updated in real time to adapt to environmental changes. Design a joint loss function, which includes state prediction loss, reward prediction loss, and constraint violation penalty term; The world model is pre-trained using an offline dataset collected from a benchmark model to establish a dynamic representation of the environment; the cumulative gradient information during the world model training process is recorded. Fit the recorded gradient information into the gradient space to construct the entire gradient manifold. .

4. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 1, characterized in that, The reinforcement learning strategy utilizes a trained world model to generate virtual interaction trajectories, providing a large amount of simulation data for reinforcement learning; a multi-objective reward function is designed, incorporating energy-saving goals, safety constraints, and comfort requirements; and cumulative reward gradient information is recorded during policy gradient algorithm updates. Fit the recorded gradient information into the gradient space to obtain the complete manifold gradient. .

5. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 1, characterized in that, Design the explicit gradient projection operator. Specifically, it includes: Using manifold gradient projection operator The gradient directions of subsequent updates from the world model and reinforcement learning are respectively directed towards the fitted gradient manifold. Directional projection; The projection operation ensures that the gradient update does not violate the system constraints; the gradient direction is fine-tuned to avoid getting trapped in local optima.

6. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 5, characterized in that, The explicit gradient projection operator is corrected using the manifold gradient projection method to find a vector within the tangent space of the manifold. This makes it similar to the original gradient. In Riemannian metric The distance below is the minimum: in, It represents the gradient manifold, which represents the geometric structure of the parameter evolution gradient in space; This represents the projection operator, which projects any vector onto the tangent space of a manifold; This represents the projection result, i.e., the vector that is closest to the original gradient in the tangent space of the manifold; This represents the original gradient at the current step, and the update direction before projection; This indicates that the manifold is at the current parameter point. The tangent space at a given point is locally linearly approximated. This indicates that the manifold is at the current parameter point. The tangent space at a given point is locally linearly approximated.

7. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 1, characterized in that, The method of synchronously updating the world model parameters and reinforcement learning policy parameters using the projection-corrected gradient specifically includes: Calculate the parameter update amount based on the corrected gradient; adjust the update step size using an adaptive learning rate; update the model parameters or control strategy; verify the performance and stability of the updated model.

8. The method for co-optimization of a world model based on constraint-aware gradient projection and reinforcement learning according to claim 1, characterized in that, The convergence conditions for the collaborative optimization include: Set convergence conditions, including gradient magnitude threshold and maximum number of iterations; check whether the convergence conditions are met after each iteration; if convergence is not achieved, return to the training reinforcement learning policy stage to continue iterating; if convergence is achieved, output the optimal result.

Images