Automatic driving track prediction method based on vision-language model, near-end strategy optimization and diffusion model

Abstract

The invention relates to the technical field of automatic driving, and discloses an automatic driving track prediction method based on a vision-language model, near-end strategy optimization and a diffusion model. VLM-DM is proposed, which is the first framework for guiding initial noise priori generation in trajectory prediction based on a diffusion model by using a vision-language model. In order to close up modal gaps, a vision-language model auxiliary prior generator is introduced, and the vision-language model auxiliary prior generator is a module trained through reinforcement learning and can generate noise vectors with semantic meaning under scalar feedback guidance provided by a vision-language model. In order to overcome the limitation of a general vision-language model in application in a specific field, a retrieval enhanced vision-language model is designed, and the model continuously accumulates field knowledge related to a track by retrieving a historical scene and continuously optimizes the evaluation ability through joint training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving trajectory prediction method based on a vision-language model, a proximal policy optimization and a diffusion model. BACKGROUND

[0002] With the rapid development of automatic driving technology, accurate and safe prediction of the future behavior of surrounding traffic participants has become a core task to achieve safe and efficient automatic driving. Traditional trajectory prediction methods are mostly based on deterministic models or imitation learning, which directly map the current state to the future trajectory by learning expert driving data. Although such methods can capture driving patterns, they are difficult to effectively handle the multi-modal nature and uncertainty in the traffic environment, and are prone to produce "averaged" or unrealistic prediction results. In recent years, generative models represented by diffusion models have shown great potential in the field of trajectory prediction. Diffusion models can gradually generate diverse and high-fidelity future trajectories from noise by simulating an "add noise-de-noise" iterative process, which naturally meets the uncertainty requirements of the prediction task and provides a powerful paradigm for generating realistic and diverse trajectories.

[0003] However, the performance of diffusion models is highly dependent on the quality of the initial noise prior. A high-quality initial noise can effectively guide the de-noising process and generate reasonable trajectories that conform to semantic and scene constraints. Currently, vision-language models provide high-level decision support for autonomous driving systems due to their strong semantic understanding and reasoning capabilities, and can generate interpretable semantic instructions such as "slow down and wait" and "prepare to turn right". Therefore, combining the high-level semantics of vision-language models with the bottom-up generation capabilities of diffusion models is an important research direction to improve prediction quality.

[0004] However, existing methods face two major challenges (roughly as shown in the core challenge diagram Figure 1 ) when integrating vision-language models and diffusion models. First, there is a modal mismatch problem: the output of the vision-language model is discrete text or high-dimensional semantic vectors, while the diffusion model requires continuous, structured low-dimensional noise tensors. This gap between "high-level semantics" and "low-level noise" makes it difficult for the vision-language model to directly and finely control the generation process of the diffusion model. Existing methods either indirectly relate it through an attention mechanism by taking its output as a text condition, or use it as a filter to sort random noise, neither of which allows the vision-language model to directly guide the generation of initial noise. Second, there is a lack of domain-specific knowledge: general vision-language models are pre-trained on large-scale general data and lack a deep understanding of domain-specific knowledge such as subtle traffic rules and complex driving strategies in the autonomous driving field, so they cannot effectively evaluate or optimize the quality of generated trajectories, limiting their role in training and supervision.

[0005] Therefore, in order to bridge the gap between high-level semantic reasoning and low-level generation control, make full use of the semantic guidance ability of the visual-linguistic model to generate high-quality initial noise prior, the present application proposes a visual-linguistic model guided diffusion model architecture VLM-DM, aiming to solve the above challenges, through innovative reinforcement learning mechanism and retrieval enhancement technology, let the visual-linguistic model not only "think", but also directly "guide" the noise generation process of the diffusion model, so as to generate safer, more compliant, and more context-aware future trajectories. SUMMARY

[0006] In order to solve the above problems, in the present application, a VLM-DM, a diffusion model guided by a visual-linguistic model (VLM), is proposed for vehicle trajectory prediction.

[0007] The technical solution of the present application is as follows: an automatic driving trajectory prediction method based on a visual-linguistic model, a proximal policy optimization, and a diffusion model, the visual-linguistic model guides the diffusion model, and the thinking and understanding of the retrieval enhanced visual-linguistic model on the vehicle driving environment are converted into guidance on the initial noise prior of the diffusion model through the proximal policy optimization algorithm to generate a future trajectory.

[0008] The steps include the following:

[0009] Step 1: generate a semantic reward signal using a retrieval enhanced visual-linguistic model;

[0010] Step 2: design a visual-linguistic model assisted prior generator to generate an initial noise prior guided by the visual-linguistic model;

[0011] Step 3: input the obtained noise into a pre-trained and frozen diffusion model, and generate a final future trajectory by gradually denoising the diffusion model ; input the future trajectory and the original scene into the retrieval enhanced visual-linguistic model module to calculate a scalar feedback reward as the supervision signal of the reinforcement learning based on the proximal policy optimization; the noise predicted by the diffusion model, the current driving scene information, are the knowledge base of RAG.

[0012] Step 4: realize the co-evolution of the visual-linguistic model assisted prior generator and the retrieval enhanced visual-linguistic model through joint training;

[0013] Step 5: In the inference stage, the trained visual-linguistic model assists the prior generator to generate optimized initial noise; for a new driving scene, an initial noise is first sampled from a standard Gaussian distribution , then an action is generated by the trained visual-linguistic model assisted prior generator , and the optimized noise is calculated , and finally is input into the diffusion model to generate a high-quality final trajectory.

[0014] The step 1 is specifically: the retrieval enhanced visual-linguistic model fuses the front view image and the bird's eye view of the superimposed predicted trajectory for visual perception through the frozen BLIP-2 encoder, retrieves the top two historical entries with the highest similarity to the current scene from a dynamically updated knowledge base to obtain context support through the IndexFlatL2 technology, and the visual-linguistic model performs semantic reasoning on the trajectory generated by the diffusion model according to the prompt word to output a semantic reward signal obtained by weighted summation of safety, rule, and preference rewards .

[0015] The step 2 is specifically: the prior generator models the initial noise optimization process as a Markov decision process, with the state composed of the driving scene and the initial noise, and the action as a continuous disturbance to the noise; a reinforcement learning framework based on proximal policy optimization is adopted, and the policy network learns a policy that maps the state to the action , and the value network estimates the state value to stabilize the training, with the goal of learning a policy that can generate optimized noise to obtain higher visual-linguistic model semantic rewards ;

[0016] The step 3 is specifically: in each training round, noise optimization, trajectory generation, and reward calculation are performed in a closed loop; based on the collected reward signals, the parameters of the policy network and the value network are updated according to the reinforcement learning based on proximal policy optimization; after each training round, the new scene-semantic reward signal pair is added to the knowledge base of the retrieval enhanced visual-linguistic model, allowing its evaluation ability to continuously accumulate domain knowledge.

[0017] , represents the penalty for collision risk, represents the evaluation of traffic rule compliance, represents the encouragement of desired driving style, are the coefficients of each reward.

[0018] The loss function of the core objective function of the reinforcement learning based on proximal policy optimization with KL divergence constraint added:

[0019]

[0020] where, denote the policy network parameters, denote the value network parameters, is the clipped policy objective function, is the value loss function, is the policy KL divergence loss function, denote the policy entropy at state at time step are coefficients for each loss function, which are as follows:

[0021] the clipped objective function

[0022]

[0023] where, is the empirical average of the expectation over time steps is the probability ratio of the new and old policy, is the probability of taking action in state computed by the current policy network, is the probability of taking action in state computed by the old policy network, is the advantage estimate, is the clipping function that limits the value of the ratio between and is the clipping range hyperparameter; the value loss function

[0024]

[0025]

[0026] where, is the value predicted by the value network for state

[0027] the policy KL divergence loss function

[0028]

[0029] where, is the KL divergence computation function, is a Gaussian distribution with zero mean and unit covariance matrix.

[0030] ​​​​​The beneficial effects of the present application: through the visual-linguistic model assisted prior generator of innovative design, the semantic reward signal (covering safety, rule compliance and driving style preference) output by the retrieval enhanced visual-linguistic model is converted into continuous disturbance to the initial noise, effectively bridging the modal gap between high-level semantic reasoning and low-dimensional noise generation, so that the trajectories generated by the diffusion model are not only diverse, but also highly consistent with the real traffic context.

[0031] The VLM-DM performs excellently in safety indicators such as collision avoidance, maintaining safe distance, and complying with traffic rules through a multi-dimensional weighted reward mechanism, while surpassing the existing strongest baseline in terms of host vehicle progress and comprehensive driving quality, proving its ability to generate more efficient and smooth driving behavior under the premise of ensuring safety, overcoming the "overly cautious" problem caused by traditional conservative strategies.

[0032] The retrieval enhanced visual-linguistic model is introduced as an evaluator, which continuously accumulates domain-specific experience through a dynamically updated knowledge base, and forms a closed-loop feedback with the visual-linguistic model assisted prior generator in joint training. This mechanism enables the system to have continuous optimization capability - the VLM evaluation is more accurate, the generator strategy is more optimal, and the two are decoupled but co-evolve, significantly improving the adaptability of the model to complex driving scenarios.

[0033] The present application successfully transfers the capabilities of general visual-linguistic large models to the professional task of automatic driving trajectory prediction, and through reinforcement learning and retrieval enhancement mechanism, it not only can "understand" the scene, but also can "guide" the generation, providing an expandable new paradigm for semantic-driven decision-making and planning in future high-level autonomous driving systems. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The existing methods to be addressed by the present application have problems.

[0035] Figure 2 The example flowchart of the present application.

[0036] Figure 3 The iterative update schematic diagram of the retrieval enhanced visual-linguistic model of the present application. DETAILED DESCRIPTION

[0037] As Figure 2 shown, it can be specifically divided into the following 5 core steps.

[0038] Step 1: A retrieval-augmented visual-language model is employed to generate a semantic evaluation signal. The retrieval-augmented visual-language model first fuses the front-view image with the bird's-eye view overlaid with the predicted trajectory through a frozen BLIP-2 encoder for visual perception, then retrieves similar historical entries from a dynamically updated knowledge base for contextual support, and finally performs semantic reasoning using a powerful multi-modal large model based on carefully designed natural language prompts to output a composite scalar reward signal that is a weighted sum of safety, rule, and preference rewards .

[0039] Step 2: To generate an initial noise prior guided by the visual-language model, a visual-language model-aided prior generator is designed. The initial noise optimization process is modeled as a Markov decision process, where the state consists of the driving scene and the initial noise, and the action is a continuous perturbation to the noise. A reinforcement learning framework based on proximal policy optimization is employed to learn a policy that maps states to actions (noise perturbations) using a policy network , and a value network is used to estimate state values to stabilize training, aiming to learn a policy that generates optimized noise that achieves higher visual-language model rewards .

[0040] Step 3: The noise optimized by the visual-language model-aided prior generator is input into a pre-trained and frozen diffusion model, which generates the final future trajectory through a step-by-step denoising process . Then, the generated trajectory is fed into the retrieval-augmented visual-language model along with the original scene to calculate a scalar feedback reward that quantifies the quality of the trajectory and serves as a supervision signal for reinforcement learning.

[0041] Step 4: The visual-language model-aided prior generator and the retrieval-augmented visual-language model are co-evolved through a joint training procedure. In each training round, the system performs noise optimization, trajectory generation, and reward calculation in a closed loop. Using proximal policy optimization, the parameters of the policy network and the value network are updated based on the collected reward signals. At the end of each round, the new scene-evaluation pair is added to the knowledge base of the retrieval-augmented visual-language model, allowing it to accumulate domain knowledge and continuously improve its evaluation capabilities, forming a decoupled but tightly coordinated and co-evolving system.

[0042] Step 5: In the inference phase, the trained visual-language model-aided prior generator is used to generate optimized initial noise. Specifically, for a new driving scene, we first sample an initial noise from a standard Gaussian distribution Then, a prior generator is assisted by a trained vision-language model to generate an action (noise perturbation), and the optimized noise is calculated. Finally Input the diffusion model to generate a high-quality final trajectory.

[0043] Before detailing the implementation examples of this invention, we will first introduce the objectives and symbol definitions of this invention. The symbols designed in this invention are shown in Table 1.

[0044] In most cases, trajectory prediction tasks can be defined as a given time length and information about the vehicle's surrounding environment Predicting the future of this vehicle Trajectory of time length Since autonomous driving trajectory prediction is typically in two-dimensional form, therefore... Position coordinates at time It is usually represented as an ordered pair , which corresponds to the position coordinates on the bird's-eye view.

[0045] The invention will be further described below with reference to the accompanying drawings and specific implementation examples. A method for predicting the trajectory of autonomous driving based on a vision-language model, proximal policy optimization, and diffusion model is described in the following steps:

[0046] Step 1: Build a retrieval-enhanced visual language model to accumulate domain knowledge and generate semantic evaluation.

[0047] To overcome the limitations of general visual language models in the field of autonomous driving due to their lack of specific knowledge, this paper designs a retrieval-enhanced visual language model as the system's "intelligent referee." This module is responsible for comprehensively evaluating the safety, compliance, and driving style of the generated trajectory and outputting a scalar reward signal. The retrieval-enhanced visual-language model consists of three collaborative components: visual perception, knowledge retrieval, and semantic reasoning.

[0048] First, in the visual perception phase, the system fuses multimodal information of the current driving scene into a composite input, including a forward-looking RGB image and a bird's-eye view overlaid with the trajectory to be evaluated. These inputs are processed by a frozen BLIP-2 encoder to generate a semantically rich unified embedding vector that encodes the spatial layout of the scene and the behavioral intent.

[0049] Secondly, during the knowledge retrieval phase, the system maintains a dynamically updated vector knowledge base. This knowledge base stores scene embeddings accumulated during historical training and their corresponding visual-language model evaluation results. For the scene to be evaluated, the system calculates its fused embedding and retrieves the most similar historical entry from the knowledge base, providing contextual support for the current decision.

[0050] Finally, in the semantic reasoning stage, a powerful multimodal large model serves as the reasoning core, receiving the fused embedding of the current scene, the retrieved historical context, and a carefully designed natural language cue, and outputs a comprehensive semantic reward signal through semantic reasoning . This signal is the weighted sum of multiple sub-rewards, and the calculation formula is as follows:

[0051]

[0052] Among them, penalize collision risk, evaluate traffic rule compliance, encourage desired driving style. This reward signal will serve as a supervisory signal for the subsequent reinforcement learning module.

[0053] Step 2: Design a visual-linguistic model assisted prior generator to generate semantic guided initial noise.

[0054] To solve the modal mismatch problem between the abstract semantic output of the visual-linguistic model and the low-dimensional noise input of the diffusion model, a reinforcement learning module, the visual-linguistic model assisted prior generator, is constructed. The goal of this module is to learn a strategy that converts the semantic reward of the retrieval enhanced visual-linguistic model into specific optimization of the initial noise.

[0055] The visual-linguistic model assisted prior generator models the process of optimizing the initial noise as a Markov decision process. The state is composed of the current driving scene and the initial noise sampled from the standard Gaussian distribution ; the action is a continuous disturbance applied to the noise vector . The strategy is implemented by a policy network that maps the state to a multidimensional Gaussian distribution in the action space:

[0056]

[0057] An action is sampled from this distribution and used to update the noise, generating the optimized noise:

[0058]

[0059] At the same time, a value network learns a value function to estimate the expected reward that can be obtained in state to stabilize policy gradient learning. The training goal of this module is to learn the optimal policy so that The diffusion model generation can guide the generation of trajectories with higher rewards.

[0060] Step 3: Final trajectory generation by diffusion model and reward computation.

[0061] This step connects the vision-language model assisted prior generator and the retrieval-augmented vision-language model, forming a closed loop.

[0062] In the trajectory generation phase, the optimized noise generated in step 2 and the current scene context are input into a pre-trained and frozen diffusion model . The diffusion model generates the final future vehicle trajectory through its step-by-step denoising mechanism:

[0063]

[0064] In the reward computation phase, the generated trajectory and the original scene are input into the retrieval-augmented vision-language module in step 1. The retrieval-augmented vision-language model performs its complete evaluation process (perception, retrieval, reasoning) and outputs a scalar reward signal:

[0065]

[0066] This reward signal quantifies the quality of the trajectory in terms of safety, compliance, and driving intent, and serves as the immediate reward for reinforcement learning, updating the policy of the vision-language model assisted prior generator.

[0067] Step 4: Joint training using proximal policy optimization for closed-loop learning.

[0068] Through the joint training process, the vision-language model assisted prior generator and the retrieval-augmented vision-language model are tightly coupled to form a co-evolutionary closed-loop system.

[0069] In each training round, the system performs the processes of steps 2 and 3, collecting state, action, and reward triplets . Subsequently, the proximal policy optimization algorithm is used to update the parameters of the policy network and the value network. Its core objective function is a clipped proxy objective function:

[0070]

[0071] where is the probability ratio of the new and old policies, ​is the advantage estimate. Meanwhile, the total loss function of the system also includes the value function loss and the policy entropy regularizer:

[0072]

[0073] Most importantly, at the end of each episode, the generated scene embedding and its corresponding visual-linguistic model evaluation results are added to the knowledge base of the retrieval-augmented visual-linguistic model . This allows the retrieval-augmented visual-linguistic model to accumulate domain knowledge continuously as training proceeds. Through this mechanism, the two modules are decoupled but tightly communicate through scalar rewards, achieving co-evolution: the visual-linguistic model helps the prior generator learn to generate better noise, while the retrieval-augmented visual-linguistic model becomes more accurate.

[0074] Step 5: Trajectory prediction with optimized prior.

[0075] In the model deployment phase, for a new driving scene, the following inference process is performed:

[0076] Sample initial noise: sample an initial noise from a standard Gaussian distribution . .

[0077] Generate optimized noise: input and the current scene to the trained visual-linguistic model assisted prior generator (policy network). The network directly outputs an action (noise disturbance) and calculates the optimized noise .

[0078] Generate final trajectory: input the optimized noise and the scene to the frozen diffusion model to generate the final, high-quality future trajectory .

[0079] Table 1 Symbol explanation

[0080]

[0081] The method of the present application is verified using the public dataset NAVSIM dataset from the field of autonomous driving. This dataset is a challenging, planning-oriented benchmark built from real driving data. NAVSIM is built based on the OpenScene dataset, which is a reduced version of the large-scale nuPlan benchmark. The dataset provides rich sensor modalities, including surround-view RGB camera images (1920x1080 resolution) and fused LiDAR point clouds from five sensors. NAVSIM focuses on evaluating planning strategies within a 4-second prediction time horizon, using a non-reactive simulation environment - i.e., the surrounding traffic does not react to the ego vehicle - enabling efficient and consistent evaluation of different methods.

[0082] In the present application, we use the NAVSIM dataset provided by to train the model VLM-DM proposed in the present application, and use to test the performance of the model. The experimental equipment used is shown in Table 2.

[0083] Table 2 Hardware and software environment of the experiment of the present application

[0084]

[0085] (1) In the present application, the comparison results of VLM-DM and baseline methods on the NAVSIM dataset are shown in Table 3. The complete model of VLM-DM is superior to all baseline methods on the dataset used, and the experimental results show the effectiveness of VLM-DM for the vehicle trajectory prediction task, and also show that the method used in this paper combines the high-level semantic guidance of the visual-linguistic model with the bottom-level generation process of the diffusion model through an innovative closed-loop feedback mechanism, thereby significantly improving the prediction performance.

[0086] (2) Among the two types of baseline used, the hybrid model that incorporates a planning module (such as ) is superior to the pure imitation learning model (such as Transfuser, LTF, DRAMA) in most cases. It is worth noting that the model performs outstandingly in the ego progress (EP) indicator, which shows that the fine-tuning specifically for this target is effective. However, VLM-DM not only surpasses it in ego progress (EP), but also achieves overall leadership in the driving quality comprehensive score (PDMS). We believe that one possible reason is that VLM-DM can balance safety, rule compliance, and driving efficiency through the retrieval of enhanced visual-linguistic model semantic reward mechanisms, thereby generating trajectories that are both safe and efficient, avoiding the low driving efficiency caused by a purely conservative strategy, which reflects the superiority of combining high-level semantic reasoning with generative models.

[0087] (3)The present application also considers a variant model of VLM-DM, and we can find that the variant model is not as good as the complete model in overall performance. The experimental results show that the retrieval enhanced visual-language model module can provide more accurate guidance for noise optimization by continuously accumulating domain expertise. First, the semantic feedback generated by the visual-language model (such as safety and regularity) determines the direction of noise optimization. Second, the retrieval enhancement mechanism provides important historical context information for the evaluation process, which has a potential positive impact on the quality of the generated trajectory. Ignoring this impact (i.e., not using the retrieval enhanced visual-language model) may not achieve further performance improvement. In addition, although the "pure intrinsic reward + visual-language model reward" variant achieves better performance compared to the baseline of pure intrinsic reward, it is still significantly worse than the complete VLM-DM, which confirms the key role of the retrieval enhanced knowledge base in the co-evolution of the system and the improvement of performance.

[0088] Table 3 Experimental results of the model VLM-DM proposed by the present application and the comparative method

[0089]

Claims

1. An autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model, characterized in that, The visual-language model guides the diffusion model. Through the proximal policy optimization algorithm, the retrieval-enhanced visual-language model's thinking and understanding of the vehicle driving environment is transformed into guidance for the initial noise prior of the diffusion model, generating the future trajectory.

2. The autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model according to claim 1, characterized in that, The steps include the following: Step 1: Generate semantic reward signals using a retrieval-enhanced visual-language model; Step 2: Design a vision-language model-assisted prior generator to generate initial noisy priors guided by the vision-language model; Step 3: The obtained noise Inputting a pre-trained and frozen diffusion model, the diffusion model is used to progressively denoise and generate the final future trajectory. ; the trajectory of the future The original scene data is fed into the retrieval-enhanced visual-language model module to calculate a scalar feedback reward. , serving as a supervisory signal for reinforcement learning based on proximal policy optimization; For noise Trajectory predicted by diffusion model Current driving scenario information For RAG's knowledge base; Step 4: Achieve co-evolution of the visual-language model-assisted prior generator and the retrieval-enhanced visual-language model through joint training; Step 5: During the inference phase, the trained visual-language model assists the prior generator in generating optimized initial noise; for a new driving scenario, initial noise is first sampled from a standard Gaussian distribution. Then, a prior generator is used to generate an action using a pre-trained vision-language model. And calculate the optimized noise. Finally Input the diffusion model to generate a high-quality final trajectory.

3. The autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model according to claim 2, characterized in that, Step 1 specifically involves: the retrieval-enhanced visual-language model performing visual perception by fusing the foreground image with a bird's-eye view overlaid with the predicted trajectory using a frozen BLIP-2 encoder; retrieving the two most similar historical entries to the current scene from a dynamically updated knowledge base using IndexFlatL2 technology to obtain contextual support; and performing semantic reasoning on the trajectory predicted by the diffusion model based on cue words, outputting a semantic reward signal obtained by weighted summation of safety, rule, and preference rewards. .

4. The autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model according to claim 2, characterized in that, Step 2 specifically involves: the prior generator modeling the initial noise optimization process as a Markov decision process, where the state consists of the driving scenario and the initial noise, and the action is a continuous perturbation of the noise; a reinforcement learning framework based on proximal policy optimization is adopted, and the policy network learns a policy that maps the state to actions. The system estimates state values ​​through a value network to stabilize training, with the goal of learning an optimized noise that generates higher semantic rewards for the visual-language model. The strategy.

5. The autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model according to claim 2, characterized in that, Step 3 specifically involves: in each training round, performing noise optimization, trajectory generation, and reward calculation in a closed loop; updating the parameters of the policy network and value network based on the collected reward signals according to reinforcement learning based on proximal policy optimization; and after each training round, adding new scene-semantic reward signal pairs to the knowledge base of the retrieval-enhanced visual-language model so that its evaluation capabilities can continuously accumulate domain knowledge.

6. The autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model according to claim 2, characterized in that, , This indicates a penalty for collision risk. This indicates an assessment of traffic rule compliance. This indicates an encouraged and desired driving style. This represents the coefficient for each reward.

7. The autonomous driving trajectory prediction method based on a vision-language model, proximal policy optimization, and diffusion model according to claim 2, characterized in that, The core objective function of reinforcement learning based on proximal policy optimization, and the loss function after adding KL divergence constraints: in, Representative policy network parameters, Representative value network parameters, It is the objective function of the pruned strategy. It is a value loss function. It is the KL divergence loss function of the policy. Representative strategy In state Entropy of time, Here are the coefficients of each loss function, which are as follows: Pruned objective function : in, It is a time step The expected average experience, It is the probability ratio of the new strategy to the old strategy. The state calculated by the current policy network Make an action The probability, The state calculated for the old policy network Make an action The probability, It is an advantage estimate. For the clipping function, the ratio The value is limited to between, It is the clipping range hyperparameter; Value loss function : in, For value network state The value of prediction; KL divergence loss function of the strategy : in, It is the KL divergence calculation function. It is a Gaussian distribution with zero mean and unit covariance matrix.

Images