World model-driven automatic driving reinforcement learning double-strategy risk perception control method and system and storage medium

Abstract

The invention discloses a world model-driven automatic driving reinforcement learning double-strategy risk perception control method and system and a storage medium. The method comprises the steps that a world model capable of being used for potential trajectory deduction is built based on a recursive state space model, a risk perception filtering mechanism is designed, and a low-risk and high-efficiency trajectory subset is screened from a generated trajectory set; a long-term strategy and short-term strategy collaborative control structure is constructed, the long-term strategy adopts a soft actor commentator architecture based on a Lagrange method to strengthen security constraints, and the short-term strategy is dynamically generated according to a prediction result of a world model to meet high-timeliness control requirements; control is realized through double-strategy fusion, and closed-loop optimization is realized based on a reward and cost combined driving mechanism in interaction with a real environment. According to the method, collaborative optimization of safety and efficiency in automatic driving is realized, generalization ability and stability performance in a complex dynamic scene are improved, and practicability of high-level automatic driving is promoted.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a world model-driven dual-strategy risk perception control method, system, and storage medium for autonomous driving reinforcement learning. Background Technology

[0002] With the rapid development of autonomous driving technology, autonomous driving systems have shown great potential in improving road traffic safety, reducing traffic accident rates, and improving traffic efficiency. However, balancing driving efficiency and safety in complex dynamic driving environments, especially when facing emergencies or complex traffic scenarios, remains a challenge in the field of autonomous driving.

[0003] Most existing autonomous driving technologies rely on traditional perception, planning, and control modules. These modules function well in simple or structured environments, but in complex, unstructured, and dynamic scenarios, they often fail to adequately account for various potential risks, leading to system instability when dealing with complex situations. To address this issue, increasing research is focusing on incorporating advanced methods such as reinforcement learning and world models to enhance the decision-making capabilities and environmental adaptability of autonomous driving systems.

[0004] Reinforcement learning (RL), a technology for decision-making based on environmental interaction, is gaining increasing attention in autonomous driving. Existing RL methods demonstrate good adaptability in complex scenarios, but their control strategies often lack sufficient safety guarantees. Therefore, RL methods incorporating safety constraints have emerged, enabling the optimization of driving strategies while ensuring vehicle safety. However, single-policy RL control methods often fail to adapt in real-time to complex and dynamic scenarios, resulting in untimely or ineffective control responses. On the other hand, world models, as a model-based predictive control method, can effectively predict future driving trajectories by modeling and predicting the driving environment, providing important references for decision-making. Using world models to predict potential trajectories can not only identify potential risks in advance but also select the optimal path from multiple possible trajectories, thereby improving the decision-making efficiency and safety of autonomous driving systems. Although existing research has begun to explore the combination of world models and reinforcement learning, most methods focus on the application of single policies, failing to fully utilize the synergistic effects of long-term and short-term policies. Furthermore, how to effectively combine safety perception mechanisms, real-time prediction results from world models, and multi-source policy collaboration remains a significant challenge in current research. Therefore, how to design a dual-strategy risk perception control method based on a world model that can both ensure system safety and meet the control requirements of high timeliness remains an urgent problem to be solved in the field of autonomous driving.

[0005] Existing autonomous driving decision-making and control methods include the risk estimation-based reinforcement learning-based autonomous driving safety interpretable decision-making method disclosed in patent document CN118396131A, which optimizes the decision-making process by introducing a risk estimation module and improves the safety and interpretability of the autonomous driving system in complex environments; the automatic manual guidance-based autonomous driving reinforcement learning method disclosed in patent document CN119761223A, which can accelerate the training process and improve model efficiency through manual guidance, but its generalization ability in complex and dynamic traffic environments is still limited; and the human driving data-guided reinforcement learning autonomous driving decision-making method proposed in patent document CN118261233A, which optimizes the design of the reward function and enhances the system's anthropomorphic decision-making ability, but the dependence of this method on human driving data limits its adaptability in unconventional driving scenarios and affects the widespread application of the system. Summary of the Invention

[0006] The purpose of this invention is to provide a world model-driven dual-strategy risk perception control method, system, and storage medium for autonomous driving reinforcement learning. This method aims to solve the problems of insufficient smoothness, low efficiency, and inadequate safety in the strategy switching of autonomous commercial vehicles in various autonomous driving scenarios in the prior art. By introducing a world model and a dual-strategy collaborative technical solution, more efficient and safer autonomous driving control can be achieved.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention includes:

[0008] In a first aspect, this invention discloses a world model-driven reinforcement learning-based dual-strategy risk perception control method for autonomous driving, which includes the following steps:

[0009] Step 1: Collect driver driving data, construct a demonstration dataset of observation-action pairs, and construct a world model of the recursive state-space model RSSM structure based on the dataset through self-supervised learning;

[0010] The world model is used to generate trajectories and predict states in the potential state space, and outputs the state, reward and cost information at future time moments.

[0011] Step 2: Based on the world model built in Step 1, perform multi-trajectory extrapolation on the current state in each control cycle to generate multiple candidate trajectory sequences, and conduct a joint evaluation of reward and cost for each trajectory;

[0012] A risk perception filtering mechanism is adopted to select a set of safe trajectories that meet the preset cost constraints from all candidate trajectories, forming a low-risk trajectories subset for strategy decision-making;

[0013] Step 3: Based on the policy optimization objective, construct a reinforcement learning control structure composed of long-term and short-term policies in synergy;

[0014] The long-term strategy is trained using a soft actor critic architecture based on Lagrange multiplier constraints and is updated using real-world interaction data and simulated data generated by the world model in the latent state space.

[0015] The short-term strategy extracts the expected value of the current action from the trajectory sequence with the maximum reward or minimum cost among the low-risk trajectories selected in step 2, and uses it as the immediate control strategy.

[0016] Step 4: Integrate the output actions of the long-term and short-term strategies in Step 3, dynamically adjust the integration weights based on the current environmental risk level, and generate the final control action;

[0017] The control actions are applied to the vehicle control system to perform interactions with the environment and collect new observation data for subsequent iterative optimization of the world model and policy network, so as to achieve closed-loop updates of the control policy.

[0018] Step 5: Continue to add the real-state-action-reward-cost data collected during the interaction to the training process of the world model and policy network;

[0019] Meanwhile, the imagined data generated by the world model in the potential state space, which includes virtual interaction sequences of state transitions, reward feedback and cost assessment predicted by the model, is used alternately with real data to update network parameters, forming a closed-loop adaptive learning mechanism that combines real and virtual experiences.

[0020] Furthermore, in step 1, the recursive state space model includes a state encoder, a state transition model, an observation decoder, a reward decoder, and a cost decoder.

[0021] The state encoder generates a latent state representation based on sensor observations, and the state transition model predicts the posterior state distribution given historical states and actions.

[0022] The observation decoder reconstructs the observations based on the potential states, and the reward decoder and cost decoder output the reward value and cost value of the trajectory segment, respectively.

[0023] The recursive state space model is trained using a multi-task loss function, which consists of state reconstruction error, reward prediction error, cost prediction error, and potential dynamic consistency error.

[0024] Furthermore, in step 2, the trajectory generation is achieved by performing several forward samplings with a preset step size by the state transition model in the current potential state to obtain several predicted trajectories containing state, action, reward and cost.

[0025] The risk perception filtering mechanism makes a judgment based on the cumulative cost estimate of each trajectory, and determines the trajectories that meet the condition that the cumulative cost does not exceed the preset risk threshold ε as low-risk trajectories, thus forming a set of safe trajectories;

[0026] The set of safe trajectories is further sorted according to predefined evaluation indicators, and the trajectory with the lowest cost or highest reward is selected for subsequent short-term strategy generation and action selection.

[0027] Furthermore, in step 3, the long-term strategy is constructed using a soft actor critic architecture based on Lagrange constraints, which includes a policy network, a value network, and two Q-value networks.

[0028] The policy network generates continuous actions based on the current observed state; two Q-value networks output action values ​​for a given state-action pair, where the action value is characterized by both cumulative reward and expected cost; and the value network estimates the long-term return for a given state.

[0029] By introducing Lagrange multipliers to constrain the expected cost, an optimization function is formed with the objective of maximizing the cumulative reward and limiting the cumulative cost to no more than a preset threshold.

[0030] The policy network parameters are updated using a proximal policy optimization approach, and a hybrid training method combining real interaction data and simulated trajectory data generated by the world model is used to obtain a long-term policy.

[0031] Furthermore, in step 3, the short-term strategy is selected from the set of low-risk trajectories generated by the world model, specifically including:

[0032] Calculate the cumulative reward and cumulative cost for each candidate trajectory, and select the trajectory with the highest cumulative reward or the lowest cost under the cost constraint as the current short-term reference trajectory.

[0033] The mean of the action sequence corresponding to each time step in the reference trajectory is extracted and used as the output action of the short-term strategy in the current state.

[0034] The short-term strategy mentioned above does not participate in the training process, but is only used as a temporary action generation method under the current control cycle;

[0035] A Gaussian distribution-based action refitting mechanism is adopted to fit the reference trajectory action sequence into a conditional Gaussian policy distribution by minimizing the KL divergence, thereby obtaining the action mean and covariance parameters, and performing noisy sampling during the control execution phase.

[0036] The final control action is obtained by weighting the long-term and short-term strategy actions according to a fusion weight α; the fusion weight is dynamically adjusted based on the number of current safe trajectory sets and the risk level.

[0037] The risk level is determined by dividing the candidate trajectories generated by the world model into ranges based on the cumulative cost values ​​output by the cost decoder. Trajectories with a cumulative cost value lower than the set value are defined as low-risk, trajectories with a cumulative cost value higher than the set value are defined as high-risk, and trajectories in the middle range are defined as medium-risk.

[0038] Furthermore, in steps 2 and 3, the calculation methods for rewards and costs are as follows:

[0039] The reward function r used t With cost function c t It consists of a weighted combination of multi-dimensional driving performance indicators, including driving efficiency indicators. Driving safety indicators Driving comfort index and collision avoidance mission completion indicators The four sub-indicators and the reward function are defined as follows:

[0040]

[0041] Where, ρ eff ρ safe ρ comf and ρ avoid These are the hyperparameters that adjust the weights of each sub-index, and satisfy ρ eff +ρ safe +ρ comf +p avoid =1;

[0042] (1) Driving efficiency indicators Based on the vehicle's current speed v t With target speed limit v limit Maximum vehicle speed v max Deviation calculation:

[0043]

[0044] When v t The reward is highest when approaching the speed limit, and decreases when deviating from the speed limit;

[0045] (2) Driving safety indicators Calculated by combining TTC, lateral safety distance, and collision loss factor:

[0046]

[0047] Among them, TTC tPredict the collision time at the current speed; d lat w represents the lateral distance to adjacent vehicles, and w represents the lane width; f represents the lateral distance to adjacent vehicles. c v is a binary index of the occurrence of the collision. x ρ is the longitudinal velocity of the vehicle. ttc ρ lat and k c is a hyperparameter, and ∈ is a small constant to prevent the denominator from being zero;

[0048] (3) Driving comfort index Smoothness assessment based on acceleration and steering wheel angle changes:

[0049]

[0050] in, and ρ represents the rate of change of longitudinal acceleration and lateral rotation, respectively. thr and ρ ste For hyperparameters;

[0051] (4) Collision avoidance task completion index Calculated by considering the relative positional changes of adjacent background vehicles i∈C\{k}:

[0052]

[0053] Here, set C represents the set of all vehicles in the traffic scenario, vehicle k represents the autonomous vehicle, i.e., the autonomous vehicle that executes control decisions, and vehicle i represents other background vehicles besides the autonomous vehicle. These represent the vehicle's state at time steps t and t-1, respectively. These represent the states of vehicle i at time steps t and t-1, respectively. Let be the two-dimensional planar position coordinates of the vehicle at time step t. Let be the two-dimensional planar position coordinates of background vehicle i at time step t. c is the squared distance between the vehicle and vehicle i at time step t, which is the squared distance between their center points. d The safety threshold distance is used to determine whether a collision avoidance calculation range has been entered; ρ i The vehicle weight indicator factor is used when the relative position of vehicle k to vehicle i is less than the safety threshold c. d , ρ i =1, otherwise ρ i =0; c r This is an adjustment coefficient used to control the weight of the collision avoidance index;

[0054] (5) Cost function c t In contrast to the reward function, the cost function is defined as a weighted combination of risk metrics:

[0055]

[0056] Where λ coll , λ jerk and λ lane is a non-negative hyperparameter used to adjust the weights of the collision risk term, acceleration term, and lane departure penalty term, respectively; f c This is a collision risk measurement function, calculated based on whether the relative distance between the vehicle and adjacent vehicles or obstacles is less than a safety threshold. Longitudinal velocity v t The second derivative with respect to time, or jerk, is used to measure the degree of abrupt change in acceleration. t is the lane departure penalty term, which is 1 if the vehicle goes beyond the lane boundary, and 0 otherwise; t is the discrete time step.

[0057] Furthermore, in step 3, the long-term strategy is optimized using a soft actor-critic architecture based on Lagrange constraints. The strategy optimization objective is to maximize the expected cumulative reward while ensuring that the cumulative cost does not exceed a preset threshold ε. c The objective function is defined as follows:

[0058]

[0059] Where J(π) is the policy objective function, π is the policy function, and given state s t Output action distribution; ρ represents the mathematical expectation operator; π For the state-action access distribution induced under policy π; s t Let a be the state at time step t. t For time step t, γ represents the action; γ∈(0,1] is the discount factor; γ t With c t These are the reward function and the cost function, respectively; λ is the Lagrange multiplier, used to dynamically adjust the cost constraint weights; α is the entropy regularization coefficient, used to control the exploration capability of the policy distribution; The entropy of the strategy distribution; t∈{0,1,…,T} is the discrete time step, and T is the finite time range, i.e. the planning / evaluation time domain;

[0060] (1) Strategy and value network optimization;

[0061] Policy Network π θ (a t |s t ) and Value Network V ψ (s t Q-value network Q φi (s t ,at ), Q φ2 (s t ,a t Iteratively update according to the following objective function

[0062]

[0063] Where, π θ (a t |s t () is a policy network with parameter θ and input state s. t Output action a t The probability distribution of V; ψ (s t ( ) represents a value network with parameter ψ and input state s. t Output the value estimate of this state; Q φi (s t ,a t () is a Q-value network with parameters φi and input state-action pairs (s) t ,a t The corresponding action value estimate is output, where i = 1, 2 represents a dual-Q network; This represents the experience replay buffer, storing the state-action-reward-cost-next-state sequence generated by real interactions or the world model, used for training sampling; t The state at time step t; a t For the action at time step t; r t The instant reward for time step t; c t Let be the instantaneous cost at time step t; λ be the Lagrange multiplier used for dual weighting of the cost constraint; γ be the discount factor used to control the decay weights of future rewards and costs; α be the entropy regularization coefficient used to balance reward maximization and policy exploration; y t The target value for the Q-value network consists of the immediate reward – cost – and the discounted value of the next state; Q φ (s t ,a t ) for calculating J π With J V The Q-value network used at that time, taking Q φ1 With Q φ2 The minimum value is

[0064] logπ θ (a t |s t ) represents the policy in state s t Choose action a t The logarithmic probability is used for entropy calculation;

[0065] (2) Lagrange multiplier update;

[0066] λ is updated online using the following formula to meet cost constraints:

[0067]

[0068] Where, η λ The learning rate of the Lagrange multiplier;

[0069] (3) Adaptive adjustment of entropy coefficient;

[0070] The entropy regularization coefficient α passes through the target entropy Dynamic adjustment:

[0071]

[0072] Where, η α The update rate of the entropy coefficient;

[0073] (4) Strategy iteration;

[0074] A soft policy iterative mechanism is employed: Soft policy evaluation: The Q-value network is iteratively updated while the policy π is fixed until it converges to the soft Q-value Q. π Soft policy improvement: Minimize approximate policy update using KL divergence:

[0075]

[0076] Where, π new For the updated strategy; π ′ π represents the candidate policy, belonging to the policy space Π; π is the existing policy function, with input state s. t Output action distribution; s t The state at time step t; a t For the action at time step t; D KL KL divergence is used to measure the difference between two distributions; exp(Q π (s t ,·)) represents the exponentialized result of the Q-value function output, used to construct a soft-maximization distribution; Z π (s t ) is the normalization factor, which is the partitioning function that ensures the right-hand side distribution is a probability distribution.

[0077] Furthermore, in step 4, the dual-strategy fusion mechanism includes a set of safe trajectories filtered based on risk perception. Elite trajectories are selected, and the action sequences of these elite trajectories are used to refit a Gaussian distribution onto a short-term strategy to achieve dynamic fusion and smooth switching with a long-term strategy; the method includes:

[0078] First, based on the risk perception filtering mechanism in step 2, a set of safe trajectories that satisfy the cumulative cost constraint is obtained. Each trajectory

[0079] Where M is the number of safe trajectories, i.e., the set. The total number of trajectories in the middle; τ j Let H be the j-th trajectory in the set; H is the trajectory length, i.e., the maximum number of time steps contained in the trajectory. Trajectories τ j The status, actions, immediate rewards, and immediate costs at time step t;

[0080] The set of safe trajectories is calculated based on cumulative rewards. Alternatively, the tracks can be sorted according to the criterion of minimizing cost, and the top K tracks with the highest reward or lowest cost can be selected to form an elite track set.

[0081] Secondly, the action sequence {a} in the elite trajectory t elite This is used to parameterize short-term policies by fitting a conditional Gaussian distribution to the KL divergence:

[0082]

[0083] Where s represents the local state of the vehicle in the current control cycle; a represents the candidate action of the vehicle in state s; π short (a|s) is the conditional probability distribution of the short-term strategy. The input is state s and the output is the distribution of action a. The mean μ(s) and covariance Σ(s) are the mean and covariance of the action distribution of the short-term strategy in state s, respectively. They are calculated from the statistical characteristics of the elite trajectory action sequence, enabling the short-term strategy to dynamically respond to the immediate risk of the local state within the current control period.

[0084] Then, the long-term strategy π long (a|s) and the refitted short-term Gaussian policy π short (a|s) are fused according to weights to ultimately control the action. for:

[0085]

[0086] in, This refers to the final control action at time step t; Output actions for the long-term strategy at time step t; The short-term policy outputs the action at time step t; α is the fusion weight of the long-term and short-term policies, with a larger value indicating a greater bias towards the long-term policy; the fusion weight α is based on the set of safe trajectories. The quantity and reward advantage of elite tracks The environmental risk level will be dynamically adjusted, among which The average cumulative reward for the elite trajectory set; The average cumulative reward for generating trajectories for long-term strategies;

[0087] Finally, an exponential smoothing function is used to process the final control action:

[0088]

[0089] Where η∈[0,1) is the smoothing coefficient.

[0090] Secondly, this invention discloses an autonomous driving dual-strategy risk perception control system based on the world model-driven autonomous driving reinforcement learning dual-strategy risk perception control method described above, comprising: a data acquisition and management module, a world model training and deduction module, a trajectory generation and risk assessment module, a dual-strategy control and fusion module, and a strategy update and optimization module.

[0091] The modules work together to achieve a closed-loop control process from data acquisition, potential trajectory generation, risk filtering, to the fusion of long-term and short-term strategies to output control actions.

[0092] The data acquisition and management module is used to collect data such as observation status, action commands, reward and cost indicators from vehicle sensors and simulation environment, and at the same time integrate the driver demonstration dataset and store it in the replay buffer.

[0093] The data acquisition and management module supports time-series slicing and standardization of historical data for use in self-supervised training of the world model and offline optimization of the policy network.

[0094] The world model training and inference module adopts a recursive state-space model structure, including a state encoder, a state transition network, an observation decoder, and a reward and cost decoder. It achieves high-fidelity environment modeling by maximizing state reconstruction accuracy, reward prediction accuracy, and potential dynamic consistency.

[0095] The world model training and extrapolation module performs multi-step forward extrapolation in the current state to generate a set of virtual trajectories containing state, action, reward and cost, which are used for risk assessment and short-term strategy generation.

[0096] The trajectory generation and risk assessment module calculates the cumulative reward and cumulative cost of each candidate trajectory based on the multi-trajectory prediction results of the world model, filters out high-risk trajectories through a preset risk threshold, and forms a set of safe trajectories.

[0097] Within the set of safe trajectories, further select the elite trajectories with the highest reward or lowest cost, and use them for the Gaussian refitting process of the short-term strategy.

[0098] The dual-strategy control and fusion module consists of a long-term strategy and a short-term strategy. The long-term strategy adopts a soft actor-critic architecture based on Lagrange constraints and is updated by combining real environment data and world model simulation data. The short-term strategy is based on the refitting of elite trajectories in the set of safe trajectories into a Gaussian policy network.

[0099] The dual-strategy control and fusion module weights and superimposes the output actions of the two strategies through fusion weights, and dynamically adjusts the fusion ratio according to risk level, elite trajectory reward advantage and strategy performance differences to generate the final control action signal.

[0100] The strategy update and optimization module is used to continuously collect real-state-action-reward-cost data during the interaction between the vehicle and the environment, and alternately use it with virtual data generated by the world model for iterative optimization of the strategy network.

[0101] The strategy update and optimization module uses a reward and cost-driven objective function to update long-term strategy parameters, and simultaneously utilizes the statistical characteristics of elite trajectories to optimize the Gaussian distribution parameters of short-term strategies, thereby achieving closed-loop adaptive optimization of the dual-strategy collaborative control architecture.

[0102] Thirdly, the present invention discloses a storage medium storing a computer program that, when the computer program is run on a computer processor, executes the world model-driven autonomous driving reinforcement learning dual-strategy risk perception control method as described above.

[0103] The computer program includes an instruction set for implementing data acquisition and management, training and trajectory deduction of the recursive state-space world model, risk-aware trajectory filtering, coordinated control of long-term and short-term strategies, dual-strategy fusion and final control action generation, and closed-loop strategy optimization based on rewards and costs.

[0104] The storage medium is a non-transitory computer-readable storage medium, including but not limited to hard disks, flash memory, optical disks, magnetic tapes, memory cards, or other media capable of storing computer instructions;

[0105] When the program is loaded into the computer and executed by it, the above instructions are invoked to realize the complete control process from the collection of the demonstration dataset, the generation of the virtual trajectory of the world model, to the dynamic fusion of the two strategies and risk perception.

[0106] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0107] First, this invention innovatively integrates world model and reinforcement learning technology. By constructing a recursive state space model with potential trajectory extrapolation capabilities, it achieves forward-looking simulation and evaluation of future driving scenarios. This enables the system to have stronger predictive capabilities and emergency decision-making efficiency when facing complex dynamic environments and sudden risk events, effectively improving the reaction speed and control reliability of autonomous commercial vehicles in various autonomous driving scenarios.

[0108] Second, this invention constructs a dual-strategy collaborative control structure. By co-optimizing a long-term strategy under Lagrange constraints and a short-term strategy that relies on the prediction results of the world model, it achieves a rapid response to local high-frequency disturbances while ensuring global security. This optimizes the stability and accuracy of action generation and provides stronger environmental adaptability under high-risk and high-dynamic conditions.

[0109] Third, the control method proposed in this invention achieves closed-loop self-improvement of the strategy through a joint optimization mechanism that integrates reward indicators and cost constraints. During real-world interaction, the system can continuously iterate its strategy and adapt its structure, significantly enhancing the online learning and task generalization capabilities of the control strategy, and ensuring stability and robustness during long-term operation.

[0110] Fourth, compared to the high dependence of traditional control methods on vehicle dynamics modeling, the data-driven structural design of this invention reduces the requirement for accurate modeling and improves the rapid deployment capability of the algorithm on different vehicle platforms and scenarios through modular architecture, thereby enhancing the scalability and engineering implementation efficiency of the entire control system. Attached Figure Description

[0111] Figure 1 This is a schematic diagram of the overall architecture of the world model-driven dual-strategy risk perception control system for autonomous driving according to the present invention.

[0112] Figure 2 This is a schematic diagram of the trajectory extrapolation and risk perception filtering process based on the world model of the present invention. Detailed Implementation

[0113] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0114] This invention discloses a world model-driven dual-strategy risk perception control method, system, and storage medium for autonomous driving reinforcement learning. This method primarily addresses the problems of low decision-making efficiency, unsmooth strategy switching, and insufficient risk perception in existing technologies for autonomous driving tasks. This embodiment focuses on how to efficiently solve autonomous driving tasks through the construction of a world model, the training of a reinforcement learning network model, and dual-strategy collaborative control. The method includes the following steps:

[0115] Step 1: Data Acquisition and World Model Construction.

[0116] The first step of this invention is to collect driving data from human drivers and construct an expert experience dataset. Using a driving simulator and simulation platform, the system collects state information perceived by the sensors of the autonomous vehicle at each moment of autonomous driving (such as the road environment, the relative positions and speeds of traffic participants, etc.), as well as the control actions taken by the human driver at that moment (such as steering wheel angle, throttle opening, braking force, etc.). Specifically, the system constructs an expert experience dataset based on observation information collected by onboard sensors (such as LiDAR, cameras, millimeter-wave radar, GPS, etc.), including the road environment, the states of traffic participants (such as position, speed, acceleration, etc.), and the driver's control actions (such as steering, acceleration, braking, etc.). Among them o t For the sensor's observation state, a t This dataset corresponds to various driving actions. It is used to train a world model, enabling autonomous driving systems to simulate different driving scenarios and learn driver decision-making behaviors.

[0117] These data are stored in matrix form for easy later use. For example... Figure 1 As shown, expert driving experience data is first acquired through a data acquisition module to generate an expert experience dataset. This dataset then interacts with the world model in a semi-supervised training mode to assist the agent's policy learning. Simultaneously, the acquired expert dataset is added to the replay buffer in a decaying manner to help the autonomous commercial vehicle's agent improve its exploration and learning efficiency in the early stages.

[0118] In this invention, the world model is an environmental dynamic simulation module built on the recursive state space model (RSSM). It is used to generate environmental state sequences, predict action results and their safety indicators without relying on real interactions, so as to simulate and evaluate potential future driving trajectories.

[0119] The world model consists of a state encoder, a state transition module, an observation decoder, and a reward-cost prediction head (reward and cost decoder). The encoder compresses and encodes high-dimensional observations acquired by vehicle sensors (such as images, laser point clouds, radar range, and speed information) into latent state representations; that is, it transforms sensor observation data into a state representation. t Mapping to latent state z t The state transition function employs an RSSM architecture with latent variable modeling, modeling the impact of action inputs on the evolution of latent states, achieving high-dimensional nested prediction from the current state to the next time step; that is, based on the current state z... t and action a t Predict the next potential state z t+1It is used to simulate environmental dynamics. The observation decoder, based on the potential state z... t Reconstructed observations t The model is trained by maximizing reconstruction accuracy. The reward prediction head and cost prediction head are used to output quantitative indicators of each predicted trajectory in terms of task benefit and safety, providing an important basis for policy training; the decoder reconstructs observations to assist in the convergence of world model training.

[0120] By training the loss function L model Joint optimization, the model's loss includes observation reconstruction error, reward and cost prediction error:

[0121]

[0122] Among them, s t Let a be the state vector at time step t; t r is the action vector at time step t; t c is the true reward value at time step t; t The true cost value for time step t; t For the actual observations at time step t (such as sensor observations); The reward decoder in the input (s) t ,a t The predicted reward value under the given conditions; For cost decoder in input (s t ,a t The predicted cost value under the following conditions; The observations predicted by the observation decoder under the input latent state conditions; For experience datasets or replay buffers, containing data generated by real interactions or models. t ,a t ,r t ,c t ,o t ) Sample; λ1 and λ2 are weighting coefficients, which control the effects of rewards, costs and observation reconstruction errors, respectively.

[0123] This world model can generate potential trajectories based on the current observation state and simulate the state, rewards, and costs at future moments, providing a basis for subsequent trajectory generation and risk assessment.

[0124] Step 2: Construct a potential trajectory extrapolation and risk perception filtering mechanism based on a world model.

[0125] Based on the world model trained in step 1, multi-trajectory extrapolation is performed to generate multiple potential trajectory sequences. Each trajectory It includes information on state, action, reward, and cost from the current moment to future moments, where τj Let H be the j-th trajectory in the set, and H be the trajectory length, i.e., the maximum number of time steps contained in the trajectory. Trajectories τ j The state, action, immediate reward, and immediate cost at time step t.

[0126] The cumulative reward and cumulative cost for each trajectory are calculated using the following formula:

[0127]

[0128] Here, γ is a discount factor used to adjust the impact of future rewards and costs. This is achieved through a preset cost threshold ε. c Filter out low-risk trajectory sets That is, condition C is satisfied. j ≤ε c The trajectory, where M is the number of safe trajectories, i.e., the set. The total number of medium-risk trajectories. The low-risk trajectory set contains possible control paths, which will serve as candidate paths for subsequent decisions. For further optimization, all selected low-risk trajectories are sorted according to reward or cost, and the top K trajectories with the highest reward or lowest cost are selected to form an elite trajectory set. Elite trajectories are subsets of trajectories that maximize rewards or minimize costs. By selecting the best trajectory from the elite trajectory set, a decision-making basis is provided for subsequent short-term strategies.

[0129] like Figure 2 As shown, based on the world model, action trajectories can be batch-generated and a candidate trajectory library can be generated. Then, a risk assessment filtering model is used to filter the trajectories layer by layer to obtain a safe trajectory set and an elite trajectory set. The risk filtering mechanism uses expected risk thresholds and soft constraint weighted indicators as conditions to ensure that the action candidates entering the strategy module have high feasibility and safety.

[0130] Step 3: Construct a dual-strategy collaborative control structure that integrates long-term and short-term coordination.

[0131] Based on the elite trajectory set, this invention designs a long-term-short-term two-layer strategy structure to achieve dynamic balance control of decision-making timeliness and stability.

[0132] The long-term strategy employs a Lagrangian Soft Actor-Critic (L-SAC) architecture based on Lagrangian multiplier-constrained optimization. It achieves joint control of action rewards and risk costs by constructing an optimization objective function with a penalty term. The objective function takes the form:

[0133]

[0134] Where λ is the dynamically adjusted cost Lagrange factor used to adjust security constraints, and α is the entropy regularization coefficient used to maintain policy diversity and ensure that the policy has sufficient exploratory power.

[0135] The action sequence {a} in the elite trajectory t elite This is used to parameterize short-term policies by fitting a conditional Gaussian distribution to the KL divergence:

[0136]

[0137] Where s represents the local state of the vehicle in the current control cycle; a represents the candidate action of the vehicle in state s; π short (a|s) is the conditional probability distribution of the short-term strategy, with the input being state s and the output being the distribution of action a; the mean μ(s) and covariance Σ(s) are the mean and covariance of the action distribution of the short-term strategy in state s, respectively, which are calculated from the statistical characteristics of the elite trajectory action sequence, enabling the short-term strategy to dynamically respond to the immediate risks of the local state within the current control period.

[0138] The short-term strategy rapidly retrieves and interpolates elite trajectories generated from the current state in the world model, employing a lightweight neural network module for trajectory matching and action output to improve local reaction speed and system timeliness. This strategy primarily executes local segments of actions within elite trajectories, making it particularly suitable for handling sudden risks and tasks requiring precise control.

[0139] Short-term strategies employ immediate control by selecting elite trajectories from low-risk paths, based on either reward maximization or cost minimization. The choice of this control action is determined by the following formula:

[0140]

[0141] Among them, R j C is the cumulative reward for the trajectory. j Let λ be the cumulative cost of the trajectory, and λ be the safety adjustment factor.

[0142] Step 4: Integration and dynamic adjustment of dual strategies.

[0143] The output actions of the long-term and short-term strategies are dynamically weighted and fused to generate the final control action.

[0144]

[0145] in, This refers to the final control action at time step t; Output actions for the long-term strategy at time step t; The short-term policy outputs the action at time step t; α is the fusion weight of the long-term and short-term policies, with a larger value indicating a greater bias towards the long-term policy; the fusion weight α is based on the set of safe trajectories. The quantity and reward advantage of elite tracks The environmental risk level will be dynamically adjusted, among which The average cumulative reward for the elite trajectory set; The average cumulative reward for generating trajectories for the long-term strategy is used; the fusion weight α is dynamically adjusted based on the risk level of the current environment and the number of safe trajectory sets. Specifically, in high-risk scenarios, the weight of short-term strategies increases; in low-risk scenarios, long-term strategies dominate. Finally, an exponential smoothing function is used to process the final control action.

[0146]

[0147] Where η∈[0,1) is the smoothing coefficient. The formula for calculating the fusion weight is:

[0148]

[0149] Where α is the fusion weight of long-term and short-term strategies, and the larger the value, the higher the proportion of long-term strategy in the final control action; σ(·) is the Sigmoid function, which is used to map the input to the (0,1) interval; The number of trajectories in the safe trajectory set to satisfy the cumulative cost constraint; N th β0 is the threshold for the number of tracks, used to distinguish between high and low levels of the number of safe tracks; β1 is the bias parameter, used to adjust the initial value of the fusion weights; β2 is the adjustment parameter, used to control the number of safe tracks. The influence of risk level on the fusion weights is β2, which is an adjustment parameter used to control the influence of risk level on the fusion weights. Risk level is a risk measure of the current environment, obtained from the cumulative value of the cost function or the output of the risk perception model, and is used to reflect the risk level of the current scenario. The final control action is executed through the vehicle control system to control the vehicle's motion state and interact with the environment.

[0150] Step 5: Reinforce the closed-loop interaction and continuous policy update mechanism of learning.

[0151] A closed-loop update mechanism based on real-world environmental interaction is constructed to achieve online optimization of the policy and model. This closed-loop mechanism includes environmental state sampling, reward-cost feedback calculation, writing experience into a replay buffer, and incremental updates of the policy network and world model. Within each control cycle, the system updates the policy based on the perceived state s. t Action a generated by the current strategy t And execute it in the autonomous vehicle to obtain feedback. t+1,r t ,c t The interactive quadruple (s) t ,a t ,r t ,c t Store in the experience buffer.

[0152] By interacting with the environment, the system continuously collects real-state-action-reward-cost data and combines it with simulated data based on the world model for iterative training to optimize the control strategy. Through this closed-loop optimization mechanism, the system can continuously adapt to new environmental changes, improving the stability and robustness of the control strategy. In each training cycle, the world model and policy network jointly optimize using alternating real and simulated data, continuously improving the system's generalization ability and stability in complex environments.

[0153] In the initial training phase, this invention incorporates human expert experience as guidance, with expert samples given a priority ratio ε. t Weighted inputs are added to the experience pool to guide the system in learning effective state-action mappings. As the system trains, this weighted input ε... t As the loss decays to zero over time, it gradually transitions to a fully autonomous learning stage. During strategy optimization, the entropy regularization loss function of reinforcement learning is used for goal-oriented training to ensure that long-term returns are maximized while meeting cost constraints. Furthermore, the bias of the world model is continuously corrected using real-vehicle data, thereby achieving a synergistic improvement in task generalization ability and execution stability.

[0154] Through the above steps, this invention provides an innovative world-model-driven dual-strategy risk perception control method for autonomous driving reinforcement learning, which can significantly improve the response capability and decision-making efficiency of autonomous commercial vehicles in complex dynamic scenarios such as high-speed driving and emergency collision avoidance. Figure 1 and Figure 2As shown, this invention achieves the generation and refined evaluation of high-quality control candidates by integrating a recursive state-space model and a trajectory risk perception screening mechanism. Based on this, it constructs a two-layer control structure that coordinates long-term and short-term strategies to balance global safety optimization and local time-efficiency control. Through online interactive feedback and a closed-loop iterative optimization mechanism, this invention enhances the generalization ability and robustness of the autonomous driving system under unstructured conditions while strengthening control accuracy and behavioral stability. Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, causing a series of operational steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that run on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0158] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0159] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A world model-driven reinforcement learning-based dual-strategy risk perception control method for autonomous driving, characterized in that, The method includes the following steps: Step 1: Collect driver driving data, construct a demonstration dataset of observation-action pairs, and construct a world model of the recursive state-space model RSSM structure based on the dataset through self-supervised learning; The world model is used to generate trajectories and predict states in the potential state space, and outputs the state, reward and cost information at future time moments. Step 2: Based on the world model built in Step 1, perform multi-trajectory extrapolation on the current state in each control cycle to generate multiple candidate trajectory sequences, and conduct a joint evaluation of reward and cost for each trajectory; A risk perception filtering mechanism is adopted to select a set of safe trajectories that meet the preset cost constraints from all candidate trajectories, forming a low-risk trajectories subset for strategy decision-making; Step 3: Based on the policy optimization objective, construct a reinforcement learning control structure composed of long-term and short-term policies in synergy; The long-term strategy is trained using a soft actor critic architecture based on Lagrange multiplier constraints and is updated using real-world interaction data and simulated data generated by the world model in the latent state space. The short-term strategy extracts the expected value of the current action from the trajectory sequence with the maximum reward or minimum cost among the low-risk trajectories selected in step 2, and uses it as the immediate control strategy. Step 4: Integrate the output actions of the long-term and short-term strategies in Step 3, dynamically adjust the integration weights based on the current environmental risk level, and generate the final control action; The control actions are applied to the vehicle control system to perform interactions with the environment and collect new observation data for subsequent iterative optimization of the world model and policy network, so as to achieve closed-loop updates of the control policy. Step 5: Continue to add the real-state-action-reward-cost data collected during the interaction to the training process of the world model and policy network; Meanwhile, the imagined data generated by the world model in the potential state space, which includes virtual interaction sequences of state transitions, reward feedback and cost assessment predicted by the model, is used alternately with real data to update network parameters, forming a closed-loop adaptive learning mechanism that combines real and virtual experiences.

2. The world model-driven dual-strategy risk perception control method for autonomous driving based on reinforcement learning, as described in claim 1, is characterized in that... In step 1, the recursive state space model includes a state encoder, a state transition model, an observation decoder, a reward decoder, and a cost decoder. The state encoder generates a latent state representation based on sensor observations, and the state transition model predicts the posterior state distribution given historical states and actions. The observation decoder reconstructs the observations based on the potential states, and the reward decoder and cost decoder output the reward value and cost value of the trajectory segment, respectively. The recursive state space model is trained using a multi-task loss function, which consists of state reconstruction error, reward prediction error, cost prediction error, and potential dynamic consistency error.

3. The world model-driven dual-strategy risk perception control method for autonomous driving reinforcement learning according to claim 2, characterized in that, In step 2, the trajectory generation is achieved by performing several forward samplings with a preset step size by the state transition model in the current potential state to obtain several predicted trajectories containing state, action, reward and cost. The risk perception filtering mechanism makes a judgment based on the cumulative cost estimate of each trajectory, and determines the trajectories that meet the condition that the cumulative cost does not exceed the preset risk threshold ε as low-risk trajectories, thus forming a set of safe trajectories; The set of safe trajectories is further sorted according to predefined evaluation indicators, and the trajectory with the lowest cost or highest reward is selected for subsequent short-term strategy generation and action selection.

4. The world model-driven dual-strategy risk perception control method for autonomous driving reinforcement learning according to claim 1, characterized in that, In step 3, the long-term strategy is constructed using a soft actor critic architecture based on Lagrange constraints, which includes a policy network, a value network, and two Q-value networks. The policy network generates continuous actions based on the current observed state; two Q-value networks output action values ​​for a given state-action pair, where the action value is characterized by both cumulative reward and expected cost; and the value network estimates the long-term return for a given state. By introducing Lagrange multipliers to constrain the expected cost, an optimization function is formed with the objective of maximizing the cumulative reward and limiting the cumulative cost to no more than a preset threshold. The policy network parameters are updated using a proximal policy optimization approach, and a hybrid training method combining real interaction data and simulated trajectory data generated by the world model is used to obtain a long-term policy.

5. The world model-driven dual-strategy risk perception control method for autonomous driving based on reinforcement learning, as described in claim 1 or 4, is characterized in that... In step 3, the short-term strategy is selected from the set of low-risk trajectories generated by the world model, specifically including: Calculate the cumulative reward and cumulative cost for each candidate trajectory, and select the trajectory with the highest cumulative reward or the lowest cost under the cost constraint as the current short-term reference trajectory. The mean of the action sequence corresponding to each time step in the reference trajectory is extracted and used as the output action of the short-term strategy in the current state. The short-term strategy mentioned above does not participate in the training process, but is only used as a temporary action generation method under the current control cycle; A Gaussian distribution-based action refitting mechanism is adopted to fit the reference trajectory action sequence into a conditional Gaussian policy distribution by minimizing the KL divergence, thereby obtaining the action mean and covariance parameters, and performing noisy sampling during the control execution phase. The final control action is obtained by weighting the long-term and short-term strategy actions according to a fusion weight α; the fusion weight is dynamically adjusted based on the number of current safe trajectory sets and the risk level. The risk level is determined by dividing the candidate trajectories generated by the world model into ranges based on the cumulative cost values ​​output by the cost decoder. Trajectories with a cumulative cost value lower than the set value are defined as low-risk, trajectories with a cumulative cost value higher than the set value are defined as high-risk, and trajectories in the middle range are defined as medium-risk.

6. The world model-driven dual-strategy risk perception control method for autonomous driving reinforcement learning according to claim 1, characterized in that, In steps 2 and 3, the calculation methods for rewards and costs are as follows: The reward function r used t With cost function c t It consists of a weighted combination of multi-dimensional driving performance indicators, including driving efficiency indicators. Driving safety indicators Driving comfort index and collision avoidance mission completion indicators The four sub-indicators and the reward function are defined as follows: Where, ρ eff ρ safe ρ comf and ρ avoid These are the hyperparameters that adjust the weights of each sub-index, and satisfy ρ eff +ρ safe +ρ comf +ρ avoid =1; (1) Driving efficiency indicators Based on the vehicle's current speed v t With target speed limit v limit Maximum vehicle speed v max Deviation calculation: When v t The reward is highest when approaching the speed limit, and decreases when deviating from the speed limit; (2) Driving safety indicators Calculated by combining TTC, lateral safety distance, and collision loss factor: Among them, TTC t Predict the collision time at the current speed; d lat w represents the lateral distance to adjacent vehicles, and w represents the lane width; f represents the lateral distance to adjacent vehicles. c v is a binary index of the occurrence of the collision. x ρ is the longitudinal velocity of the vehicle. ttc ρ lat and k c is a hyperparameter, and ∈ is a small constant to prevent the denominator from being zero; (3) Driving comfort index Smoothness assessment based on acceleration and steering wheel angle changes: in, and ρ represents the rate of change of longitudinal acceleration and lateral rotation, respectively. thr and ρ ste For hyperparameters; (4) Collision avoidance task completion index Calculated by considering the relative positional changes of adjacent background vehicles i∈C\{k}: Here, set C represents the set of all vehicles in the traffic scenario, vehicle k represents the autonomous vehicle, i.e., the autonomous vehicle that executes control decisions, and vehicle i represents other background vehicles besides the autonomous vehicle. These represent the vehicle's state at time steps t and t-1, respectively. These represent the states of vehicle i at time steps t and t-1, respectively. Let be the two-dimensional planar position coordinates of the vehicle at time step t. Let be the two-dimensional planar position coordinates of background vehicle i at time step t. c is the squared distance between the vehicle and vehicle i at time step t, which is the squared distance between their center points. d The safety threshold distance is used to determine whether a collision avoidance calculation range has been entered; ρ i The vehicle weight indicator factor is used when the relative position of vehicle k to vehicle i is less than the safety threshold c. d , ρ i =1, otherwise ρ i =0; c r This is an adjustment coefficient used to control the weight of the collision avoidance index; (5) Cost function c t In contrast to the reward function, the cost function is defined as a weighted combination of risk metrics: Where λ coll , λ jerk and λ lane is a non-negative hyperparameter used to adjust the weights of the collision risk term, acceleration term, and lane departure penalty term, respectively; f c This is a collision risk measurement function, calculated based on whether the relative distance between the vehicle and adjacent vehicles or obstacles is less than a safety threshold. Longitudinal velocity v t The second derivative with respect to time, or jerk, is used to measure the degree of abrupt change in acceleration. t is the lane departure penalty term, which is 1 if the vehicle goes beyond the lane boundary, and 0 otherwise; t is the discrete time step.

7. The world model-driven dual-strategy risk perception control method for autonomous driving based on reinforcement learning, as described in claim 1 or 6, is characterized in that... In step 3, the long-term strategy is optimized using a soft actor-critic architecture based on Lagrange constraints. The optimization objective is to maximize the expected cumulative reward while ensuring that the cumulative cost does not exceed a preset threshold ε. c The objective function is defined as follows: Where J(π) is the policy objective function, π is the policy function, and given state s t Output action distribution; ρ represents the mathematical expectation operator; π For the state-action access distribution induced under policy π; s t Let a be the state at time step t. t For time step t, γ represents the action; γ∈(0,1] is the discount factor; γ t With c t These are the reward function and the cost function, respectively; λ is the Lagrange multiplier, used to dynamically adjust the cost constraint weights; α is the entropy regularization coefficient, used to control the exploration capability of the policy distribution; The entropy of the strategy distribution; t∈{0,1,…,T} is the discrete time step, and T is the finite time range, i.e. the planning / evaluation time domain; (1) Strategy and value network optimization; Policy Network π θ (a t |s t ) and Value Network V ψ (s t Q-value network Q φi (s t ,a t ), Q φ2 (s t ,a t Iterative updates are performed according to the following objective function: y t =r t -λc t +γ(V ψ (s t+1 )) Where, π θ (a t |s t () is a policy network with parameter θ and input state s. t Output action a t The probability distribution of V; ψ (s t ( ) represents a value network with parameter ψ and input state s. t Output the value estimate of this state; Q φi (s t ,a t () is a Q-value network with parameters φi and input state-action pairs (s) t ,a t The corresponding action value estimate is output, where i = 1, 2 represents a dual-Q network; This represents the experience replay buffer, storing the state-action-reward-cost-next-state sequence generated by real interactions or the world model, used for training sampling; t The state at time step t; a t For the action at time step t; r t The instant reward for time step t; c t Let be the instantaneous cost at time step t; λ be the Lagrange multiplier used for dual weighting of the cost constraint; γ be the discount factor used to control the decay weights of future rewards and costs; α be the entropy regularization coefficient used to balance reward maximization and policy exploration; y t The target value for the Q-value network consists of the immediate reward – cost – and the discounted value of the next state; Q φ (s t ,a t ) for calculating J π With J V The Q-value network used at that time, taking and The minimum value is logπ θ (a t |s t ) represents the policy in state s t Choose action a t The logarithmic probability is used for entropy calculation; (2) Lagrange multiplier update; λ is updated online using the following formula to meet cost constraints: Where, η λ The learning rate of the Lagrange multiplier; (3) Adaptive adjustment of entropy coefficient; The entropy regularization coefficient α passes through the target entropy Dynamic adjustment: Where, η α The update rate of the entropy coefficient; (4) Strategy iteration; A soft policy iterative mechanism is employed: Soft policy evaluation: The Q-value network is iteratively updated while the policy π is fixed until it converges to the soft Q-value Q. π Soft policy improvement: Minimize approximate policy update using KL divergence: Where, π new For the updated strategy; π ′ π represents the candidate policy, belonging to the policy space Π; π is the existing policy function, with input state s. t Output action distribution; s t The state at time step t; a t For the action at time step t; D KL KL divergence is used to measure the difference between two distributions; exp(Q π (s t ,·)) represents the exponentialized result of the Q-value function output, used to construct a soft-maximization distribution; Z π (s t ) is the normalization factor, which is the partitioning function that ensures the right-hand side distribution is a probability distribution.

8. The world model-driven dual-strategy risk perception control method for autonomous driving based on reinforcement learning, as described in claim 7, is characterized in that... In step 4, the dual-strategy fusion mechanism includes a set of safe trajectories filtered based on risk perception. Elite trajectories are selected, and the action sequences of these elite trajectories are used to refit a Gaussian distribution onto a short-term strategy to achieve dynamic fusion and smooth switching with a long-term strategy; the method includes: First, based on the risk perception filtering mechanism in step 2, a set of safe trajectories that satisfy the cumulative cost constraint is obtained. Each trajectory Where M is the number of safe trajectories, i.e., the set. The total number of trajectories in the middle; τ j Let H be the j-th trajectory in the set; H is the trajectory length, i.e., the maximum number of time steps contained in the trajectory. Trajectories τ j The status, actions, immediate rewards, and immediate costs at time step t; The set of safe trajectories is calculated based on cumulative rewards. Alternatively, the tracks can be sorted according to the criterion of minimizing cost, and the top K tracks with the highest reward or lowest cost can be selected to form an elite track set. Secondly, the action sequence {a} in the elite trajectory t elite This is used to parameterize short-term policies by fitting a conditional Gaussian distribution to the KL divergence: Where s represents the local state of the vehicle in the current control cycle; a represents the candidate action of the vehicle in state s; π short (a|s) is the conditional probability distribution of the short-term strategy. The input is state s and the output is the distribution of action a. The mean μ(s) and covariance Σ(s) are the mean and covariance of the action distribution of the short-term strategy in state s, respectively. They are calculated from the statistical characteristics of the elite trajectory action sequence, enabling the short-term strategy to dynamically respond to the immediate risk of the local state within the current control period. Then, the long-term strategy π long (a|s) and the refitted short-term Gaussian policy π short (a|s) are fused according to weights to ultimately control the action. for: in, This refers to the final control action at time step t; Output actions for the long-term strategy at time step t; The short-term policy outputs the action at time step t; α is the fusion weight of the long-term and short-term policies, with a larger value indicating a greater bias towards the long-term policy; the fusion weight α is based on the set of safe trajectories. The quantity and reward advantage of elite tracks The environmental risk level will be dynamically adjusted, among which The average cumulative reward for the elite trajectory set; The average cumulative reward for generating trajectories for long-term strategies; Finally, an exponential smoothing function is used to process the final control action: Where η∈[0,1) is the smoothing coefficient.

9. The world model-driven dual-strategy risk perception control method for autonomous driving based on reinforcement learning, as described in claim 1, is characterized in that... The autonomous driving dual-strategy risk perception control system constructed based on the method includes: a data acquisition and management module, a world model training and inference module, a trajectory generation and risk assessment module, a dual-strategy control and fusion module, and a strategy update and optimization module. The modules work together to achieve a closed-loop control process from data acquisition, potential trajectory generation, risk filtering, to the fusion of long-term and short-term strategies to output control actions. The data acquisition and management module is used to collect data such as observation status, action commands, reward and cost indicators from vehicle sensors and simulation environment, and at the same time integrate the driver demonstration dataset and store it in the replay buffer. The data acquisition and management module supports time-series slicing and standardization of historical data for use in self-supervised training of the world model and offline optimization of the policy network. The world model training and inference module adopts a recursive state-space model structure, including a state encoder, a state transition network, an observation decoder, and a reward and cost decoder. It achieves high-fidelity environment modeling by maximizing state reconstruction accuracy, reward prediction accuracy, and potential dynamic consistency. The world model training and extrapolation module performs multi-step forward extrapolation in the current state to generate a set of virtual trajectories containing state, action, reward and cost, which are used for risk assessment and short-term strategy generation. The trajectory generation and risk assessment module calculates the cumulative reward and cumulative cost of each candidate trajectory based on the multi-trajectory prediction results of the world model, filters out high-risk trajectories through a preset risk threshold, and forms a set of safe trajectories. Within the set of safe trajectories, further select the elite trajectories with the highest reward or lowest cost, and use them for the Gaussian refitting process of the short-term strategy. The dual-strategy control and fusion module consists of a long-term strategy and a short-term strategy. The long-term strategy adopts a soft actor-critic architecture based on Lagrange constraints and is updated by combining real environment data and world model simulation data. The short-term strategy is based on the refitting of elite trajectories in the set of safe trajectories into a Gaussian policy network. The dual-strategy control and fusion module weights and superimposes the output actions of the two strategies through fusion weights, and dynamically adjusts the fusion ratio according to risk level, elite trajectory reward advantage and strategy performance differences to generate the final control action signal. The strategy update and optimization module is used to continuously collect real-state-action-reward-cost data during the interaction between the vehicle and the environment, and alternately use it with virtual data generated by the world model for iterative optimization of the strategy network. The strategy update and optimization module uses a reward and cost-driven objective function to update long-term strategy parameters, and simultaneously utilizes the statistical characteristics of elite trajectories to optimize the Gaussian distribution parameters of short-term strategies, thereby achieving closed-loop adaptive optimization of the dual-strategy collaborative control architecture.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when run on a computer processor, executes the world model-driven autonomous driving reinforcement learning dual-strategy risk perception control method as described in claim 1 or 9. The computer program includes an instruction set for implementing data acquisition and management, training and trajectory deduction of the recursive state-space world model, risk-aware trajectory filtering, coordinated control of long-term and short-term strategies, dual-strategy fusion and final control action generation, and closed-loop strategy optimization based on rewards and costs. The storage medium is a non-transitory computer-readable storage medium, including but not limited to hard disks, flash memory, optical disks, magnetic tapes, memory cards, or other media capable of storing computer instructions; When the program is loaded into the computer and executed by it, the above instructions are invoked to realize the complete control process from the collection of the demonstration dataset, the generation of the virtual trajectory of the world model, to the dynamic fusion of the two strategies and risk perception.

Images