An expert automatic driving decision-making method based on reinforcement learning and diffusion model

Abstract

The application discloses an expert automatic driving decision-making method based on reinforcement learning and diffusion model, comprising the following steps: (1) constructing a data analysis module, receiving multi-dimensional BEV graph information input, and obtaining a feature vector containing environment information through a convolutional neural network and a multilayer perceptron; (2) constructing a random sample generation module based on an A* algorithm, generating a conventional path planning sample for training; (3) constructing a sample generation module based on a diffusion model, generating more sample data for training by using an extreme traffic scene dataset; (4) constructing a multi-expert decision-making module, accepting the feature vector and outputting a continuous space motion vector; and (5) constructing a decision safety layer, predicting the environment state at the next time step by using the BEV graph information, and performing risk assessment on the motion vector. The application enhances the generalization ability in extreme scenes through the diffusion model, and improves the decision robustness and smoothness in various traffic scenes through the mixed expert architecture.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving decision-making algorithms, specifically relating to an expert autonomous driving decision-making method based on reinforcement learning and diffusion models. Background Technology

[0002] The development of autonomous driving technology has placed higher demands on decision-making algorithms. Current mainstream autonomous driving decision-making methods are mainly based on deep learning and reinforcement learning techniques, using large amounts of driving data to train models for environmental perception and decision-making planning. However, existing technologies suffer from two key technical shortcomings in practical applications.

[0003] First, the generalization ability of models in extreme traffic scenarios is insufficient. Existing autonomous driving systems rely on data from regular driving scenarios for training, making it difficult to effectively handle decision-making problems in extreme traffic scenarios (such as sudden emergency lane changes, complex intersection interactions, and emergency avoidance under adverse weather conditions). Because extreme scenario data constitutes a very small proportion of the training set, the model's decision-making accuracy in these scenarios is significantly lower than in regular scenarios. According to relevant technical research, the decision-making accuracy of existing systems in extreme scenarios is on average 35-45% lower than in regular scenarios. The main reason is the scarcity of extreme scenario data, making it difficult for models to learn the driving behavior patterns of these "long-tail scenarios." Traditional data augmentation methods (such as random pruning and rotation) cannot generate realistic and diverse extreme scenario samples, and therefore cannot effectively solve the problem of scarce extreme scenario data.

[0004] Secondly, the robustness and smoothness of decision-making in diverse traffic scenarios are insufficient. Existing technologies often employ a single model architecture, making it difficult to simultaneously adapt to the decision-making needs of different traffic scenarios. This leads to abrupt decision-making and unnatural behavior during scenario transitions. For example, when the system switches from a regular road driving scenario to a complex intersection scenario, the vehicle may suddenly change speed or steering. This lack of smooth decision-making directly affects the consistency and predictability of the autonomous driving system's decisions. In Level 4 autonomous driving systems, the frequency of such abrupt decision-making occurs in 23.7% of cases, resulting in decreased reliability of the system's decisions in complex scenarios.

[0005] In terms of model training, existing methods suffer from low data utilization efficiency. Data is abundant in conventional scenarios, but scarce in extreme scenarios, resulting in a severe shortage of training samples for models in extreme situations. Although some studies have attempted to improve model performance through transfer learning or pre-training techniques, these methods cannot fundamentally solve the technical problem of scarce data in extreme scenarios and are difficult to guarantee decision consistency across diverse scenarios.

[0006] Therefore, there is an urgent need for an innovative autonomous driving decision-making method that can effectively solve the problem of insufficient generalization ability caused by the scarcity of data in extreme scenarios, while improving the robustness and smoothness of decision-making in diverse scenarios. Summary of the Invention

[0007] To address the aforementioned issues, this invention discloses an expert autonomous driving decision-making method based on reinforcement learning and diffusion models. By fusing diffusion models to generate extreme scenario data, using deep reinforcement learning to optimize decision-making strategies, and employing a hybrid expert architecture to achieve adaptive decision-making, the method significantly improves the system's generalization ability in extreme scenarios and the smoothness of decision-making in diverse scenarios, while ensuring the safety of the decision-making process.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] An expert autonomous driving decision-making method based on reinforcement learning and diffusion models includes the following steps:

[0010] (1) Constructing a data parsing module: receiving multi-dimensional BEV graph information input, and obtaining feature vectors containing environmental information through convolutional neural networks and multilayer perceptrons;

[0011] (2) Construct a random sample generation module based on the A* algorithm: Based on the A* algorithm, the number and location of obstacles in a fixed map size are randomly generated to obtain path planning result samples for training;

[0012] (3) Construct a sample generation module based on the diffusion model: Based on the diffusion model, generate more sample data for training by using traffic sample datasets under extreme conditions;

[0013] (4) Construct a multi-expert decision-making module: The multi-expert decision-making module consists of two experts: a general decision-making expert based on pre-trained reinforcement learning and a complex decision-making expert trained on extreme samples; The multi-expert decision-making module accepts the feature vector in step (1), and its decision results are integrated through a soft-gated network composed of a small multilayer sensing mechanism to obtain a motion vector in a continuous space.

[0014] (5) Construct a decision safety layer: The safety layer uses the BEV graph information input in step (1) to predict the environmental state of the next time step, and the continuous spatial motion vector in step (4) to calculate the index, and then judge the risk of the current decision. If the risk index is greater than the threshold, the expert weights output by the soft gating network in step (4) will be adjusted to obtain the planning result again. Otherwise, the decision result of the multi-expert decision module will be output, that is, the continuous spatial motion vector in step (4).

[0015] Furthermore, the specific steps for constructing the multi-expert decision-making module in step (4) are as follows:

[0016] (4.1) The general decision expert uses the path planning result sample generated in step (2) to clone behavior based on the SAC (flexible action-evaluation) algorithm. Its pre-trained weight result initializes the Actor in SAC with the help of the warm-start mechanism, and then is used for single-step decision reasoning.

[0017] (4.2) Complex decision experts are also based on the SAC algorithm. They use the extreme sample data generated in step (3) for initialization training. The initialization steps are the same as in step (4.1), and then used for single-step decision reasoning.

[0018] The beneficial effects of this invention are as follows:

[0019] The present invention discloses an expert autonomous driving decision-making method based on reinforcement learning and diffusion model. By using diffusion model and reinforcement learning method, the generalization ability of autonomous driving algorithm under extreme traffic scenarios is improved. By using hybrid expert method, the robustness and decision smoothness of autonomous driving algorithm under various traffic scenarios are improved. Attached Figure Description

[0020] Figure 1 This is a flowchart of the overall method.

[0021] Figure 2 It is a flowchart of the training and reasoning process for general decision-making experts.

[0022] Figure 3 It is a flowchart of the training and reasoning process for diffusion models and complex decision experts.

[0023] Figure 4 It is a diagram showing the results of single-step and continuous reasoning by general decision experts.

[0024] Figure 5 This is an example diagram of the input samples and generated samples of the diffusion model. Detailed Implementation

[0025] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] like Figure 1 As shown, the expert autonomous driving decision-making method based on reinforcement learning and diffusion models of the present invention includes:

[0027] Step 1: Construction and Feature Extraction of the Data Parsing Module

[0028] The system receives information from a Bird's Eye View (BEV) map, which includes obstacle distribution, obstacle types, and obstacle data. To enhance the model's ability to recognize traffic signals, this invention transforms traffic light states and signage rules into virtual obstacles and writes them into the BEV map to form a unified and computable environmental representation.

[0029] The BEV image is processed by a three-layer convolutional neural network to extract features, outputting an 8192-dimensional environment vector. Subsequently, the vehicle speed, acceleration, and the angle difference between the vehicle's current heading and the target direction are concatenated to form additional state features, resulting in an 8195-dimensional feature vector.

[0030] The feature vector is input into a three-layer multilayer perceptron (MLP) for dimensionality reduction, ultimately yielding a 128-dimensional vector with key decision information, which serves as the input for subsequent decision modules.

[0031] Step 2: Constructing a random sample generation module based on A*:

[0032] To construct the base trajectory samples that can be used for pre-training of the decision-maker, such as Figure 2 As shown, this invention generates obstacles of varying numbers and distributions in a fixed-grid BEV map using a random method. By setting a start and end point, the A* algorithm is used to calculate feasible trajectory paths.

[0033] The trajectory data is split into time steps and transformed into a 128-dimensional state vector consistent with step (1) through a CNN + MLP structure, which is used for behavior clone training and weight initialization of general experts.

[0034] Step 3: Construction of the extreme scenario sample generation module based on the diffusion model:

[0035] To address the problem of scarce data in extreme traffic scenarios, such as Figure 3 As shown, this invention introduces a conditional diffusion model, which establishes a mapping from Gaussian noise to the true trajectory distribution through a process of progressive noise addition and reverse denoising.

[0036] A conditional temporal diffusion framework based on the InterHub dataset is adopted, segmenting the trajectory by time. The portion before T is used as the conditional input, and the portion after T is used as the prediction target to generate high-quality, dynamically consistent extreme trajectory samples, including: sudden emergency lane change scenarios, rapid vehicle cut-in scenarios, and dynamic obstacle avoidance scenarios. The InterHub dataset is used as the training input, and the diffusion model is used for sample generation. The input and generated samples are as follows: Figure 4 As shown, the generated data serves as training samples for complex decision-making experts, improving the model's generalization ability in long-tail scenarios.

[0037] The trajectory data is processed in the same way as in step two to obtain a 128-dimensional state vector for subsequent training.

[0038] Step 4: Building the Hybrid Expert Decision-Making Module:

[0039] The Mixture of Experts (MoEs) structure consists of two reinforcement learning experts:

[0040] General decision experts: such as Figure 2 As shown, based on the SAC algorithm, the 128-dimensional state vector generated in step two is used for behavior cloning pre-training. A warm-start mechanism is used to initialize the Actor network to enhance convergence speed and stability, enabling the expert to acquire some initial reasoning ability. Using the initialized general decision expert, single-step and continuous reasoning are performed on a map with random obstacles. The results are as follows: Figure 5 As shown.

[0041] Complex Decision Expert: Based on the SAC algorithm, the 128-dimensional state vectors of extreme scenario samples generated by the diffusion model in step 3 are used for behavior cloning pre-training, and the Actor network is initialized through a warm-start mechanism, enabling the expert to acquire reasoning and decision-making capabilities under extreme traffic conditions.

[0042] The two experts each output a continuous action vector:

[0043] ;

[0044] in, This is a continuous action vector, which is a two-dimensional vector. The first dimension is... This refers to the target speed of the vehicle at the next moment, in units of... The second dimension is This refers to the angle difference between the vehicle's frontal direction at the next moment and at the present moment; it indicates a rightward deviation at the next moment. Positive, deflected to the left Negative, unit is degree.

[0045] Simultaneously, 128-dimensional features are input into a multilayer perceptron soft-gated network to obtain weights. Merge according to the following structure:

[0046] ;

[0047] in, These are all values ​​between 0 and 1, representing the weights of the two expert decisions in the overall expert decision outcome.

[0048] This enables smooth strategy switching under different traffic environments.

[0049] Step 5: Constructing the Decision Security Layer:

[0050] The fused decision-making action and the BEV diagram are simultaneously input into the safety layer. Based on the vehicle dynamics model and risk detection mechanism, the next environmental state is predicted and the risk score is calculated. .

[0051] Three types of safety indicators are defined: future collision probability detection, minimum braking distance constraint detection, and action rationality detection. The calculations for each indicator are as follows:

[0052] Future collision probability:

[0053] ;

[0054] Minimum braking distance constraint:

[0055] ;

[0056] Reasonableness of action:

[0057] ;

[0058] in, The pre-defined minimum safe distance for a vehicle, i.e., the minimum allowable distance between the vehicle and the nearest surrounding object, is expressed in units of... ; The vehicle's current speed, in units of ; The time interval between two decisions is equivalent to the time step of each decision, pre-set, and the unit is 1. ; The maximum allowable acceleration is set in advance, in units of... ; The maximum permissible angle of motion is set in advance, in degrees; and The meaning is the same as above.

[0059] The three indicators are combined using pre-defined weights to obtain a risk score:

[0060] ;

[0061] like > If a threshold is set, the weights of the hybrid expert fusion will be adjusted and the process will be replanned; otherwise, the fused decision action will be output as a control command.

[0062] in, These are all values ​​between 0 and 1, representing the three types of safety indicators in the final risk score. Weights in the calculation.

[0063] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. An expert-led autonomous driving decision-making method based on reinforcement learning and diffusion models, characterized in that, Includes the following steps: (1) Construct a data parsing module, It receives multi-dimensional bird's-eye view information input, extracts environmental perception features through convolutional neural networks and multilayer perceptrons, and obtains feature vectors containing information on obstacles, road structures, and dynamic object states; (2) Construct a random sample generation module based on the A* algorithm. By randomly generating the number and location of obstacles within a fixed map space, path planning result samples are obtained and used for training behavioral examples of the decision model. (3) Construct a sample generation module based on a diffusion model. A diffusion model is trained using a dataset of extreme traffic scenarios, and the diffusion model generates more extreme scenario samples for training, in order to supplement the scarce long-tail samples in the original dataset. (4) Construct a multi-expert decision-making module. It includes a general decision expert based on pre-trained reinforcement learning and a complex decision expert based on extreme scenarios. The outputs of the two experts are adaptively fused by a soft-gated network composed of a small multilayer perception mechanism according to the input feature vector to generate a continuous spatial motion vector. (5) Construct a decision-making security layer. Based on the BEV graph, the environmental state of the next time step is predicted, and the risk index is calculated on the continuous spatial motion vector output in step (4). When the risk index exceeds the threshold, the expert weight of the soft gating network is dynamically adjusted and the motion vector is regenerated. Otherwise, the decision result of step (4) is output.

2. The expert autonomous driving decision-making method based on reinforcement learning and diffusion model according to claim 1, characterized in that: The general decision expert mentioned in step (4) is trained using the SAC algorithm. First, the path planning samples generated in step (2) are used for behavior cloning pre-training, and the Actor network in the SAC algorithm is initialized through a soft-start mechanism to enhance stability and convergence speed.

3. The expert autonomous driving decision-making method based on reinforcement learning and diffusion model according to claim 1, characterized in that: The complex decision expert described in step (4) is trained using the extreme scenario samples generated by the diffusion model in step (3) based on the SAC algorithm to optimize the strategy performance in complex environments.

4. The expert autonomous driving decision-making method based on reinforcement learning and diffusion model according to claim 1, characterized in that: The soft-gated network in step (4) adopts the structure of a multilayer perceptron, calculates the weights of two experts based on the input feature vector, and outputs the fused motion vector in a continuous manner to achieve smooth scene transition and improve decision consistency.

5. The expert autonomous driving decision-making method based on reinforcement learning and diffusion model according to claim 1, characterized in that: The risk indicators mentioned in step (5) include constraint parameters such as collision probability, road deviation probability and minimum safe distance. If the risk value is greater than the set threshold, the weights of the general expert and the complex expert are adjusted until the risk value is lower than the threshold.

6. The expert autonomous driving decision-making method based on reinforcement learning and diffusion model according to claim 1, characterized in that: The extreme scenario samples generated by the diffusion model in step (3) include traffic accident scenarios, vehicle rapid entry scenarios, and dynamic obstacle emergency avoidance scenarios.

7. The expert autonomous driving decision-making method based on reinforcement learning and diffusion model according to claim 1, characterized in that: The continuous spatial motion vector output in step (4) includes the target velocity and the turning angle.

Images