Automatic driving decision-making method for unsignalized intersection based on DSAC and self-attention
By introducing quantile reward modeling and self-attention networks, the problems of differentiability optimization of discrete action space and multi-vehicle interaction modeling in autonomous driving at unsignalized intersections are solved, achieving stable and efficient decision-making in complex traffic environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JILIN UNIVERSITY
- Filing Date
- 2026-04-10
- Publication Date
- 2026-07-07
AI Technical Summary
Existing reinforcement learning methods struggle to achieve differentiability in the discrete action space for autonomous driving decision-making at unsignalized intersections, failing to effectively characterize the uncertainty of reward distribution in multi-vehicle interactions. Furthermore, traditional neural networks have limitations in extracting spatiotemporal interaction features of multiple vehicles, leading to instability in the learning process and policy oscillations.
We employ a method based on DSAC and self-attention mechanism, which introduces quantile reward modeling and categorical distribution to achieve end-to-end optimization of the discrete action space. We also combine a self-attention network to capture the spatiotemporal interaction features of multiple vehicles and construct a distributed value estimation model to generate stable decision-making strategies.
It improves the accuracy and stability of return estimation, enhances decision-making accuracy and robustness in complex traffic scenarios, reduces collision rate and policy fluctuation, and achieves safer and more efficient autonomous driving decision-making.
Smart Images
Figure CN122347876A_ABST
Abstract
Description
Technical Field
[0002] This invention belongs to the field of autonomous driving decision-making technology for unsignalized intersections, specifically involving an autonomous driving decision-making method for unsignalized intersections based on the Distributed Soft Actor Critic (DSAC) algorithm and a self-attention mechanism. Background Technology
[0003] In high-risk, complex traffic scenarios such as unsignaled intersections, how autonomous vehicles can make safe, real-time, and efficient traffic decisions in dynamic multi-vehicle environments remains a core challenge in the field of intelligent driving. Especially in such traffic environments, the behavior of multiple traffic participants is highly uncertain and their interactions are complex. Traditional rule-driven or optimization methods are inadequate to cope with this complexity and cannot simultaneously ensure safety, real-time performance, and traffic efficiency.
[0004] Existing research techniques are mainly divided into three categories: rule-based and optimization-based methods, game theory-based interaction modeling methods, and reinforcement learning-based policy learning methods. Among them, the SAC algorithm, due to its framework based on maximum entropy theory, has high sampling efficiency and policy stability, and is widely used in control tasks in continuous action spaces. However, the application of existing reinforcement learning methods in complex traffic scenarios still has several significant limitations: on the one hand, SAC is originally designed for continuous action spaces, making it difficult to achieve differentiability optimization in discrete action spaces, resulting in limited gradient propagation and unstable learning processes; on the other hand, traditional Q-networks can only model expected rewards and cannot effectively characterize the uncertainty of reward distribution in the environment, resulting in an inability to accurately reflect policy differences under different risk levels in high-dimensional, multi-vehicle interaction scenarios; in addition, existing neural network structures have limitations in extracting spatiotemporal interaction features of multiple vehicles, making it difficult to guarantee policy stability and generalization ability in complex decision-making environments.
[0005] To address these issues, Distributed Reinforcement Learning (DRL) has gained significant attention in recent years. This method effectively improves the stability and robustness of the algorithm by learning the distribution and quantile function of rewards, enhancing its ability to model uncertainty and avoiding overestimation of rewards and policy oscillations that may occur in traditional methods. However, existing distributed algorithms are mostly designed for low-dimensional or continuous control tasks and have not yet provided systematic solutions for differentiable optimization problems in discrete action spaces and multi-agent interaction modeling. Furthermore, modeling high-dimensional state features and multi-vehicle relationships in traffic scenarios remains a challenge, limiting the practical application of these methods in complex traffic decision-making. Summary of the Invention
[0006] To address the shortcomings of existing methods in discrete action space optimization, multi-vehicle interaction modeling, and reward uncertainty characterization, this invention proposes an autonomous driving decision-making method for unsignaled intersections based on DSAC and an ego-attention mechanism. This method effectively improves the accuracy and stability of reward estimation by introducing quantile-based distributional Q-functions, and achieves end-to-end optimization of the discrete action space by combining categorical distributions and a softmax function. Furthermore, this invention introduces an ego-attention network to model the state characteristics of autonomous vehicles, thereby effectively capturing the spatiotemporal correlations between multiple vehicles, particularly improving the accuracy and robustness of decision-making in complex unsignaled intersection tasks.
[0007] The objective of this invention is achieved through the following technical solution:
[0008] This invention provides an autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention, comprising the following steps:
[0009] S1. Environment Setup and Feature Preprocessing: Construct a simulation environment for an unsignalized intersection, obtain the state information of the vehicle and surrounding vehicles, define the vehicle model, state space, action space, and reward function, and normalize and structure the state information to form a standardized feature tensor.
[0010] S2. Self-attention interaction modeling: The standardized feature tensor is input into the self-attention network, with the self-vehicle features as the query and the surrounding vehicle features as the key and value. The interaction weights between the self-vehicle and each surrounding vehicle are calculated through a multi-head self-attention mechanism. After weighted aggregation, the high-dimensional feature vector of the self-vehicle that integrates the spatiotemporal interaction information of multiple vehicles is output.
[0011] S3. Distributed Value Estimation and Strategy Generation: Construct a decision-making and planning model based on distributed value estimation. Take the high-dimensional feature vector of the autonomous vehicle as input, and achieve end-to-end differentiability optimization in the discrete longitudinal action space through the Categorical distribution and softmax function. Combine quantile reward modeling and maximum entropy mechanism to output the passage decision strategy of autonomous vehicles in the scenario of unsignalized intersections.
[0012] Further, in step S1, the vehicle model definition includes:
[0013] ;
[0014] ;
[0015] ;
[0016] ;
[0017] ;
[0018] ;
[0019] ;
[0020] In the formula, The target lateral velocity; Position control gain; This is the lateral distance between the vehicle and the center line of the target lane. This represents the change in heading angle; The vehicle's current longitudinal speed; This refers to the lane heading angle; The desired heading angle; This is the vehicle's current actual heading angle; For heading angle change rate command; For heading control gain; The steering angle of the front wheels; This is the distance between the center of the front wheel and the vehicle's center of gravity. The calculated longitudinal acceleration of the vehicle; It is the maximum acceleration; It is the expected speed; It is a constant acceleration parameter; It is the actual distance between the following vehicle and the vehicle in front; This is the desired dynamic safety distance; It is the minimum relative distance between two vehicles in the same lane; It is the time interval required to achieve the security objective; It is the relative speed between the vehicle following and the vehicle in front; It's deceleration.
[0021] Furthermore, in step S1, the state space is represented as follows:
[0022] ;
[0023] in, This indicates whether the vehicle exists in a binary state within the environment; , This represents the vehicle's coordinates in the plane; , Indicates that the vehicle is in and Velocity component in the direction; , The cosine and sine values represent the vehicle's heading angle;
[0024] The action space is a discrete longitudinal control space, including three control actions: acceleration, deceleration, and stopping.
[0025] Furthermore, in step S1, the total reward The calculation formula is:
[0026] ;
[0027] in, This is the penalty value when a vehicle collides. The reward value when the vehicle reaches the target location; The bonus value for vehicle speed; This is the vehicle time penalty value.
[0028] Further, step S2 includes:
[0029] S21. Embedding of input features
[0030] The original features of the vehicle and surrounding vehicles are mapped to a high-dimensional space through a multilayer perceptron; the features of each vehicle are embedded and transformed into a unified high-dimensional feature representation.
[0031] S22. Query-Key-Value Calculation
[0032] Based on the embedded features of the input, the model calculates the query, key, and value using three weight matrices:
[0033] Query: The query characteristics of a vehicle are represented as follows ;
[0034] Key: Key characteristics of surrounding vehicles, represented as ;
[0035] Value: The value characteristic of surrounding vehicles, represented as ;
[0036] in, It is the input embedding feature matrix; , , These are weight matrices prepared for querying, key-value calculation, and other operations, respectively.
[0037] S23. Calculate the attention matrix
[0038] Calculate the attention matrix using the dot product of the query and the key. :
[0039] ;
[0040] in, It is the dimension of the embedded features. It is the dot product result of the query and the key, through The operation yields the correlation between each pair of query keys;
[0041] S24. Feature Aggregation
[0042] Using the calculated attention matrix The values are weighted and summed to obtain the weighted feature representation:
[0043] ;
[0044] S25. Calculation of Output Features and Multi-Head Self-Attention
[0045] A multi-head self-attention mechanism is adopted, in which each head independently calculates attention and outputs the corresponding feature representation;
[0046] Combine the outputs of all the heads together;
[0047] Through a fully connected layer The concatenated features are mapped to obtain the final high-dimensional feature vector of the vehicle that integrates spatiotemporal interaction information from multiple vehicles:
[0048]
[0049] in, Indicates the number of heads. Indicates the first Output of individual heads.
[0050] Further, step S3 includes:
[0051] S31. Initialization Phase
[0052] Initialize the policy network separately and target policy network ;
[0053] Initialize two Q networks respectively, i.e. and Two Q-networks are used to calculate the distributed Q-value for each state-action pair. The Q-network calculation formula is as follows:
[0054] ;
[0055] in, This indicates the current state and includes all information in the environment; It is in state The action of making a selection; It is the feedforward process of the Q network, which outputs the Q value of each state-action pair;
[0056] S32. Environmental Interaction and Sample Collection Phase
[0057] By interacting with the environment, the vehicle obtains its current state. State features are generated by encoding through a self-attention network. The autonomous vehicle uses a policy network based on a categorical distribution to calculate the selection probability of each discrete action. The specific formula for calculating the action probability distribution is as follows:
[0058] ;
[0059] in, It is the probability distribution of classified actions output by the policy network; This is the unnormalized bias of the neural network output;
[0060] The vehicle samples based on this probability distribution to obtain specific discrete actions. ;
[0061] S33. Adaptive Temperature Regulation
[0062] Each time the vehicle takes action, the temperature parameter is adaptively adjusted based on the number of training iterations, as shown in the following formula:
[0063] ;
[0064] in, and Temperature parameters The minimum and maximum values; It is the adjustment coefficient; It is the attenuation factor; It represents the number of training iterations;
[0065] S34. Calculate Quantile Returns
[0066] The vehicle operates according to the current strategy network. Perform Q-network calculations to obtain the quantile reward value for each state-action pair. and ;
[0067] Calculate the target Q value The formula is as follows:
[0068]
[0069] in, It's an instant reward; It is a discount factor; It is a termination flag, indicating whether the current state is terminated; It is the temperature coefficient; The next state The probability of choosing an action;
[0070] S35.Q Network Update
[0071] use The loss function is achieved by minimizing the current Q-network output. and With the target Q value The differences between them are used to optimize the parameters of the Q network. The loss function is as follows:
[0072]
[0073] in, For the target Q value, This is the current prediction value of the Q network;
[0074] S36. Policy Network Parameter Update
[0075] The parameters of the policy network are updated by maximizing the objective function of policy optimization. The objective function for strategy optimization is:
[0076]
[0077] in, For temperature parameters; It is the minimum Q-value output by the Q-network; It is in state Select action The probability of the strategy;
[0078] S37. Target network soft update
[0079] The update formula for the target network is:
[0080]
[0081]
[0082] in, and These are the parameters of the current Q network and the policy network, respectively; and These are the parameters of the corresponding target Q-network and target policy network, respectively; It is a soft update coefficient that controls the update rate;
[0083] S38. Iterative Training Loop
[0084] Repeat steps S32 to S37 until the training epochs or convergence criteria are met, and output the final policy. .
[0085] The present invention has the following beneficial effects:
[0086] This invention introduces quantile-based distributional QFunction to effectively characterize the distributional uncertainty of returns in the environment, improve the accuracy and stability of return estimation, avoid the overestimation of returns and policy oscillation problems that may occur in traditional Q-networks, and thus enhance the robustness of the algorithm in complex dynamic environments.
[0087] This invention incorporates the Ego-Attention self-attention network, which can effectively capture the spatiotemporal interaction relationships between vehicles. By modeling the spatiotemporal features between multiple vehicles, this invention can provide a more stable and secure decision-making strategy in complex multi-vehicle interaction environments, enhancing the algorithm's generalization ability and adaptability.
[0088] By employing a discrete action space differentiable optimization strategy based on Categorical distribution and softmax processing, this invention can achieve end-to-end optimization, ensure effective gradient propagation, stabilize the reinforcement learning training process, and avoid the learning instability problem caused by the inability to optimize the discrete action space in traditional methods.
[0089] Through these innovations, this invention enables more accurate, efficient, and stable decision-making in complex traffic scenarios, especially in high-risk environments such as unsignaled intersections, thereby improving the performance and safety of autonomous driving systems.
[0090] Experimental results show that, compared with the traditional SAC method, the present invention significantly improves training efficiency and decision safety in autonomous driving intersection decision-making tasks, and significantly reduces collision rate and policy fluctuation. Attached Figure Description
[0091] Figure 1 This is an overall flowchart of the autonomous driving decision-making method for unsignaled intersections based on DSAC and self-attention, as described in an embodiment of the present invention.
[0092] Figure 2 This is a flowchart illustrating the specific steps of an embodiment of the present invention;
[0093] Figure 3 This is a schematic diagram of a signalless intersection constructed in an embodiment of the present invention;
[0094] Figure 4 A comparison chart of collision rates from three algorithms in a comparative experiment;
[0095] Figure 5 A comparison chart of rewards from three algorithms in a comparative experiment;
[0096] Figure 6 The graph shows a speed comparison of the three algorithms in a comparative experiment. Detailed Implementation
[0097] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments:
[0098] This embodiment provides an autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention. By introducing quantile reward modeling, a self-attention interaction mechanism, and a differentiable optimization strategy for the discrete action space, it achieves traffic decision-making and planning for autonomous vehicles in multi-vehicle interaction scenarios. The entire technical solution includes three main stages: environment setup, self-attention interaction modeling, and decision-making and planning based on distributed value estimation. The specific steps are as follows: Figure 1 , Figure 2 As shown:
[0099] S1. Environment Setup and Feature Preprocessing: Construct a simulation environment for an unsignalized intersection, acquire the state information of the vehicle and surrounding vehicles, define the vehicle model, state space, action space, and reward function, and normalize and structure the state information to form a standardized feature tensor, providing standardized and usable input data for subsequent perception and decision-making modules.
[0100] S11. Road Construction
[0101] In this embodiment, the environment is built based on the highway-env library and adopts a four-way intersection design. The entire road network consists of multiple lanes, including straight lanes, left-turn lanes, and right-turn lanes, forming a complete intersection model. In the current scenario, this invention assumes that under intelligent connected conditions, autonomous vehicles (autonomous vehicles) can accurately perceive key information such as their own position and speed, as well as that of surrounding vehicles. At the intersection, horizontal straight and right-turn lanes have the highest priority, followed by horizontal left-turn lanes, then perpendicular straight and right-turn lanes, with the perpendicular left-turn lane having the lowest priority. This priority setting ensures traffic safety and efficiency at the intersection while optimizing the interaction between different lanes. The design of each lane considers traffic flow and vehicle turning radius in different directions, ensuring traffic safety and smooth flow. The constructed unsignalized intersection is shown below. Figure 3 As shown.
[0102] S12. Vehicle Model
[0103] In the simulation environment of this embodiment, the behavior of the vehicle and surrounding vehicles is managed by a hierarchical control architecture. For the vehicle, the upper-level control uses a reinforcement learning decision-making strategy, primarily responsible for managing longitudinal behavior such as acceleration, deceleration, and stopping; while the lower-level control employs a lateral tracking controller to ensure the vehicle travels along the planned trajectory. For surrounding vehicles, an Intelligent Driving Model (IDM) is used as the upper-level control strategy, responsible for making decisions regarding their longitudinal behavior, while the lower level also uses a lateral tracking controller to keep the vehicles traveling along the predetermined path. Since lane changes are not allowed at intersections, the travel route of each vehicle is planned before entering the intersection, and the lateral tracking controller is responsible for ensuring that the vehicles strictly follow the planned path.
[0104] The lateral tracking controller involves position control and heading control, and its expression is as follows:
[0105]
[0106]
[0107]
[0108]
[0109]
[0110] in, For the target lateral velocity, For position control gain, This is the lateral distance between the vehicle and the center line of the target lane. The change in heading angle. The vehicle's current longitudinal speed, The lane heading angle, For the desired heading angle, This is the vehicle's current actual heading angle. This is the heading angle change rate command (or desired yaw rate). For heading control gain, For the front wheel steering angle, This is the distance between the center of the front wheel and the center of gravity of the vehicle.
[0111] The longitudinal acceleration of surrounding vehicles is controlled by the IDM (Integrated Vehicle Management) system. Its control expression is as follows:
[0112]
[0113]
[0114] in, To calculate the longitudinal acceleration of the vehicle, It is the maximum acceleration. It is the vehicle's current longitudinal speed. It is the expected speed. It is a constant acceleration parameter. It is the actual distance between the following vehicle and the vehicle in front. It is the desired dynamic safety distance. It is the minimum relative distance between two vehicles in the same lane. It is the time interval required for the security objective. It is the relative speed between the vehicle following and the vehicle in front. It is the deceleration based on comfort requirements.
[0115] S13. State Space and Action Space
[0116] In this embodiment, state information is described by the vehicle's kinematic characteristics, specifically including each vehicle's spatial coordinates, velocity, velocity components, and heading angle. The state space design ensures that it reflects the vehicle's dynamic behavior and its interaction with the surrounding environment. The state space can be represented as:
[0117]
[0118] in, This indicates whether the vehicle exists in a binary state (0 or 1) in the environment. , This represents the vehicle's coordinates in the plane. , Indicates that the vehicle is in and velocity components in the direction, , These represent the cosine and sine values of the vehicle's heading angle.
[0119] Based on the environmental configuration, the action space is a discrete longitudinal control space, including only three control actions: acceleration, deceleration, and stopping. Since the intersection environment does not allow vehicles to change lanes, lateral control remains static, meaning vehicles can only move along the lane and do not perform lateral lane changes.
[0120] S14. Reward Function and Termination Condition
[0121] To ensure that autonomous vehicles can complete their tasks safely and efficiently at intersections, the reward function is designed considering multiple factors, primarily including safety, reaching the destination, and speed. The specific design is as follows:
[0122] When a vehicle is involved in a collision, it will be severely penalized. The penalty value is:
[0123]
[0124] A substantial reward will be given when the vehicle successfully reaches the target location. The reward value is:
[0125]
[0126] Vehicles will receive a reward if their speed stays within a certain range, encouraging them to maximize their speed while avoiding excessive speeds. The reward value is calculated using the following formula:
[0127]
[0128] in and These are the minimum and maximum speeds, respectively.
[0129] To encourage vehicles to reach the destination as quickly as possible, a time penalty will be deducted for each step:
[0130]
[0131] Total Rewards The calculation formula is:
[0132]
[0133] Each round in the environment will terminate immediately when any of the following conditions are met: a vehicle collision occurs; the vehicle reaches the target; or the maximum value of the simulation step is reached. These termination conditions ensure that each simulation conforms to the constraints of a real traffic environment and can effectively evaluate vehicle behavior.
[0134] S15. Feature Preprocessing and Encoding
[0135] For feature preprocessing, the range of each input feature is limited before it is passed to the neural network. For example, location features ( , The velocity characteristics are limited to the range of [-100, 100]. , The dimensions are then restricted to the range [-20, 20]. These range restrictions help the model maintain numerical consistency during training and avoid computational instability caused by differences in feature dimensions. The feature dimensions are then rearranged according to the network input requirements to form a standardized feature tensor in the form of "batch × number of vehicles × number of features".
[0136] S2. Self-attention interaction modeling: The standardized feature tensor is input into the self-attention (Ego-Attention) network. The self-vehicle features are used as queries and the features of surrounding vehicles are used as keys and values. The interaction weights between the self-vehicle and each surrounding vehicle are calculated through a multi-head self-attention mechanism. After weighted aggregation, the high-dimensional feature vector of the self-vehicle that integrates the spatiotemporal interaction information of multiple vehicles is output.
[0137] S21. Embedding of input features
[0138] The original features of the vehicle and surrounding vehicles (such as position, speed, heading angle, etc.) are mapped to a high-dimensional space using a multilayer perceptron (MLP). After embedding, the features of each vehicle become a unified high-dimensional feature representation, which facilitates subsequent computation and aggregation.
[0139] S22. Query-Key-Value Calculation
[0140] Based on the embedded features of the input, the model calculates the query, key, and value using three weight matrices:
[0141] Query ( The query characteristics of a vehicle are represented as follows: ,
[0142] key( ): Key features of surrounding vehicles, represented as ,
[0143] value( ): Value characteristics of surrounding vehicles, represented as .
[0144] in, It is the input embedding feature matrix; , , These are weight matrices prepared for querying, key calculation, and value calculation, respectively.
[0145] S23. Calculate the attention matrix
[0146] Calculate the attention matrix using the dot product of the query and the key. This is used to represent the relationship between a vehicle and surrounding vehicles. The calculation formula is as follows:
[0147]
[0148] in, It is the dimension of the embedded features. It is the dot product result of the query and the key, through The operation yields the correlation between each pair of query keys.
[0149] S24. Feature Aggregation
[0150] Using the calculated attention matrix value ( We perform a weighted summation to obtain the weighted feature representation:
[0151]
[0152] This operation represents the aggregation of vehicle features based on attention weights to obtain updated features for each vehicle.
[0153] S25. Calculation of Output Features and Multi-Head Self-Attention
[0154] A multi-head self-attention mechanism is employed, where each head independently computes attention and outputs its corresponding feature representation. Then, the outputs of all heads are concatenated to form a richer feature representation. This is achieved through a fully connected layer (…). The concatenated features are mapped to obtain the final high-dimensional feature vector of the self-vehicle, which integrates spatiotemporal interaction information from multiple vehicles.
[0155]
[0156] in, Indicates the number of heads. Indicates the first Output of individual heads.
[0157] The self-attention mechanism module effectively captures the complex interactions between the vehicle and other vehicles, providing richer representations for the autonomous driving decision-making module. Through the self-attention network (Ego-Attention), the model adaptively learns the interactions between vehicles, particularly the influence of the vehicle on other surrounding vehicles. By introducing a multi-head self-attention mechanism, the model can simultaneously focus on different aspects of vehicle interaction information, providing global and local vehicle dynamics information for decision-making, thus enhancing the system's perception and decision-making capabilities in complex traffic environments.
[0158] S3. Distributed Value Estimation and Strategy Generation: Construct a decision-making and planning model based on Distributed Soft Actor-Critic (DSAC). Using the high-dimensional feature vector of the autonomous vehicle as input, end-to-end differentiable optimization is achieved in the discrete longitudinal action space through the Categorical distribution and softmax function. Combined with quantile reward modeling and maximum entropy mechanism, the optimal decision strategy for autonomous vehicle communication at unsignalized intersections is output, enabling autonomous vehicles to obtain a safer, more stable, and more efficient passage strategy in unsignalized intersection environments.
[0159] S31. Initialization Phase
[0160] First, initialize the policy networks respectively. and target policy network This provides initial goals for the subsequent learning process;
[0161] Next, two Q-networks are initialized respectively, namely and These two Q-networks are used to calculate the distributed Q-value for each state-action pair. The Q-network calculation formula is:
[0162]
[0163] in, It indicates the current state and includes all information in the environment (such as position, speed, direction, etc.). It is in state The action to choose from, It is the feedforward process of the Q network, which outputs the Q value of each state-action pair;
[0164] Subsequently, the Replay Buffer is initialized to store the experiential data of the autonomous vehicle during its environmental interactions; in addition, hyperparameters for reinforcement learning, such as the learning rate, are set. Discount factor And the number of quantiles, such as num_quantiles.
[0165] S32. Environmental Interaction and Sample Collection Phase
[0166] The current state is obtained by the autonomous vehicle (self-driving vehicle) interacting with the environment. State features are generated by encoding through an Ego-Attention network. Autonomous vehicles use a policy network based on a categorical distribution to calculate the selection probability of each discrete action. The specific formula for calculating the action probability distribution is as follows:
[0167]
[0168] in, It is the probability distribution of classified actions output by the policy network. This represents the unnormalized bias (logits) output by the neural network. After processing with softmax, the probability value of each available discrete action is obtained. The autonomous vehicle samples according to this probability distribution to obtain the specific discrete action. (Such as acceleration, deceleration, and holding). This interaction information (including state, action, reward, next state, etc.) will be stored in the Replay Buffer for use in subsequent training.
[0169] S33. Adaptive Temperature Regulation
[0170] To enhance the learning effect of the strategy, an adaptive temperature adjustment mechanism is introduced. Temperature parameter This controls the balance between exploring and utilizing the strategy. Whenever the autonomous vehicle takes action, the temperature parameter automatically adjusts according to changes in the environment; that is, it adaptively adjusts the temperature parameter based on the number of training iterations, as shown in the following formula:
[0171]
[0172] in, and Temperature parameters The minimum and maximum values, It is the adjustment coefficient. It is the attenuation factor. This refers to the number of training iterations. Temperature parameters are adjusted adaptively. Autonomous vehicles can adjust their exploration and utilization strategies according to the complexity of the environment to improve the stability and robustness of the model.
[0173] S34. Calculate Quantile Returns
[0174] Autonomous vehicles according to the current policy network Perform Q-network calculations to obtain the quantile reward value for each state-action pair. and Then, calculate the target Q value. The target Q value is calculated based on the environmental transition model and the reward signal, as shown in the following formula:
[0175]
[0176] in, It's an instant reward. It is a discount factor. It is a termination flag, indicating whether the current state is terminated. It's a temperature coefficient that adjusts the level of exploration. The next state The probability of choosing an action.
[0177] S35.Q Network Update
[0178] use The loss function is achieved by minimizing the current Q-network output. and With the target Q value The differences between them are used to optimize the parameters of the Q network. The loss function is as follows:
[0179]
[0180] in, For the target Q value, This is the current prediction value of the Q network.
[0181] To improve training stability, use The loss function is used to calculate the error. The loss function is a combination of L1 and L2 loss characteristics, which balances sensitivity to outliers by controlling the magnitude of the error. Specifically, The loss function can use L2 loss (i.e., squared error) when the error is small (i.e., close to the target value), and L1 loss (i.e., absolute error) when the error is large. The purpose of this is to reduce the excessive penalty for outliers while maintaining smoothness at smaller errors.
[0182] S36. Policy Network Parameter Update
[0183] The parameters of the policy network are updated by maximizing the objective function of policy optimization. In the DSAC method based on distributed value estimation, the goal of policy optimization is not only to maximize the expected reward but also to increase exploratory nature and prevent autonomous vehicles from getting stuck in local optima. Specifically, a maximum entropy policy is used when updating the policy, ensuring that action selection depends not only on the reward value but also on the entropy (uncertainty) of the action. This policy introduces a maximum entropy mechanism to improve the exploratory nature of the policy and avoid premature convergence. Therefore, the objective function of policy optimization is:
[0184]
[0185] in, The temperature parameters mentioned above, It is the minimum Q-value output by the Q-network. It is in state Select action The strategy probability.
[0186] S37. Target network soft update
[0187] After each training period, a soft update of the target network is performed. The update formula for the target network is:
[0188]
[0189]
[0190] in, and These are the parameters of the current Q network and policy network, respectively. and These are the parameters of the corresponding target Q-network and target policy network, respectively. It is the soft update coefficient, which controls the update rate.
[0191] S38. Alternate Training Loop
[0192] Repeat steps S32-S37 until the required number of training epochs or the convergence condition is met, then output the final policy. .
[0193] By combining the Categorical distribution and softmax processing, the DSAC module achieves differentiable optimization in the discrete action space and improves decision-making efficiency through quantile reward optimization of the Q-network. The entire training process, through maximum entropy policy optimization and distributed Q-network updates, enables autonomous vehicles to make efficient decisions in complex traffic environments. Through this optimization method, DSAC ensures an efficient and stable learning process, improving the reliability and performance of autonomous driving decision-making systems.
[0194] This embodiment combines a distributed soft actor critic algorithm (DSAC) with a self-attention mechanism to address decision-making problems in complex traffic environments such as unsignaled intersections. By introducing a distributed Q-network model, the reward distribution in multi-vehicle environments can be effectively modeled, avoiding the overestimation problem that may occur with traditional Q-learning methods. Simultaneously, the self-attention mechanism, through an Ego-Attention network, models the spatiotemporal relationships between multiple vehicles, adaptively capturing the interaction features between vehicles and improving the model's perception capabilities in complex traffic interaction scenarios. Combining these two technologies, this invention not only improves the accuracy of vehicle behavior prediction but also significantly enhances the model's stability and robustness in dynamic and uncertain traffic scenarios.
[0195] In the decision-making process, this embodiment employs a discrete action space differentiable sampling strategy based on a categorical distribution, addressing the problem that traditional reinforcement learning methods cannot achieve differentiable optimization in a discrete action space. Combined with a reinforcement learning optimization method based on quantile rewards, decision-making efficiency and safety are further improved. Furthermore, the introduction of a policy network and a maximum entropy mechanism optimizes the decision-making capabilities of autonomous vehicles in complex traffic environments, ensuring an efficient and stable learning process. Ultimately, this invention can provide efficient and stable decision support for intelligent driving systems, demonstrating significant advantages, especially in high-risk and high-complexity scenarios such as unsignaled intersections.
[0196] The following are the comparative experimental results of this invention and existing algorithms:
[0197] Taking left-turn decision training at an unsignalized intersection as an example, this patent conducts a comparative experiment based on three algorithms (SAC algorithm with self-attention mechanism, DSAC algorithm without self-attention mechanism, and DSAC algorithm with self-attention mechanism). The following shows the trends of collision rate, reward value, and speed of the three algorithms during the training process.
[0198] like Figures 4 to 6 As shown, all three algorithms exhibited high collision rates in the early stages of training. This is likely because, in the initial phase, the algorithms had not yet converged, and the exploration phase was relatively aggressive, leading to a high number of collisions. However, as training progressed, the DSAC algorithm with self-attention (blue curve) showed a significant advantage in the later stages of training, with its collision rate continuously decreasing and eventually stabilizing at the lowest level, demonstrating its efficient decision-making ability and better stability in complex traffic environments. In contrast, the DSAC algorithm without self-attention (red curve) and the SAC algorithm with self-attention (green curve) had relatively high collision rates in the later stages of training, and their collision rates fluctuated significantly during training, indicating that these algorithms had poor stability during training when facing complex environments. Although their collision rates were relatively similar in the initial stages, as training progressed, the DSAC algorithm with self-attention gradually demonstrated stronger robustness and higher decision-making accuracy.
[0199] In terms of reward value trends, the DSAC algorithm with self-attention mechanism significantly outperforms the other two algorithms. This algorithm effectively improves the estimation accuracy of rewards, especially in complex environments, where the policy can converge to a higher reward level more quickly. Conversely, without self-attention mechanisms, the SAC and DSAC algorithms show slower reward value increases and more fluctuations during training, indicating that the policy may oscillate during convergence.
[0200] The speed charts show that the speeds of the three algorithms tend to stabilize in the later stages of training, with the DSAC algorithm, which has a self-attention mechanism, reaching a stable state in a shorter time and exhibiting more stable speed control. The SAC algorithm, on the other hand, shows greater fluctuations in the early stages of training, demonstrating weaker speed control capabilities.
Claims
1. A decision-making method for automated driving at unsignalized intersections based on DSAC and self-attention, characterized in that, Includes the following steps: S1. Environment Setup and Feature Preprocessing: Construct a simulation environment for an unsignalized intersection, obtain the state information of the vehicle and surrounding vehicles, define the vehicle model, state space, action space, and reward function, and normalize and structure the state information to form a standardized feature tensor. S2. Self-attention interaction modeling: The standardized feature tensor is input into the self-attention network, with the self-vehicle features as the query and the surrounding vehicle features as the key and value. The interaction weights between the self-vehicle and each surrounding vehicle are calculated through a multi-head self-attention mechanism. After weighted aggregation, the high-dimensional feature vector of the self-vehicle that integrates the spatiotemporal interaction information of multiple vehicles is output. S3. Distributed Value Estimation and Strategy Generation: Construct a decision-making and planning model based on distributed value estimation. Take the high-dimensional feature vector of the autonomous vehicle as input, and achieve end-to-end differentiability optimization in the discrete longitudinal action space through the Categorical distribution and softmax function. Combine quantile reward modeling and maximum entropy mechanism to output the passage decision strategy of autonomous vehicles in the scenario of unsignalized intersections.
2. The autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention as described in claim 1, characterized in that, In step S1, the vehicle model definition includes: ； ； ； ； ； ； ； In the formula, The target lateral velocity; Position control gain; This is the lateral distance between the vehicle and the center line of the target lane. This represents the change in heading angle; The vehicle's current longitudinal speed; This refers to the lane heading angle; The desired heading angle; This is the vehicle's current actual heading angle; For heading angle change rate command; For heading control gain; The steering angle of the front wheels; This is the distance between the center of the front wheel and the vehicle's center of gravity. The calculated longitudinal acceleration of the vehicle; It is the maximum acceleration; It is the expected speed; It is a constant acceleration parameter; It is the actual distance between the following vehicle and the vehicle in front; This is the desired dynamic safety distance; It is the minimum relative distance between two vehicles in the same lane; It is the time interval required to achieve the security objective; It is the relative speed between the vehicle following and the vehicle in front; It's deceleration.
3. The autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention as described in claim 1, characterized in that, In step S1, the state space is represented as follows: ； in, This indicates whether the vehicle exists in a binary state within the environment; , Represents the vehicle in a plane coordinate; , Indicates that the vehicle is in and Velocity component in the direction; , Indicates the vehicle's heading angle The cosine and sine values; The action space is a discrete longitudinal control space, including three control actions: acceleration, deceleration, and stopping.
4. The autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention as described in claim 1, characterized in that, In step S1, the total reward The calculation formula is: ； in, This is the penalty value when a vehicle collides. The reward value when the vehicle reaches the target location; The bonus value for vehicle speed; This is the vehicle time penalty value.
5. The autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention as described in claim 1, characterized in that, Step S2 includes: S21. Embedding of input features The original features of the vehicle and surrounding vehicles are mapped to a high-dimensional space through a multilayer perceptron; the features of each vehicle are embedded and transformed into a unified high-dimensional feature representation. S22. Query-Key-Value Calculation Based on the embedded features of the input, the model calculates the query, key, and value using three weight matrices: Query: The query characteristics of a vehicle are represented as follows , Key: Key characteristics of surrounding vehicles, represented as , Value: The value characteristic of surrounding vehicles, represented as ; in, It is the input embedding feature matrix; , , These are weight matrices prepared for querying, key-value calculation, and other operations, respectively. S23. Calculate the attention matrix Calculate the attention matrix using the dot product of the query and the key. : ； in, It is the dimension of the embedded features. It is the dot product result of the query and the key, through The operation yields the correlation between each pair of query keys; S24. Feature Aggregation Using the calculated attention matrix The values are weighted and summed to obtain the weighted feature representation: ； S25. Calculation of Output Features and Multi-Head Self-Attention A multi-head self-attention mechanism is adopted, in which each head independently calculates attention and outputs the corresponding feature representation; Combine the outputs of all the heads together; Through a fully connected layer The concatenated features are mapped to obtain the final high-dimensional feature vector of the vehicle that integrates spatiotemporal interaction information from multiple vehicles: ； in, Indicates the number of heads. Indicates the first Output of individual heads.
6. The autonomous driving decision-making method for unsignalized intersections based on DSAC and self-attention as described in claim 1, characterized in that, Step S3 includes: S31. Initialization Phase Initialize the policy network separately and target policy network ; Initialize two Q networks respectively, i.e. and Two Q-networks are used to calculate the distributed Q-value for each state-action pair. The Q-network calculation formula is as follows: ； in, This indicates the current state and includes all information in the environment; It is in state The action of making a selection; It is the feedforward process of the Q network, which outputs the Q value of each state-action pair; S32. Environmental Interaction and Sample Collection Phase By interacting with the environment, the vehicle obtains its current state. State features are generated by encoding through a self-attention network. The autonomous vehicle uses a policy network based on a categorical distribution to calculate the selection probability of each discrete action. The specific formula for calculating the action probability distribution is as follows: ； in, It is the probability distribution of classified actions output by the policy network; This is the unnormalized bias of the neural network output; The vehicle samples based on this probability distribution to obtain specific discrete actions. ; S33. Adaptive Temperature Regulation Each time the vehicle takes action, the temperature parameter is adaptively adjusted based on the number of training iterations, as shown in the following formula: ； in, and Temperature parameters The minimum and maximum values; It is the adjustment coefficient; It is the attenuation factor; It represents the number of training iterations; S34. Calculate Quantile Returns The vehicle operates according to the current strategy network. Perform Q-network calculations to obtain the quantile reward value for each state-action pair. and ; Calculate the target Q value The formula is as follows: ； in, It's an instant reward; It is a discount factor; It is a termination flag, indicating whether the current state is terminated; It is the temperature coefficient; The next state The probability of choosing an action; S35.Q Network Update use The loss function is achieved by minimizing the current Q-network output. and With the target Q value The differences between them are used to optimize the parameters of the Q network. The loss function is as follows: ； in, For the target Q value, This is the current prediction value of the Q network; S36. Policy Network Parameter Update The parameters of the policy network are updated by maximizing the objective function of policy optimization. The objective function for strategy optimization is: ； in, For temperature parameters; It is the minimum Q-value output by the Q-network; It is in state Select action The probability of the strategy; S37. Target network soft update The update formula for the target network is: ； ； in, and These are the parameters of the current Q network and the policy network, respectively; and These are the parameters of the corresponding target Q-network and target policy network, respectively; It is a soft update coefficient that controls the update rate; S38. Iterate through training loops, repeating steps S32 to S37 until the required number of training rounds or convergence criteria are met, then output the final policy. .