Left-turn control method for unsignalized intersection based on double-layer attention mechanism and discrete SAC algorithm

Abstract

The application relates to a left-turn control method for a signal-free intersection based on a double-layer attention mechanism and a discrete SAC algorithm, which comprises the following steps: S1, a simulation model is constructed, wherein the simulation model comprises an intersection simulation environment, a self-vehicle model and a he-vehicle model, and data interaction between the self-vehicle model, the he-vehicle model and the intersection simulation environment is established; S2, a trajectory planning process of the self-vehicle model is defined as a Markov decision process, a first model is constructed, and a state space, an action space and a reward are configured, wherein parameters in the state space are obtained based on the simulation model; and S3, a maximum entropy objective function is constructed, and a strategy with the maximum value of the maximum entropy objective function is selected as a left-turn control. Compared with the prior art, the application improves the intersection safety by 29%, and effectively displays the obstacle avoidance capability on the space-time trajectory.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving decision-making at unsignalized intersections, and in particular to a left-turn control method for unsignalized intersections based on a two-layer attention mechanism and a discrete SAC algorithm. Background Technology

[0002] With the increase in motor vehicles, traffic safety and efficiency issues at unsignalized intersections have become increasingly prominent, and traditional traffic management methods are no longer sufficient to meet people's travel needs. Meanwhile, with the continuous advancement of computer technology, communication technology, and robotics technology, autonomous driving technology has made significant progress in recent years. To solve the traffic problems at unsignalized intersections and to achieve a safer and more efficient transportation system, researching the application of autonomous driving technology at unsignalized intersections has important practical significance. Unsignalized intersections pose a major challenge to autonomous driving technology. Traditional traffic detection and control methods often cannot effectively cope with the complex vehicle and pedestrian interaction scenarios at unsignalized intersections, thus failing to ensure safety. With the rapid development of automotive engineering and artificial intelligence technologies, adaptive cruise control (ACC) and other advanced sensor technologies, such as visual information, lidar, and ultrasound, have provided possibilities for autonomous driving at unsignalized intersections. However, the practical application of these technologies still faces many challenges, such as perception difficulties, complex algorithm design, and sensor fusion. Summary of the Invention

[0003] The purpose of this invention is to provide a signalless intersection left-turn control method based on a two-layer attention mechanism and a discrete SAC algorithm.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A signalless intersection left-turn control method based on a two-layer attention mechanism and a discrete SAC algorithm includes:

[0006] Step S1: Construct a simulation model, wherein the simulation model includes an intersection simulation environment, a vehicle model, and other vehicle models, and establish data interaction between the vehicle model, other vehicle models, and the intersection simulation environment;

[0007] Step S2: Define the trajectory planning process of the autonomous vehicle model as a Markov decision process, construct the first model, and configure the state space, action space, and reward, wherein the parameters in the state space are obtained based on the simulation model;

[0008] Step S3: Construct the maximum entropy objective function and select the strategy with the largest maximum entropy objective function value as the left turn control.

[0009] The mathematical expression for the autonomous vehicle model is:

[0010]

[0011] Where: x is the abscissa of the vehicle in the intersection coordinate system, y is the ordinate of the vehicle in the intersection coordinate system, v is the vehicle's speed, ψ is the vehicle's current heading angle, β is the vehicle's slip angle at the center of gravity, and a is the vehicle's velocity.

[0012] The self-driving vehicle model uses proportional control.

[0013] The mathematical expression for the other vehicle model is:

[0014]

[0015] Where: a max For the maximum acceleration, a min v0 is the minimum acceleration, v0 is the target vehicle speed, d0 is the required headway when vehicles are in congestion, d is the current headway, d* is the desired headway, and T is the required minimum headway.

[0016] The action space consists of three discrete commands: deceleration, maintaining the current speed, and deceleration.

[0017] The reward is:

[0018] r = R safe +R eff

[0019] Where: r is the reward, R safe As a safety reward, R eff Rewards for utility;

[0020] The utility reward includes arrival sub-reward and speed sub-reward.

[0021] The utility reward includes:

[0022] R eff =R eff-1 +R eff-2

[0023]

[0024] Where: R eff-1 To reach the sub-reward, R eff-2 For speed sub-rewards, k arrive It is a constant.

[0025] The speed sub-reward is the sum of the speed reward values ​​for each step, where the speed reward value for any step is:

[0026] If the current speed is within the pre-configured speed range, the speed bonus value is a value between 0 and 1; otherwise, the speed bonus value is 0.

[0027] The objective function for maximizing entropy is:

[0028]

[0029] Where: π* represents the optimal policy, π is the policy, T is the total number of time steps, t is the time step, and E is the value of E. (( s_t, a_t ) Let r(s) be the expected value at step t, γ be the discount rate, and s be the expected value at step t. t ,a t ) represents state s t and action a t The reward at that time, where α is the temperature coefficient. For policy π in state s t Entropy is defined as the probability distribution of performing different actions under different states.

[0030] The first model is implemented using a two-layer attention mechanism and a discrete SAC algorithm.

[0031] In the discrete SAC algorithm, a neural network with parameter θ is used to parameterize the soft Q-function, and the soft Q-function is trained to minimize the soft Bellman residual.

[0032]

[0033] Among them: J Q (θ) is the replay buffer for past experiences, D is the replay buffer for past experiences, and Q is the replay buffer for past θ (s t ,a t p(s) is the parameterized soft Q-function of the neural network with parameter θ. t ,a t ) represents the state s t Next, execute action a t Then, the environment transitions to the next state s. y+1 The probability distribution, The result is the target Q-network estimation.

[0034] In the discrete SAC algorithm, the objective function of the SAC algorithm is modified to adapt to an output probability of π. φ (a t |s t (rather than density conditions)

[0035] In the discrete SAC algorithm, the improved mathematical expression for the soft state value function is:

[0036] V(s t ):=π(s t ) T [Q(s t )-αlog(π(s t ))]

[0037] Where: V(s) t ) is a soft state value function, π(s) t ) represents the state s t The probability distribution of the agent taking each action, i.e., the policy π in state s. t The output Q(s) t ) represents the state s t Under these conditions, the agent takes the soft Q value for each action, i.e., in state s. t Below, the expected cumulative reward for each action taken by the agent is log(π(s)). t Let π represent the strategy π in state s. t The logarithmic probability distribution of the output under the given conditions.

[0038] In the discrete SAC algorithm, the temperature loss is:

[0039]

[0040] in: It serves as a benchmark for representing entropy, is used to calculate temperature loss, and measures the randomness or uncertainty of a strategy.

[0041] In the sampling process of the Discrete SAC algorithm, two experience replay buffers are established: a successful experience buffer and a failed experience buffer, which store successful and unsuccessful sample experiences respectively. When a new sample needs to be stored, it is placed into the corresponding experience buffer according to its characteristics, replacing the oldest sample data in that buffer, thus ensuring the balance of positive and negative sample data in the overall experience sample. During the sample retrieval process, samples are collected from the two buffers for replay based on the magnitude of the time difference error. The mathematical expression for the experience replay process is:

[0042] δ=r(s t ,a t )+γQ′(s t+1 ,a t+1 )-Q(s t ,a t )

[0043]

[0044] Where: δ is the time difference error value, Q′(s) t+1 ,a t+1Let ) represent the target network Q in state s t+1 Next, execute action a t+1 The estimated value, n s n is the number of samples collected from the successful experience buffer. f N is the number of samples collected from the failure experience buffer. s N is the capacity of the success experience buffer. f The capacity of the failure experience buffer.

[0045] The dual-layer attention mechanism includes a feature attention layer and a target attention layer.

[0046] The feature attention layer adopts a self-attention mechanism architecture, using the state sequence obtained from the environment as input, treating the corresponding value of each vehicle entity as the feature value of the sequence, and the input vector undergoes a linear transformation to generate a matrix of query, key, and value.

[0047] The feature attention layer generates its output by weighting the value matrix and the attention weight matrix, and then summing them to generate the output of the layer. The target attention layer adopts a self-attention architecture and generates a query matrix uniquely using only the vehicle information. After calculating the attention score, the target attention layer interacts with the query matrix by combining it with a key matrix derived from all vehicle entity information.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. A motion planning controller based on the discrete SAC algorithm was designed for unsignalized intersections. This controller optimizes traffic efficiency while ensuring safety.

[0050] 2. By designing incremental rewards for utility, combining the maximum entropy objective function with the integration of multi-level attention mechanisms, the representation capabilities at the feature level and the target vehicle entity level are enhanced, allowing for hierarchical processing of information, thereby enabling accurate interpretation and response to dynamic changes in complex traffic scenarios.

[0051] 3. To address the issue of uneven sample distribution during training, a dual-experience buffer mechanism is introduced. This method significantly improves the stability and efficiency of the training process by balancing the sampling of positive and negative experiences, thereby accelerating the learning speed and improving model quality. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the main steps of the method of the present invention;

[0053] Figure 2 This is a schematic diagram of an intersection scenario;

[0054] Figure 3To simulate the scene and attention distribution map;

[0055] Figure 4 This is a schematic diagram of the overall framework of the autonomous driving decision-making strategy based on a two-layer attention mechanism and a discrete SAC algorithm. Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0057] A signalless intersection left-turn control method based on a two-layer attention mechanism and a discrete SAC algorithm, such as... Figure 1 , Figure 3 and Figure 4 As shown, it includes:

[0058] Step S1: Construct a simulation model, which includes an intersection simulation environment, a vehicle model, and other vehicle models, and establish data interaction between the vehicle model, other vehicle models, and the intersection simulation environment;

[0059] Specifically, a simulation platform like Highway-env, an open-source autonomous driving simulation platform, is built. Figure 2 The illustrated scenario depicts an unprotected left-turn at a four-way, two-lane urban intersection, consisting of multiple entrance lanes, internal roads, and exit lanes. This intersection lacks traffic light control and does not employ a priority traffic strategy for turning protection. The scenario uses an absolute coordinate system, with the intersection's center point set as the origin. A vehicle model, acting as an intelligent connected vehicle, enters the intersection from the south entrance lane and exits from the west exit lane after passing through the intersection. Other vehicles randomly enter the intersection from various entrance lanes and exit from different exit lanes without a clear direction. In this scenario, the invention assumes that, under intelligent connected conditions, the intelligent vehicle can accurately perceive its own position, speed, and other key information about surrounding vehicles. This application uses an intelligent vehicle model to interact with the simulation environment in real time under control commands. The lower-level controller of the intelligent vehicle model employs proportional control, receiving actions from the upper-level strategy and executing the control. It performs speed tracking and lane keeping based on the target speed output from the upper level. Building on this foundation, intelligent vehicles will explore a driving strategy that balances safety and efficiency, enabling them to safely and efficiently navigate along predetermined trajectories while avoiding dense traffic flows within intersections.

[0060] Step S2: Define the trajectory planning process of the autonomous vehicle model as a Markov decision process, construct the first model, and configure the state space, action space, and reward. The parameters in the state space are obtained based on the simulation model.

[0061] In this embodiment, the mathematical expression for the vehicle model is:

[0062]

[0063] Where: x is the abscissa of the vehicle in the intersection coordinate system, y is the ordinate of the vehicle in the intersection coordinate system, v is the vehicle's speed, ψ is the vehicle's current heading angle, β is the vehicle's slip angle at the center of gravity, and a is the vehicle's velocity.

[0064] The self-propelled vehicle model uses proportional control.

[0065] The mathematical expression for his car model is:

[0066]

[0067] Where: a max For the maximum acceleration, a min v0 is the minimum acceleration, v0 is the target vehicle speed, d0 is the required headway when vehicles are in congestion, d is the current headway, d* is the desired headway, and T is the required minimum headway.

[0068] In this embodiment, the action space consists of three discrete commands: deceleration, maintaining the current speed, and deceleration.

[0069] The reward is:

[0070] r = R safe +R eff

[0071] Where: r is the reward, R safe As a safety reward, R eff Rewards for utility;

[0072] In urban intersection environments, vehicle collisions are extremely detrimental to traffic safety management. Therefore, efforts must be made to avoid such incidents during intersections. To address the goal of collision avoidance, a safety-oriented reward factor is established to measure traffic safety. The specific safety rewards are as follows:

[0073]

[0074] Where: k safe The constant value reflects the importance of safety in the overall design of the reward function.

[0075] To optimize vehicle efficiency, it's necessary to encourage autonomous vehicles to operate within the target speed range. Therefore, when a vehicle's current speed reaches the desired speed, it is rewarded. On the other hand, by setting arrival rewards, the vehicle's arrival behavior is incentivized, encouraging autonomous vehicles to approach their destination safely and efficiently. Thus, utility rewards include arrival sub-rewards and speed sub-rewards.

[0076] Specifically, utility rewards include:

[0077] R eff =R eff-1 +R eff-2

[0078]

[0079] Where: R eff-1 To reach the sub-reward, R eff-2 For speed sub-rewards, k arrive It is a constant.

[0080] The speed sub-reward is the sum of the speed reward values ​​for each step, where the speed reward value for any step is:

[0081] If the current speed is within the pre-configured speed range, the speed bonus value is a value between 0 and 1; otherwise, the speed bonus value is 0.

[0082] Step S3: Construct the maximum entropy objective function and select the strategy with the largest maximum entropy objective function value as the left turn control.

[0083] The SAC algorithm is an advanced offline reinforcement learning algorithm that cleverly combines maximum entropy learning with the Actor-Critic framework, using policy entropy maximization to balance the agent's exploration and exploitation. The policy network (Actor) outputs the probability distribution of each action and selects actions based on the current state; the value network (Critic) evaluates the value of the current state-action pair and provides feedback for updating the policy; the temperature parameter controls the balance between exploration and exploitation, dynamically adjusted by maximizing the entropy objective function, which is:

[0084]

[0085] Where: π * The optimal strategy is defined by π, where π is the strategy, T is the total number of time steps, t is the number of time steps, and E is the value of E. ((s_t,a_t) Let r(s) be the expected value at step t, γ be the discount rate, and s be the expected value at step t. t ,a t ) represents state s t and action a t The reward at that time, where α is the temperature coefficient. For policy π in state s t The entropy at a given location is calculated using the following formula: The strategy is the probability distribution of performing different actions under different states.

[0086] The π learns the dynamic characteristics and reward mechanism of the environment to optimize its parameters in order to maximize cumulative reward in the long run. The agent selects actions based on the output of the policy network and observes the feedback from the environment (new states and rewards). This experience is used to update the network weights to increase the probability of obtaining higher rewards in the future. π^* is the ideal policy network that yields the maximum expected cumulative reward in the target scenario, representing the agent's optimal behavior in that environment. The SAC algorithm works by alternating between policy evaluation and policy improvement. Policy evaluation involves collecting samples of states and actions using the current policy π, and then using these samples to update the value function. Policy improvement involves updating the policy π to make it closer to the optimal policy π*.

[0087] The first model is implemented using a two-layer attention mechanism and the discrete SAC algorithm.

[0088] In the discrete SAC algorithm, a neural network with parameter θ is used to parameterize the soft Q-function, and the soft Q-function is trained to minimize the soft Bellman residual.

[0089]

[0090] Among them: J Q (θ) is the replay buffer for past experiences, D is the replay buffer for past experiences, and Q is the replay buffer for past θ (s t ,a t p(s) is the parameterized soft Q-function of the neural network with parameter θ. t ,a t ) represents the state s t Next, execute action a t Then, the environment transitions to the next state s. t+1 The probability distribution, The result is the target Q-network estimation.

[0091] Building upon this, SAC discrete modifies the objective function of the SAC algorithm for discrete state spaces to adapt to π. φ (a t |s tThis addresses the case where the output is probability rather than density. First, the soft Q-function is improved for the finite action space, enabling it to output the Q-value for each possible action instead of a single state-action pair, thus improving computational efficiency. Second, under discrete conditions, policy π can directly output the desired action distribution instead of the mean and covariance of the action distribution. Furthermore, a softmax function is applied at the end of policy π to update the policy, ensuring the output of an effective probability distribution. The definition modifications are shown in Table 1.

[0092] Table 1

[0093]

[0094] Since the action space is discrete, the soft state-value function can be calculated directly without forming a Monte Carlo estimate, thus reducing the variance in the estimate. The improved mathematical expression for the soft state-value function is as follows:

[0095] V(s t ):=π(s t ) T [Q(s t )-αlog(π(s t ))]

[0096] Where: V(s) t ) is a soft state value function, π(s) t ) represents the state s t The probability distribution of the agent taking each action, i.e., the policy π in state s. t The output Q(s) t ) represents the state s t Under these conditions, the agent takes the soft Q value for each action, i.e., in state s. t Below, the expected cumulative reward for each action taken by the agent is log(π(s)). t Let π represent the strategy π in state s. t The logarithmic probability distribution of the output under the given conditions.

[0097] Similarly, the calculation of temperature loss has also been improved:

[0098]

[0099] in: It serves as a benchmark for representing entropy, is used to calculate temperature loss, and measures the randomness or uncertainty of a strategy.

[0100] Since this strategy can output an accurate action distribution for calculating the expected value, no reparameterization technique is needed during the minimization of the loss function. Therefore, the objective of this strategy is modified to:

[0101]

[0102] To address the imbalance between successful and failed samples during the sampling process of the Discrete SAC algorithm and improve training stability and efficiency, a dual-experience-buffer-first experience replay algorithm, D-PER, is proposed. With successful and unsuccessful experiences stored and replayed separately, two experience replay buffers are established: a successful experience buffer and a failed experience buffer, storing successful and unsuccessful sample experiences respectively. When a new sample needs to be stored, it is placed into the corresponding experience buffer based on its characteristics, replacing the oldest sample data in that buffer, thus ensuring the balance of positive and negative sample data in the overall experience samples. During sample retrieval, samples are collected from the two buffers for replay based on the magnitude of the time error. This experience replay process can be expressed by the following formula:

[0103] δ=r(s t ,a t )+γQ′(s t+1 ,α t+1 )-Q(s t ,a t )

[0104]

[0105] Where: δ is the time difference error value, Q′(s) t+1 ,a t+1 Let ) represent the target network Q in state s t+1 Next, execute action a t+1 The estimated value, n s n is the number of samples collected from the successful experience buffer. f N is the number of samples collected from the failure experience buffer. s N is the capacity of the success experience buffer. f The capacity of the failure experience buffer.

[0106] The two-layer attention mechanism includes a feature attention layer and a target attention layer.

[0107] Attention mechanisms have been widely applied in deep reinforcement learning, acting on the feature information dimension and considered to significantly improve the learning performance of neural networks. In the complex environment of autonomous driving, vehicles need to extract important features from a large amount of input information. Multi-layer attention mechanisms optimize feature representation by weighting features, reducing interference from irrelevant information, helping the agent better understand the environmental state, and improving the efficiency and performance of dynamic decision-making and information aggregation. Inspired by self-attention mechanisms, this study introduces a two-layer attention mechanism based on a hierarchical feature structure, including a feature attention layer and a target attention layer, enhancing the information representation capabilities of both the feature and target layers, enabling accurate responses to dynamic changes in complex traffic scenarios.

[0108] The feature attention layer employs a self-attention mechanism architecture to process input features, optimizing feature representations and using a state sequence X obtained from the environment as input. It treats the corresponding value of each vehicle entity as a feature value of the sequence. The input vector undergoes the following linear transformation to generate a matrix of query, key, and value:

[0109] Q = XW Q K = XW K V = XW V

[0110] in: Let d represent the trainable weight matrix, where d is the dimension of the features. d It represents the dimension of the key and the query.

[0111] For the matrix above, attention scores are obtained by calculating the similarity between the query and the key:

[0112]

[0113] The feature attention layer generates its output by weighting the value matrix with the attention weight matrix and then summing them.

[0114] The target attention layer employs a self-attention architecture similar to the feature layer, but uniquely generates a query matrix using only autonomous vehicle information. It calculates an attention score using the same method described above, optimizing the decision-making process by interacting with this query matrix against a key matrix derived from all vehicle entity information. This two-layer attention mechanism improves the decision-making efficiency and safety of autonomous vehicles at unsignalized intersections by processing and integrating information at different levels.

[0115] In the DRL learning process, the configuration of the state space and behavior space is crucial. Agents observe and acquire the environmental state through interaction and execute actions according to the corresponding policies. Subsequently, at the next time step, they receive rewards from environmental feedback to evaluate the actions performed.

[0116] Furthermore, in this embodiment, the state space is composed of the position and speed information acquired by the onboard equipment of the intelligent vehicle and the position and speed data of surrounding vehicles sensed by roadside sensors. That is:

[0117] s = [presence, x, y, v] s ,v y ]

[0118] Furthermore, the autonomous vehicle decision controller, as the upper-level control system, needs to provide necessary control commands to the lower-level proportional controller to achieve safe and efficient driving operations. Therefore, the action space of the DRL algorithm consists of three discrete commands:

[0119] In a preferred embodiment of the present invention, the verification process for the method of this application is as follows:

[0120] Step 1: To demonstrate the performance advantages of the dual empirical buffer sampling algorithm and the dual-layer attention mechanism algorithm in autonomous driving decision-making at unsignalized intersections, a comparative analysis is conducted with several other reinforcement learning algorithms. This invention selects the DQN algorithm as a representative algorithm for discrete control scenarios, whose application in various environments has been widely verified and has achieved good results. Meanwhile, the PPO algorithm, as a typical representative of policy gradient methods and sharing the Critic and Actor-based architecture with the SAC algorithm, is included in the comparison scope. Furthermore, this invention also includes comparisons with the discrete SAC algorithm and the discrete SAC algorithm using dual empirical pool sampling, aiming to verify the effectiveness of the proposed algorithm optimization measures.

[0121] Step 2: Set the training period for all five algorithms to 3000 segments and obtain the average reward change curves for each algorithm. From the algorithm performance, the rewards increased rapidly in the early stages. This is because the vehicles were primarily learning how to increase speed to obtain more speed rewards. After approximately 500 segments, almost all algorithms had learned how to obtain rewards from efficient driving behavior. Further increases in reward values ​​require the identification and obstacle avoidance of dense traffic flow at intersections to avoid safety penalties. During this process, the DA_SACD and DPER_SACD algorithms showed better performance than the initial algorithms. The SACD algorithm, compared to other reinforcement learning algorithms, was also able to identify some dangerous states, thus achieving higher average rewards in some segments.

[0122] Step 3: Evaluate the safety performance and impact on average speed of various algorithms during and after training. By comparing these algorithms with the benchmark DQN algorithm, their specific performance in terms of efficiency and safety is illustrated. Furthermore, by setting weighting coefficients, a comprehensive score that balances safety and efficiency is calculated.

[0123] Score=ω1I s +ω2I e

[0124] Where ω1 represents the weighting coefficient of the security objective, which is taken as 0.495; I s ω represents the safety improvement rate; ω2 represents the weighting coefficient of the efficiency target, which is set to 0.15; I e This indicates the rate of speed improvement.

[0125] As training progressed, the DA_SACD algorithm significantly reduced the collision rate, demonstrating the best performance. In contrast, other algorithms continued to experience higher collision frequencies at the end of training, with a 29% reduction in collision rate and only a 6% reduction in speed, thus achieving a balance between efficiency and safety.

[0126] Step 4: Compare the initial discrete SAC algorithm with the algorithm proposed in this invention in a specific time and space. Based on the vehicle outline, draw the trajectories of the vehicle and the surrounding vehicles closely related to the vehicle, and compare and analyze the obstacle avoidance performance of the two algorithms: The discrete SAC algorithm runs at a higher speed, but fails to effectively identify obstacle vehicles, thus causing a collision; while the improved DA_SACD algorithm can identify dangerous situations while maintaining high speed, and safely pass through unsignalized intersections by utilizing gaps in traffic flow.

[0127] Step 5: The key information that autonomous vehicles focus on under different traffic conditions is visualized in the form of attention distribution heatmaps, providing support for understanding and optimizing driving strategies.

[0128] The images demonstrate that, at a certain distance from the intersection, the driving strategy proposed in this invention can effectively identify potential future conflicting vehicles, thereby enabling efficient and safe passage through the intersection by utilizing gaps in traffic flow. When new vehicles appear, the autonomous vehicle can quickly redistribute attention weights based on the current relationship between the vehicle and surrounding vehicles. After the vehicle passes through the intersection, it can shift its focus from the previously conflicting vehicles to the vehicles it is currently following, based on the current state and task requirements, while ignoring information irrelevant to the current driving objective, such as vehicles about to enter the current intersection as shown in the driving scenario, demonstrating superior flexibility.

[0129] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A signalless intersection left-turn control method based on a two-layer attention mechanism and a discrete SAC algorithm, characterized in that, include: Step S1: Construct a simulation model, wherein the simulation model includes an intersection simulation environment, a vehicle model, and other vehicle models, and establish data interaction between the vehicle model, other vehicle models, and the intersection simulation environment; Step S2: Define the trajectory planning process of the autonomous vehicle model as a Markov decision process, construct the first model, and configure the state space, action space, and reward, wherein the parameters in the state space are obtained based on the simulation model; Step S3: Construct the maximum entropy objective function and select the strategy with the largest maximum entropy objective function value as the left turn control; The action space consists of three discrete commands: deceleration, maintaining the current speed, and deceleration. The reward is: r=R safe +R eff Where: r is the reward, R safe As a safety reward, R eff Rewards for utility; The utility reward includes arrival sub-reward and speed sub-reward; The utility reward includes: R eff =R eff-1 +R eff-2 Where: R eff-1 To reach the sub-reward, R eff-2 For speed sub-rewards, It is a constant; The speed sub-reward is the sum of the speed reward values ​​for each step, where the speed reward value for any step is: If the current speed is within the pre-configured speed range, the speed bonus value is a value between 0 and 1; otherwise, the speed bonus value is 0. The objective function for maximizing entropy is: in: The optimal strategy is... Let T be the total number of time steps, t be the number of time steps, and E be the time step. ((s_t,a_t ) Let γ be the expected value at step t, and γ be the discount rate. For state s t and action a t The reward at that time, where α is the temperature coefficient. For strategy In state s t Entropy is defined as the probability distribution of performing different actions under different states.

2. The signalless intersection left-turn control method based on a two-layer attention mechanism and discrete SAC algorithm according to claim 1, characterized in that, The mathematical expression for the autonomous vehicle model is: Where: x is the abscissa of the vehicle in the intersection coordinate system, y is the ordinate of the vehicle in the intersection coordinate system, v is the vehicle's speed, ψ is the vehicle's current heading angle, β is the vehicle's slip angle at the center of gravity, and a is the vehicle's velocity.

3. The signalless intersection left-turn control method based on a two-layer attention mechanism and discrete SAC algorithm according to claim 2, characterized in that, The self-driving vehicle model uses proportional control.

4. The signalless intersection left-turn control method based on a two-layer attention mechanism and discrete SAC algorithm according to claim 1, characterized in that, The mathematical expression for the other vehicle model is: in: a max For maximum acceleration, For minimum acceleration, v 0 is the target vehicle speed. d 0 represents the required headway when traffic is congested. d This is the current distance between the front of the vehicles. d * represents the desired front-end clearance. T This is the minimum headway required.

5. The signalless intersection left-turn control method based on a two-layer attention mechanism and discrete SAC algorithm according to claim 1, characterized in that, The first model is implemented using a two-layer attention mechanism and a discrete SAC algorithm. In the discrete SAC algorithm, a neural network with parameter θ is used to parameterize the soft Q-function, and the soft Q-function is trained to minimize the soft Bellman residual. in: Let D be a replay buffer for past experiences. Let θ be the parameterized soft Q-function of the neural network. To indicate the state Next action Then, the environment transitions to the next state. The probability distribution, The result is the target Q-network estimation. In the discrete SAC algorithm, the objective function of the SAC algorithm is modified to adapt to the output probability. Rather than density conditions; In the discrete SAC algorithm, the improved mathematical expression for the soft state value function is: in: It is a soft state value function. To indicate the state Below, the probability distribution of the agent's various actions, i.e., the policy. In state The output below, To indicate the state Under these conditions, the agent takes the soft Q value for each action, that is, in the state... Below, the expected cumulative reward for each action taken by the agent. To represent the strategy In state The logarithmic probability distribution of the output under the given conditions. In the discrete SAC algorithm, the temperature loss is: in: It serves as a benchmark for representing entropy, is used to calculate temperature loss, and measures the randomness or uncertainty of a strategy.

6. The signalless intersection left-turn control method based on a two-layer attention mechanism and discrete SAC algorithm according to claim 5, characterized in that, In the sampling process of the Discrete SAC algorithm, two experience replay buffers are established: a successful experience buffer and a failed experience buffer, which store successful and unsuccessful sample experiences respectively. When a new sample needs to be stored, it is placed into the corresponding experience buffer according to its characteristics, replacing the oldest sample data in that buffer, thus ensuring the balance of positive and negative sample data in the overall experience sample. During the sample retrieval process, samples are collected from the two buffers for replay based on the magnitude of the time difference error. The mathematical expression for the experience replay process is: in: This is the time difference error value. To represent the target network In state Next action The estimated value, The number of samples collected from the successful experience buffer. N represents the number of samples collected from the failure experience buffer. The capacity of the successful experience buffer, The capacity of the failure experience buffer.

7. The signalless intersection left-turn control method based on a two-layer attention mechanism and discrete SAC algorithm according to claim 6, characterized in that, The dual-layer attention mechanism includes a feature attention layer and a target attention layer. The feature attention layer adopts a self-attention mechanism architecture, using the state sequence obtained from the environment as input, treating the corresponding value of each vehicle entity as the feature value of the sequence, and the input vector undergoes a linear transformation to generate a matrix of query, key, and value. The feature attention layer generates its output by weighting the value matrix and the attention weight matrix, and then summing them to generate the output of the layer. The target attention layer adopts a self-attention architecture and generates a query matrix uniquely using only the vehicle information. After calculating the attention score, the target attention layer interacts with the query matrix by combining it with a key matrix derived from all vehicle entity information.

Images