Non-signalized intersection left-turn control method based on double-layer attention mechanism and discrete SAC algorithm

Abstract

The invention relates to a non-signalized intersection left-turn control method based on a double-layer attention mechanism and a discrete SAC algorithm, and the method comprises the steps: S1, building a simulation model which comprises an intersection simulation environment, a self-vehicle model and an other-vehicle model, and building the data interaction between the self-vehicle model and the intersection simulation environment, and between the other-vehicle model and the intersection simulation environment; s2, a trajectory planning process of the self-vehicle model is defined as a Markov decision process, a first model is constructed, a state space, an action space and rewards are configured, and parameters in the state space are obtained based on a simulation model; and S3, constructing a maximization entropy objective function, and selecting a strategy with the maximum maximization entropy objective function value as left-turn control. Compared with the prior art, the method has the advantages that the intersection safety is improved by 29%, and the effective obstacle avoidance capability is displayed on the spatial-temporal trajectory.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving decision-making at unsignalized intersections, and in particular to a left-turn control method for unsignalized intersections based on a double-layer attention mechanism and a discrete SAC algorithm. Background Art

[0002] With the increase of motor vehicles, the traffic safety and efficiency issues at unsignalized intersections have become increasingly prominent, and traditional traffic management methods can no longer meet people's travel needs. At the same time, with the continuous advancement of computer technology, communication technology, and robotics technology, autonomous driving technology has made great progress in recent years. In order to solve the traffic problems at unsignalized intersections and to achieve a safer and more efficient transportation system, it is of great practical significance to study the application of autonomous driving technology at unsignalized intersections. Unsignalized intersections are a major challenge facing autonomous driving technology. Traditional traffic detection and control methods often cannot effectively cope with the complex vehicle and pedestrian interaction scenarios at unsignalized intersections, and therefore cannot ensure safety. With the rapid development of automotive engineering and artificial intelligence technology, adaptive cruise control systems (ACC) and other advanced sensor technologies, such as visual information, lidar, and ultrasound, have made autonomous driving at unsignalized intersections possible. However, the practical application of these technologies still faces many challenges, such as perception difficulties, complex algorithm design, and sensor fusion. Summary of the invention

[0003] The purpose of the present invention is to provide a left-turn control method for an unsignalized intersection based on a double-layer attention mechanism and a discrete SAC algorithm.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A left-turn control method for unsignalized intersections based on a double-layer attention mechanism and a discrete SAC algorithm, comprising:

[0006] Step S1: constructing a simulation model, wherein the simulation model includes a road intersection simulation environment, a self-vehicle model and an other-vehicle model, and establishing data interaction between the self-vehicle model and the other-vehicle model and the road intersection simulation environment;

[0007] Step S2: defining the trajectory planning process of the ego-vehicle model as a Markov decision process, constructing a first model, and configuring a state space, an action space, and a reward, wherein the parameters in the state space are obtained based on the simulation model;

[0008] Step S3: construct a maximum entropy objective function, and select the strategy with the largest value of the maximum entropy objective function as the left turn control.

[0009] The mathematical expression of the ego vehicle model is:

[0010]

[0011] Where: x is the horizontal coordinate of the vehicle in the intersection coordinate system, y is the vertical coordinate of the vehicle in the intersection coordinate system, v is the speed of the vehicle, ψ is the current heading angle of the vehicle, β is the slip angle of the vehicle at the center of gravity, and a is the speed of the vehicle.

[0012] The ego vehicle model adopts proportional control.

[0013] The mathematical expression of the other car model is:

[0014]

[0015] Among them: a max is the maximum acceleration, a min is the minimum acceleration, v0 is the target speed, d0 is the required headway in congested vehicles, d is the current headway, d* is the expected headway, and T is the minimum headway time required.

[0016] The action space consists of three discrete numbers, namely deceleration, maintaining current speed and deceleration;

[0017] The rewards are:

[0018] r=R safe +R eff

[0019] Among them: r is the reward, R safe For safety bonus, R eff Reward for utility;

[0020] The utility reward includes an arrival sub-reward and a speed sub-reward.

[0021] The utility rewards include:

[0022] R eff =R eff-1 +R eff-2

[0023]

[0024] Where: R eff-1 To reach the sub-reward, R eff-2 is the speed sub-reward, k arrive is a constant.

[0025] The speed sub-reward is the sum of the speed reward values ​​of each step, where the speed reward value of any step is:

[0026] If the current speed is within the preconfigured speed range, the speed reward value takes a value between 0 and 1, otherwise, the speed reward value is 0.

[0027] The maximum entropy objective function is:

[0028]

[0029] Among them: π* is the optimal strategy, π is the strategy, T is the total number of time steps, t is the time step, E (( s_t,a_t ) is the mathematical expectation of the tth step, γ is the discount rate, r(s t ,a t ) is state s t and action a t The reward is , α is the temperature coefficient, For strategy π in state s t The strategy is the probability distribution of performing different actions in different states.

[0030] The first model is implemented using a double-layer attention mechanism and a discrete SAC algorithm.

[0031] In the discrete SAC algorithm, the soft Q-function is parameterized using a neural network with parameter θ, and the soft Q-function is trained to minimize the soft Bellman residual:

[0032]

[0033] Among them: J Q (θ) is, D is the replay buffer of past experience, Q θ (s t ,a t ) is the soft Q function parameterized by the neural network with parameter θ, p(s t ,a t ) represents the state s t Next, perform action a t After that, the environment moves to the next state s y+1 The probability distribution of Estimation result for the target Q network;

[0034] In the discrete SAC algorithm, the objective function of the SAC algorithm is modified to adapt to the output probability of π φ (a t |s t ) rather than density conditions;

[0035] In the discrete SAC algorithm, the improved mathematical expression of the soft state value function is:

[0036] V(s t ):=π(s t ) T [Q(s t )-αlog(π(s t ))]

[0037] Where: V(s t ) is the soft state value function, π(s t ) represents the state s t Under this condition, the probability distribution of each action taken by the agent is t The output under Q(s t ) represents the state s t Under this condition, the agent takes the soft Q value of each action, that is, in state s t Under this condition, the expected cumulative reward for each action taken by the agent is log(π(s t )) represents the strategy π in state s t The logarithmic probability distribution of the output under

[0038] In the discrete SAC algorithm, the temperature loss is:

[0039]

[0040] in: It is a baseline value for entropy, used to calculate temperature loss, and is used to measure the randomness or uncertainty of a strategy.

[0041] In the sampling process of the discrete SAC algorithm, two experience replay buffers are established, one successful experience buffer and one failed experience buffer, to store successful and unsuccessful sample experiences respectively, while the successful and unsuccessful experiences are stored and replayed respectively. When a new sample needs to be stored, it is placed in the corresponding experience buffer according to its characteristics, and the earliest sample data in the buffer is replaced to ensure the balance of positive and negative sample data in the overall experience sample. In the sample retrieval process, samples are collected from the two buffers for replay according to the size of the time difference error. The mathematical expression of the experience replay process is:

[0042] δ=r(s t ,a t )+γQ′(s t+1 ,a t+1 )-Q(s t ,a t )

[0043]

[0044] Where: δ is the time difference error value, Q′(s t+1 ,a t+1) represents the target network Q in state s t+1 Next, perform action a t+1 The estimated value of s is the number of samples collected from the successful experience buffer, n f is the number of samples collected from the failure experience buffer, N is the number of samples, N s is the capacity of the success experience buffer, N f The capacity of the failure experience buffer.

[0045] The dual-layer attention mechanism includes a feature attention layer and a target attention layer.

[0046] The feature attention layer adopts the architecture of the self-attention mechanism, uses the state sequence obtained from the environment as input, regards the corresponding value of each vehicle entity as the feature value of the sequence, and the input vector is linearly transformed to generate a matrix of query, key and value;

[0047] The feature attention layer generates the output of the feature attention layer by weighting the value matrix with the attention weight matrix, and adds them together to generate the output of this layer. The target attention layer adopts a self-attention architecture and uniquely generates a query matrix using only the vehicle information. After calculating the attention score, the target attention layer interacts with the query matrix with the key matrix derived from all vehicle entity information.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. For unsignalized intersections, a motion planning controller based on the discrete SAC algorithm is designed. This controller optimizes traffic efficiency while ensuring safety.

[0050] 2. By designing the utility reward as an increasing reward, combining the designed maximization entropy objective function, and integrating the multi-level attention mechanism, the representation capabilities at the feature level and the target vehicle entity level are enhanced, allowing hierarchical processing of information, thereby accurately interpreting and responding to dynamic changes in complex traffic scenarios.

[0051] 3. In order to solve the problem of uneven sample distribution during training, a dual experience buffer mechanism is introduced. This method significantly improves the stability and efficiency of the training process by balancing the sampling of positive and negative experiences, thereby accelerating the learning speed and improving the model quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic flow chart of the main steps of the method of the present invention;

[0053] Figure 2 This is a schematic diagram of the intersection scene;

[0054] Figure 3To simulate scenes and attention distribution maps;

[0055] Figure 4 Schematic diagram of the overall framework of the autonomous driving decision-making strategy with a dual-layer attention mechanism and discrete SAC algorithm. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0057] A left-turn control method for unsignalized intersections based on a dual-layer attention mechanism and discrete SAC algorithm, such as Figure 1 , Figure 3 and Figure 4 As shown, including:

[0058] Step S1: constructing a simulation model, wherein the simulation model includes a road intersection simulation environment, a self-vehicle model and an other-vehicle model, and establishing data interaction between the self-vehicle model and the other-vehicle model and the road intersection simulation environment;

[0059] Specifically, we use the open source autonomous driving simulation platform Highway-env to build a Figure 2 The unprotected left turn scene of the four-way two-lane urban intersection shown is composed of three parts: entrance roads, internal roads and exit roads in multiple directions. The intersection is not equipped with signal light control, nor does it adopt a priority traffic strategy for turning protection. The scene uses an absolute coordinate system, and the center point of the intersection is set as the coordinate origin. The self-vehicle model, as an intelligent networked vehicle, enters the intersection from the south entrance road of the intersection, and leaves the intersection from the west exit road after passing the intersection; other vehicles enter the intersection randomly from various entrance roads of the intersection, and leave the intersection from different exit roads in an undirected manner. In the current scenario, the present invention assumes that under the condition of intelligent networking, the intelligent vehicle can accurately perceive the key information such as the position and speed of itself and surrounding vehicles. This application uses an intelligent vehicle model to interact with the simulation environment in real time under control commands. The lower-level controller of the intelligent vehicle model adopts proportional control and executes control after receiving the action output by the upper-level strategy. It performs speed tracking and lane keeping according to the target speed output by the upper layer. On this basis, the intelligent vehicle will explore a driving strategy that balances the two goals of safety and efficiency, and achieves safe and efficient passage through the intersection along a predetermined trajectory while avoiding the dense traffic flow inside the intersection.

[0060] Step S2: defining the trajectory planning process of the ego-vehicle model as a Markov decision process, constructing a first model, and configuring the state space, action space, and reward, wherein the parameters in the state space are obtained based on the simulation model;

[0061] In this embodiment, the mathematical expression of the vehicle model is:

[0062]

[0063] Where: x is the horizontal coordinate of the vehicle in the intersection coordinate system, y is the vertical coordinate of the vehicle in the intersection coordinate system, v is the speed of the vehicle, ψ is the current heading angle of the vehicle, β is the slip angle of the vehicle at the center of gravity, and a is the speed of the vehicle.

[0064] The ego vehicle model uses proportional control.

[0065] The mathematical expression of the other car model is:

[0066]

[0067] Among them: a max is the maximum acceleration, a min is the minimum acceleration, v0 is the target speed, d0 is the required headway in congested vehicles, d is the current headway, d* is the expected headway, and T is the minimum headway time required.

[0068] In this embodiment, the action space consists of three discrete numbers, namely, deceleration, maintaining the current speed, and deceleration;

[0069] The rewards are:

[0070] r=R safe +R eff

[0071] Among them: r is the reward, R safe For safety bonus, R eff Reward for utility;

[0072] In the urban intersection environment, the occurrence of vehicle collisions is extremely detrimental to traffic safety management. Therefore, efforts must be made to avoid such incidents during the intersection process. In order to solve the goal of avoiding collisions, a safety-oriented reward factor is established to measure traffic safety. The safety reward is specifically:

[0073]

[0074] Where: k safe is a constant, reflecting the importance of safety in the overall design of the reward function.

[0075] In order to achieve the optimization goal of single-vehicle efficiency, it is necessary to encourage autonomous vehicles to travel within the target speed range. Therefore, when the current speed of the autonomous vehicle reaches the expected speed, a certain reward is given to the vehicle. On the other hand, by setting the arrival reward, the vehicle's arrival behavior is rewarded, prompting the autonomous vehicle to approach its destination safely and efficiently. Therefore, the utility reward includes the arrival sub-reward and the speed sub-reward.

[0076] Specifically, utility rewards include:

[0077] R eff =R eff-1 +R eff-2

[0078]

[0079] Where: R eff-1 To reach the sub-reward, R eff-2 is the speed sub-reward, k arrive is a constant.

[0080] The speed sub-reward is the sum of the speed reward values ​​of each step, where the speed reward value of any step is:

[0081] If the current speed is within the preconfigured speed range, the speed reward value takes a value between 0 and 1, otherwise, the speed reward value is 0.

[0082] Step S3: construct a maximum entropy objective function, and select the strategy with the largest value of the maximum entropy objective function as the left turn control.

[0083] The SAC algorithm is an advanced offline reinforcement learning algorithm that cleverly combines maximum entropy learning with the Actor-Critic framework, and uses policy entropy maximization to balance the exploration and utilization of the agent. The policy network (Actor) outputs the probability distribution of each action and selects actions based on the current state; the value network (Critic) evaluates the value of the current state-action pair and provides feedback for updating the policy; the temperature parameter controls the balance between exploration and utilization, and is dynamically adjusted by maximizing the entropy objective. The maximum entropy objective function is:

[0084]

[0085] Where: π * is the optimal strategy, π is the strategy, T is the total number of time steps, t is the time step, E ((s_t,a_t) is the mathematical expectation of the tth step, γ is the discount rate, r(s t ,a t ) is state s t and action a t The reward is , α is the temperature coefficient, For strategy π in state s t The entropy at is calculated as A policy is the probability distribution of performing different actions in different states.

[0086] π learns the dynamic characteristics and reward mechanism of the environment to optimize its parameters in order to maximize the cumulative reward in the long run. The agent chooses actions based on the output of the policy network and observes the feedback (new states and rewards) from the environment. These experiences are used to update the weights of the network to increase the probability of obtaining higher rewards in the future. π^* is the ideal policy network that can bring the maximum expected cumulative reward in the target scenario, representing the optimal behavior of the agent in the environment. The SAC algorithm works by alternating between policy evaluation and policy improvement. Policy evaluation involves using the current policy π to collect samples of states and actions, and then using these samples to update the value function. Policy improvement involves updating the policy π to make it closer to the optimal policy π*.

[0087] The first model is implemented using a two-layer attention mechanism and discrete SAC algorithm.

[0088] In the discrete SAC algorithm, the soft Q-function is parameterized using a neural network with parameter θ and trained to minimize the soft Bellman residual:

[0089]

[0090] Among them: J Q (θ) is, D is the replay buffer of past experience, Q θ (s t ,a t ) is the soft Q function parameterized by the neural network with parameter θ, p(s t ,a t ) represents the state s t Next, perform action a t After that, the environment moves to the next state s t+1 The probability distribution of Estimation result for the target Q network;

[0091] On this basis, for discrete state space, SAC discrete modifies the objective function of the SAC algorithm to adapt to π φ (a t |s t) outputs probability instead of density. First, the soft Q function is improved for the limited action space so that it can output the Q-value of each possible action instead of a single state-action pair, thereby improving the computational efficiency. On the other hand, under discrete conditions, the policy π can directly output the required action distribution instead of the mean and covariance of the action distribution. In addition, a softmax function is applied at the end of the policy π to implement the policy update to ensure the output of a valid probability distribution. The definition modifications are shown in Table 1:

[0092] Table 1

[0093]

[0094] Since the action space is discrete, the calculation of the soft state value function does not need to form a Monte Carlo estimate, but directly calculates the expected value to reduce the variance in the estimate. The improved mathematical expression of the soft state value function is as follows:

[0095] V(s t ):=π(s t ) T [Q(s t )-αlog(π(s t ))]

[0096] Where: V(s t ) is the soft state value function, π(s t ) represents the state s t Under this condition, the probability distribution of each action taken by the agent is t The output under Q(s t ) represents the state s t Under this condition, the agent takes the soft Q value of each action, that is, in state s t Under this condition, the expected cumulative reward for each action taken by the agent is log(π(s t )) represents the strategy π in state s t The logarithmic probability distribution of the output under

[0097] Likewise, the calculation of temperature losses has been improved:

[0098]

[0099] in: It is a baseline value for entropy, used to calculate temperature loss, and is used to measure the randomness or uncertainty of a strategy.

[0100] Since this strategy can output an accurate action distribution to calculate the expected value, there is no need to re-parameterize the technique in the process of minimizing the loss function. Therefore, the goal of this strategy is modified to:

[0101]

[0102] In the sampling process of the discrete SAC algorithm, in order to adjust the imbalance between successful samples and failed samples during the training process and improve the stability and efficiency of training, a dual experience buffer priority experience replay algorithm D-PER is proposed. In the case where successful and unsuccessful experiences are stored and replayed separately, two experience replay buffers are established, a successful experience buffer and a failed experience buffer, to store successful and unsuccessful sample experiences respectively. When a new sample needs to be stored, it is placed in the corresponding experience buffer according to the characteristics of the sample, and the earliest sample data in the buffer is replaced, thereby ensuring the balance of positive and negative sample data in the overall experience sample. In the sample retrieval process, samples are collected from the two buffers for replay according to the size of the time error. This experience replay process can be expressed by the following formula:

[0103] δ=r(s t ,a t )+γQ′(s t+1 ,α t+1 )-Q(s t ,a t )

[0104]

[0105] Where: δ is the time difference error value, Q′(s t+1 ,a t+1 ) represents the target network Q in state s t+1 Next, perform action a t+1 The estimated value of s is the number of samples collected from the successful experience buffer, n f is the number of samples collected from the failure experience buffer, N is the number of samples, N s is the capacity of the success experience buffer, N f The capacity of the failure experience buffer.

[0106] The two-layer attention mechanism includes a feature attention layer and a target attention layer.

[0107] Attention mechanisms have been widely used in deep reinforcement learning. They work on the dimension of feature information and are believed to significantly improve the learning performance of neural networks. In the complex environment of autonomous driving, vehicles need to extract important features from a large amount of input information. Multi-layer attention mechanisms optimize feature representation by weighting features, reduce the interference of irrelevant information, help agents better understand the state of the environment, and improve the efficiency and performance of dynamic decision-making and information aggregation. Inspired by the self-attention mechanism, this study introduces a two-layer attention mechanism based on a hierarchical feature structure, including a feature attention layer and a target attention layer, which enhances the information representation capabilities of the feature layer and the target layer, enabling accurate responses to dynamic changes in complex traffic scenes.

[0108] The feature attention layer uses the architecture of the self-attention mechanism to process the input features and optimize the feature representation, using the state sequence X obtained from the environment as input. It regards the corresponding value of each vehicle entity as the feature value of the sequence. The input vector undergoes the following linear transformation to generate a matrix of query, key, and value:

[0109] Q=XW Q ,K=XW K ,V=XW V

[0110] in: represents the trainable weight matrix, d is the dimension of the feature, d d is the dimensionality of the key and query.

[0111] For the above matrix, the attention score is obtained by calculating the similarity between the query and the key:

[0112]

[0113] The feature attention layer generates the output of the layer by weighting the value matrix with the attention weight matrix and adding them together.

[0114] The target attention layer uses a self-attention architecture similar to the feature layer, but only uses the autonomous vehicle information to uniquely generate the query matrix. It uses the same method as above to calculate the attention score, and optimizes the decision-making process by interacting this query matrix with the key matrix derived from all vehicle entity information. The two-layer attention mechanism improves the decision-making efficiency and safety of autonomous vehicles at non-signalized intersections by processing and integrating information at different levels.

[0115] In the DRL learning process, the configuration of state space and behavior space is crucial. Agents observe and obtain the state of the environment through interaction and perform actions according to the corresponding strategy. Subsequently, at the next time step, they receive rewards from environmental feedback to evaluate the actions performed.

[0116] In addition, in this embodiment, the state space is composed of the position and speed information obtained by the on-board equipment of the intelligent vehicle and the position and speed data information of the surrounding vehicles sensed by the roadside sensors. That is:

[0117] s=[presence,x,y,v s ,v y ]

[0118] In addition, the autonomous vehicle decision controller as the upper control system needs to provide the necessary control commands to the lower proportional controller to achieve safe and efficient driving operations. Therefore, the action space of the DRL algorithm consists of three discrete commands:

[0119] In a preferred embodiment of the present invention, the verification process of the present application method is as follows:

[0120] Step 1. In order to prove the performance advantages of the dual experience buffer sampling algorithm and the dual-layer attention mechanism algorithm in autonomous driving decision-making at unsignaled intersections, they are compared and analyzed with several other reinforcement learning algorithms. The present invention selects the DQN algorithm as a representative algorithm in discrete control scenarios. Its application in various environments has been widely verified and achieved good results. At the same time, the PPO algorithm is included in the comparison range because it is a typical representative of the policy gradient method and shares the Critic and Actor-based architecture with the SAC algorithm. In addition, the present invention also includes a comparison with a discrete SAC algorithm and a discrete SAC algorithm using dual experience pool sampling, aiming to verify the effectiveness of the proposed algorithm optimization measures.

[0121] Step 2: Set the training cycle of the five algorithms to 3000 segments, and obtain the change curve of the average reward of each algorithm. From the performance of the algorithm, the reward has increased rapidly in the early stage. This is because the vehicle is mainly learning how to increase the speed to obtain more speed rewards. After about 500 segments, almost all algorithms have learned how to obtain rewards from efficient driving behavior. The subsequent increase in reward value requires certain recognition of dense traffic in the intersection and obstacle avoidance to avoid safety penalties. In this process, the DA_SACD algorithm and the DPER_SACD algorithm show better performance than the initial algorithm, and the SACD algorithm can also identify some dangerous states compared to other reinforcement learning algorithms, so that higher average rewards can be obtained in some segments.

[0122] Step 3: Evaluate the safety performance of various algorithms and their impact on average speed during and after training. By comparing these algorithms with the baseline DQN algorithm, their specific performance in terms of efficiency and safety is clarified. In addition, by setting weight coefficients, a comprehensive score that takes into account both safety and efficiency is calculated:

[0123] Score=ω1I s +ω2I e

[0124] Among them, ω1 represents the weight coefficient of the safety target, which is 0.495; I s represents the safety improvement rate; ω2 represents the weight coefficient of the efficiency target, which is 0.15; I e Indicates the speed improvement rate.

[0125] As training progressed, the DA_SACD algorithm significantly reduced its collision rate, showing the best performance. In contrast, the other algorithms continued to experience higher collision frequencies at the end of training, with a 29% reduction in collision rate and only a 6% reduction in speed, thus achieving a balance between efficiency and safety.

[0126] Step 4: Compare the initial discrete SAC algorithm with the algorithm proposed in the present invention at a specific time and space. Considering the vehicle outline, draw the trajectories of the vehicle and the surrounding vehicles closely related to the vehicle, and compare and analyze the obstacle avoidance performance of the two algorithms: the discrete SAC algorithm runs at a higher speed, but fails to effectively identify the obstacle vehicle, thus causing a collision; while the improved DA_SACD algorithm can identify dangerous situations while maintaining a high speed, and use the gaps in the traffic flow to safely pass through the unsignalized intersection.

[0127] Step 5: Visualize the key information that the autonomous driving vehicle pays attention to under different traffic conditions in the form of an attention distribution heat map, providing support for understanding and optimizing driving strategies.

[0128] The images show that: when a certain distance away from the intersection, the driving strategy proposed by the present invention can effectively identify potential conflicting vehicles in the future, so as to efficiently utilize the gaps in traffic flow and safely pass through the intersection; when a new vehicle appears, the autonomous driving car can quickly reallocate attention weights according to the current relationship between the surrounding vehicles and the vehicle itself; when the vehicle passes through the intersection, it can shift the focus from the previous conflicting vehicle to the current following vehicle according to the current status and task requirements, while ignoring information that is not related to the current driving goal, such as the vehicle that is about to enter the current intersection shown in the driving scenario, demonstrating superior flexibility.

[0129] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A left-turn control method for unsignalized intersections based on a dual-layer attention mechanism and a discrete SAC algorithm, characterized in that: include: Step S1: constructing a simulation model, wherein the simulation model includes a road intersection simulation environment, a self-vehicle model and an other-vehicle model, and establishing data interaction between the self-vehicle model and the other-vehicle model and the road intersection simulation environment; Step S2: defining the trajectory planning process of the ego-vehicle model as a Markov decision process, constructing a first model, and configuring a state space, an action space, and a reward, wherein the parameters in the state space are obtained based on the simulation model; Step S3: construct a maximum entropy objective function, and select the strategy with the largest value of the maximum entropy objective function as the left turn control.

2. The left-turn control method for an unsignalized intersection based on a dual-layer attention mechanism and a discrete SAC algorithm according to claim 1, characterized in that: The mathematical expression of the ego vehicle model is: Where: x is the horizontal coordinate of the vehicle in the intersection coordinate system, y is the vertical coordinate of the vehicle in the intersection coordinate system, v is the speed of the vehicle, ψ is the current heading angle of the vehicle, β is the slip angle of the vehicle at the center of gravity, and a is the speed of the vehicle.

3. The left-turn control method for an unsignalized intersection based on a dual-layer attention mechanism and a discrete SAC algorithm according to claim 2, characterized in that: The ego vehicle model adopts proportional control.

4. The left-turn control method for an unsignalized intersection based on a dual-layer attention mechanism and a discrete SAC algorithm according to claim 1, characterized in that: The mathematical expression of the other car model is: Among them: a max is the maximum acceleration, a min is the minimum acceleration, v0 is the target speed, d0 is the required headway in congested vehicles, d is the current headway, d* is the expected headway, and T is the minimum headway time required.

5. The method for controlling left turn at unsignalized intersection based on double-layer attention mechanism and discrete SAC algorithm according to claim 1, characterized in that: The action space consists of three discrete numbers, namely deceleration, maintaining current speed and deceleration; The rewards are: r=R safe +R eff Among them: r is the reward, R safe For safety bonus, R eff Reward for utility; The utility reward includes an arrival sub-reward and a speed sub-reward. The utility rewards include: R eff =R eff-1 +R eff-2 Where: R eff-1 To reach the sub-reward, R eff-2 is the speed sub-reward, k arrive is a constant.

6. The method for controlling left turn at unsignalized intersection based on double-layer attention mechanism and discrete SAC algorithm according to claim 5, characterized in that: The speed sub-reward is the sum of the speed reward values ​​of each step, where the speed reward value of any step is: If the current speed is within the preconfigured speed range, the speed reward value takes a value between 0 and 1, otherwise, the speed reward value is 0.

7. The method for controlling left turn at an unsignalized intersection based on a dual-layer attention mechanism and a discrete SAC algorithm according to claim 6, characterized in that: The maximum entropy objective function is: Where: π * is the optimal strategy, π is the strategy, T is the total number of time steps, t is the time step, E ((s_t,a_t) is the mathematical expectation of the tth step, γ is the discount rate, r(s t ,a t ) is state s t and action a t The reward is α, the temperature coefficient, H(π(.|s t )) is the strategy π in state s t The strategy is the probability distribution of performing different actions in different states.

8. The method for controlling left turn at an unsignalized intersection based on a dual-layer attention mechanism and a discrete SAC algorithm according to claim 7, characterized in that: The first model is implemented using a double-layer attention mechanism and a discrete SAC algorithm. In the discrete SAC algorithm, the soft Q-function is parameterized using a neural network with parameter θ, and the soft Q-function is trained to minimize the soft Bellman residual: Among them: J Q (θ) is, D is the replay buffer of past experience, Q θ (s t ,a t ) is the soft Q function parameterized by the neural network with parameter θ, p(s t ,a t ) represents the state s t Next, perform action a t After that, the environment moves to the next state s t+1 The probability distribution of Estimation result for the target Q network; In the discrete SAC algorithm, the objective function of the SAC algorithm is modified to adapt to the output probability of π φ (a t |s t ) rather than density conditions; In the discrete SAC algorithm, the improved mathematical expression of the soft state value function is: V(s t ):=π(s t ) T [Q(s t )-alog(π(s t ))] Where: V(s t ) is the soft state value function, π(s t ) represents the state s y Under this condition, the probability distribution of each action taken by the agent is t The output under Q(s t ) represents the state s t Under this condition, the agent takes the soft Q value of each action, that is, in state s t Under this condition, the expected cumulative reward for each action taken by the agent is log(π(s t )) represents the strategy π in state s t The logarithmic probability distribution of the output under In the discrete SAC algorithm, the temperature loss is: in: It is a baseline value for entropy, used to calculate temperature loss, and is used to measure the randomness or uncertainty of a strategy.

9. The method for controlling left turn at an unsignalized intersection based on a dual-layer attention mechanism and a discrete SAC algorithm according to claim 8, characterized in that: In the sampling process of the discrete SAC algorithm, two experience replay buffers are established, one successful experience buffer and one failed experience buffer, to store successful and unsuccessful sample experiences respectively, while the successful and unsuccessful experiences are stored and replayed respectively. When a new sample needs to be stored, it is placed in the corresponding experience buffer according to its characteristics, and the earliest sample data in the buffer is replaced to ensure the balance of positive and negative sample data in the overall experience sample. In the sample retrieval process, samples are collected from the two buffers for replay according to the size of the time difference error. The mathematical expression of the experience replay process is: δ=r(s t ,a t )+γQ′(s t+1 ,a t+1 )-Q(s t ,a t ) Where: δ is the time difference error value, Q′(s t+1 ,a t+1 ) represents the target network Q in state s t+1 Next, perform action a t+1 The estimated value of s is the number of samples collected from the successful experience buffer, n f is the number of samples collected from the failure experience buffer, N is the number of samples, N s is the capacity of the success experience buffer, N f The capacity of the failure experience buffer.

10. The method for controlling left turn at unsignalized intersection based on double-layer attention mechanism and discrete SAC algorithm according to claim 9, characterized in that: The dual-layer attention mechanism includes a feature attention layer and a target attention layer. The feature attention layer adopts the architecture of the self-attention mechanism, uses the state sequence obtained from the environment as input, regards the corresponding value of each vehicle entity as the feature value of the sequence, and the input vector is linearly transformed to generate a matrix of query, key and value; The feature attention layer generates the output of the feature attention layer by weighting the value matrix with the attention weight matrix, and adds them together to generate the output of this layer. The target attention layer adopts a self-attention architecture and uniquely generates a query matrix using only the vehicle information. After calculating the attention score, the target attention layer interacts with the query matrix with the key matrix derived from all vehicle entity information.

Images