A method and device for left turn and merging decision at unsignalized intersection and storage medium

Abstract

The present application relates to a kind of left turn and merge decision method, device and storage medium of signal intersection, the present application constructs the simulation environment of left turn and merge of signal intersection to simulate the diverse left turn and merge environment;Based on simulation environment to construct reinforcement learning environment, and define state space and action space, construct reinforcement learning reward function;State space includes the position information, speed, heading angle of ego vehicle and environment vehicle;Action space is the longitudinal acceleration of ego vehicle;The reinforcement learning reward function is composed of the sum of safety reward, efficiency reward and timeout penalty three parts;Improved reinforcement learning decision model is constructed by introducing memory unit and cross attention mechanism;The improved reinforcement learning decision model is trained using diverse simulation environment to make left turn and merge decision at signal intersection.The safety pass rate of the decision method of the present application is improved by about 10%~15%, collision rate and timeout rate significantly decrease, with significant superiority.

Description

Technical Field

[0001] This invention relates to the field of decision planning for left turns and merging at unsignalized intersections, and more particularly to a method, apparatus, and storage medium for decision planning for left turns and merging at unsignalized intersections. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Unsignaled intersections, as a common road layout, exhibit significantly higher randomness and uncertainty than other road scenarios. Randomness manifests in the randomness of vehicle arrival times, directions, and speeds, as well as the location and timing of traffic conflict points within the intersection. Uncertainty arises from the differences in speed adjustment strategies employed by vehicles when crossing the intersection. Some vehicles slow down in advance, cautiously navigating the intersection, while others may maintain higher speeds and pass quickly. With the rapid development of autonomous driving technology, its potential in alleviating traffic congestion and improving road safety is gradually emerging. However, to achieve advanced levels of autonomous driving, in-depth research into complex scenarios such as intersections is crucial. Particularly at unsignaled intersections, left turns or merging can pose dangers and lead to traffic accidents due to the intersection's unique structure.

[0004] Existing research primarily focuses on autonomous T-junction driving strategies based on connected vehicle technology. These strategies involve developing a Free-Side Module (FSM) within the motion planner, constructing a time window to manage the FSM's driving state transitions, and designing a path-following controller based on a reverse stepping method that can simultaneously track the planned path and speed. Results show that the controlled vehicle can safely and quickly pass through the intersection using the proposed driving strategy. However, the experiments only involved one environmental vehicle and did not consider the impact of multiple environmental vehicles. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a method, apparatus, and storage medium for making decisions on left turns and merging at unsignalized intersections.

[0006] This invention provides a method for left turn and merging decision-making at unsignalized intersections, including:

[0007] A simulation environment for left turns and merging at unsignalized intersections is constructed to simulate diverse left-turn and merging scenarios. The simulation environment includes: the intersection, the vehicle to be controlled, and environmental vehicles that influence the vehicle's left-turn and merging decisions.

[0008] A reinforcement learning environment is constructed based on a simulation environment, and a state space and action space are defined to construct the reinforcement learning reward function. The state space includes the position information, velocity, and heading angle of the vehicle and the surrounding vehicles; the action space is the longitudinal acceleration of the vehicle; the reinforcement learning reward function consists of the sum of three parts: safety reward, efficiency reward, and timeout penalty.

[0009] Construct an improved reinforcement learning decision model that incorporates memory units and cross-attention mechanisms;

[0010] Utilize diverse simulation environments to train and improve reinforcement learning decision-making models;

[0011] An improved reinforcement learning decision-making model, after training, is used to make left-turn and merging decisions at unsignalized intersections.

[0012] Furthermore, a simulation environment for left turns and merging at unsignalized intersections is built based on the Carla platform to simulate various left-turn and merging scenarios, including:

[0013] Construct a coordinate system to describe objects in the simulation environment;

[0014] In the constructed simulation environment, the environment vehicle and the driver vehicle are defined using a coordinate system and vehicle parameters. The environment vehicle and the driver vehicle are described by their respective states, which are determined by their positions, speeds, and heading angles. The positions are defined by the coordinate system in the simulation environment. The vehicle body range of the driver vehicle and the environment vehicle is represented by a box, where the center of the box is the vehicle position coordinate, the orientation of the box depends on the heading angle of the vehicle, and the size of the box is set according to the size of the vehicle.

[0015] An intersection is defined as a bounded area in the coordinate system;

[0016] The initial speed and position of vehicles in the environment are randomly generated, and they travel on the roads at intersections at set speeds to create diverse left-turn and merging environments.

[0017] Furthermore, the safety rewards calculated based on safety indicators are as follows:

[0018] ;

[0019] in, These are preset reward scores determined by safety indicators. The destination corresponds to the target range where the vehicle is located after turning left and merging. Whether the vehicle has reached the destination can be determined by detecting whether the vehicle has reached the target range. Whether the vehicle has collided is determined by whether the vehicle's bounding box and any other vehicle's bounding box in the environment intersect at any time.

[0020] The efficiency reward is designed based on the vehicle's current speed. The efficiency reward is as follows: , As a limiting factor, This is the vehicle's current speed;

[0021] A maximum simulation runtime threshold T is set, and a timeout penalty is defined using this maximum simulation runtime threshold T:

[0022] ;

[0023] in, These are preset reward scores determined by time indicators.

[0024] Furthermore, the improved reinforcement learning decision-making model includes: a cross-attention mechanism, a first value network, a second value network, a first target value network, a second target value network, a decision network, and a target decision network.

[0025] Furthermore, the decision network and the target decision network have the same structure, which includes stacked fully connected layers and ReLU activation functions, followed by memory units and tanh activation functions. The memory units are used to retain historical valid information of the decision network.

[0026] Furthermore, the first value network and the second value network, as well as the corresponding first target value network and second target value network, have the same structure, including: stacked fully connected layers and ReLU activation functions followed by memory units and ReLU activation functions, and finally, the value Q of the state and action is obtained by mapping through a fully connected layer.

[0027] Furthermore, the cross-attention mechanism uses the vehicle's state vector from the original state vector as the query vector. Using the environment and vehicle state vectors in the original state vector as the key Sum By using queries to calculate the relationship and correlation between the vehicle's state and the states of other vehicles in the environment that require focused attention, the vehicle's state vector and the environment's state vector are integrated into a new attention state vector after cross-attention.

[0028] Furthermore, the reinforcement learning training process for the improved reinforcement learning decision-making model is as follows:

[0029] Using stochastic model parameters , and To initialize the first value network Second value network and decision-making networks ;

[0030] Using the initial first value network Second value network and decision-making networks Initialize the first target value network Second target value network and target decision network ;

[0031] Initialize experience pool B, which is an array used to store states, actions, next states, and rewards;

[0032] Iteratively improve the training of the reinforcement learning decision model:

[0033] At any iteration step m, initialize the environment vehicle and the self vehicle to obtain the initial original state vector;

[0034] Using cross-attention mechanism to handle arbitrary time steps during left turn and merging. The original state vector is used as the query in the cross-attention mechanism, with the vehicle state vector in the original state vector as the key and the environmental vehicle state vector in the original state vector as the key and value. The query is used to calculate the relationship and correlation between the vehicle's state and the state of other environmental vehicles that need to be focused on. After cross-attention, the vehicle state vector and the environmental state vector are integrated into a new attention state vector. ;

[0035] attention state vector Decision network input to iteration step m ;

[0036] According to the output of the decision network and random Gaussian noise To generate actions That is, the longitudinal acceleration of the vehicle: ;

[0037] Based on the obtained action And reinforcement learning reward functions to obtain reward values and the original state vector for the next step t+1;

[0038] any time step The original state vector, action Reward value The original state vector for the next step is stored in experience pool B, and the number of samples in experience pool B reaches the set number.

[0039] N samples are randomly selected from experience pool B;

[0040] For each sample, the current target decision network Attention state vector based on the original state vector of the next step in the sample To calculate the target action:

[0041] truncated random Gaussian noise , where c is the cutoff threshold;

[0042] Take the current first target value network Second target value network Attention state vector based on the original state vector in the next step Calculate the current target Q value based on the target action: ,in, Discount factor;

[0043] Take the current first value network Second Value Network Attention state vector based on the current original state vector and current action Predicting value to minimize the mean squared error loss between the target Q value and the value. To update the first value network using the Adam optimizer. Second Value Network Parameters;

[0044] Based on the current first value network The model parameters θ of the decision network are updated using a deterministic policy gradient:

[0045] ,

[0046] in, Let be the decision objective function, representing the path along the first value network. The calculated value is used to update the model parameters of the decision network based on the gradient direction of the action.

[0047] The goal decision and value network are updated regularly based on the two value networks of the decision network.

[0048] Secondly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for making left turns and merging at unsignalized intersections.

[0049] Thirdly, the present invention provides a left-turn and merging decision-making device for unsignalized intersections, including at least one memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the left-turn and merging decision-making method for unsignalized intersections.

[0050] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:

[0051] This invention designs the reinforcement learning environment, reward function, and reinforcement learning architecture, and introduces a memory unit into the improved reinforcement learning decision model to enhance its temporal memory capability and improve the processing performance of continuous state data. The cross-attention mechanism introduced into the improved reinforcement learning decision model enhances the interaction modeling capability between the vehicle and surrounding vehicles, making left-turn and merging decisions at unsignalized intersections more consistent with real-world traffic environments. In Carla simulation platform tests, the proposed improved reinforcement learning decision model improves the safe pass rate by approximately 10%–15% in left-turn and merging scenarios at unsignalized intersections, while significantly reducing the collision rate and timeout rate, verifying the superiority of this application. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a method for making left turns and merging at an unsignalized intersection, provided in an embodiment of the present invention;

[0055] Figure 2 An architecture diagram of a left-turn and merging decision-making method at an unsignalized intersection provided by an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the decision network or target decision network structure in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of the first value network or first target value network, and the second value network or second target value network in an embodiment of the present invention.

[0058] Figure 5 This is a schematic diagram of the memory unit described in an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the cross-attention mechanism described in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] Example 1

[0063] like Figure 1 As shown, a method for left turn and merging decision-making at an unsignalized intersection according to the present invention includes:

[0064] A simulation environment for left turns and merging at unsignalized intersections was built based on the Carla platform to simulate various left-turn and merging scenarios. The simulation environment includes: the intersection, the vehicle to be controlled, and other vehicles influencing the vehicle's left-turn and merging decisions. The specific process includes:

[0065] Construct a coordinate system to describe objects in the simulation environment;

[0066] In the constructed simulation environment, a coordinate system and vehicle parameters are used to define the environmental vehicles and the driver vehicle. The states of the environmental and driver vehicles are described by their position, velocity, and heading angle, with position defined by the coordinate system within the simulation environment. A bounding box represents the vehicle's and the environment vehicle's body extent, where the center of the box represents the vehicle's position coordinates, the box's orientation depends on the vehicle's heading angle, and the box's size is set according to the vehicle's size. For example... Figure 1 In the simulation environment at the bottom right, vehicles marked with dashed boxes are the driver's vehicles, while vehicles marked with solid boxes are environment vehicles.

[0067] In the constructed simulation environment, intersections are defined using a coordinate system region; an intersection is defined as a bounded region within the coordinate system. For example... Figure 1 The simulation environment in the lower right corner shows a signalless intersection including T-shaped and Y-shaped intersections.

[0068] The initial speed and position of vehicles in the environment are randomly generated, and they travel on the roads at intersections at set speeds to create diverse left-turn and merging environments.

[0069] Under the control of the improved reinforcement learning decision model constructed in this invention, the vehicle performs left turn or merge tasks in various left turn and merge environments, in order to achieve safe and efficient left turn and merge decisions.

[0070] We construct a reinforcement learning environment based on a simulation environment, define the state space and action space, and construct the reward function for reinforcement learning.

[0071] In the specific implementation of this application, the state space includes the position information, speed, and heading angle of the vehicle and the position information, speed, and heading angle of the environment vehicles. For both the vehicle and the environment vehicles, the position information includes x-axis and y-axis coordinates in the coordinate system of the simulation environment. During the simulation, the position information, speed, and heading angle of the vehicle and the environment vehicles in the simulation environment are acquired and concatenated to form a high-dimensional state vector.

[0072] The action space is the longitudinal acceleration of the vehicle. a It is used to control the longitudinal acceleration, braking or cruising of the vehicle when turning left or merging, so as to achieve precise control of the longitudinal speed of the vehicle when turning left or merging.

[0073] To achieve safe and efficient left turn and merge decisions, the reinforcement learning reward function of this application consists of three parts: safety reward, efficiency reward, and timeout penalty. Among them, the safety reward is a positive reward for reaching the destination and a penalty for collision; the efficiency reward is proportional to the speed; and the timeout penalty is defined as a negative reward for failing to complete the task within the time limit.

[0074] To ensure driving safety, this invention sets a penalty reward that is relatively larger than any positive reward score for safety rewards, thereby encouraging the improved reinforcement learning decision-making model to prioritize collision avoidance. The safety reward calculated based on safety indicators is as follows:

[0075] ;

[0076] in, These are preset reward scores determined by safety indicators.

[0077] The destination corresponds to the target area where the vehicle is located after turning left and merging; whether the vehicle has reached the target area can be determined by detecting whether the vehicle has reached the destination; whether the vehicle has collided is determined by whether the vehicle's bounding box and any other vehicle's bounding box in the environment intersect at any given time.

[0078] The efficiency bonuses calculated based on efficiency indicators are as follows:

[0079] To evaluate the efficiency of the completed task, this invention uses the vehicle's current speed to design an efficiency bonus. To prevent excessive speed from interfering with the vehicle's passage through the intersection, a limiting coefficient is added to the speed. The efficiency reward is: .

[0080] The timeout penalty calculated based on the time index is as follows:

[0081] To encourage safe rather than conservative behavior, this invention sets a maximum simulation runtime threshold T. One example of this threshold is 30 seconds, and the maximum simulation runtime threshold T is used to define a timeout penalty.

[0082] ;

[0083] in, These are preset reward scores determined by time indicators.

[0084] During the simulation, after the vehicle performs any action, it updates the state of the simulation environment and obtains the corresponding reward through the reinforcement learning reward function mentioned above.

[0085] This invention constructs an improved reinforcement learning decision-making model that incorporates memory units and a cross-attention mechanism. This improved reinforcement learning decision-making model addresses the problems of high collision rates and low intersection efficiency caused by the randomness of oncoming traffic from one side of an unsignalized intersection and the untimely adjustment of vehicle acceleration.

[0086] In the specific implementation process, such as Figure 1 As shown, the improved reinforcement learning decision model includes:

[0087] Cross-attention mechanism, first value network, second value network, first target value network, second target value network, decision network, and target decision network.

[0088] The improved reinforcement learning decision-making model includes a dual-value network and a decision network. Memory units are introduced into both networks to provide memory functions and retain historical information meaningful for left-turn and merging decisions. Simultaneously, a cross-attention mechanism is introduced to focus on the interaction between the vehicle and surrounding vehicles. Both the memory units and the cross-attention mechanism enhance the improved reinforcement learning decision-making model's understanding of the left-turn and merging environment at unsignalized intersections. This, combined with deep reinforcement learning, addresses the decision-making problem at unsignalized intersections considering left-turn or merging scenarios involving vehicles approaching from one side.

[0089] Cross-attention mechanisms are used to model the interaction between the autonomous vehicle and other vehicles in the environment; such as Figure 6 As shown, the cross-attention mechanism uses the vehicle's state vector as the query vector. , , These are the vehicle state vectors. The system uses the vehicle's x-axis coordinates, y-axis coordinates, velocity, and heading angle; a cross-attention mechanism calculates the relationship and correlation between the vehicle's state and other environmental vehicles requiring focused attention using queries; and uses the environmental vehicle state vector as the key. Sum Taking two environmental vehicles as an example, the keys and values ​​are the state vectors of the two environmental vehicles:

[0090] ,

[0091] in, These are two environmental vehicle state vectors. The system includes the x-axis and y-axis coordinates, velocity, and heading angle of the two vehicles in the environment; it also contains key state information regarding interactions and action choices between the environmental vehicles and the driver vehicle. By dynamically focusing attention weights on the most critical environmental vehicles, the model's ability to model interactive behaviors is enhanced.

[0092] The formula for calculating the cross-attention mechanism is as follows:

[0093] ;

[0094] in, This is the scaling factor.

[0095] After cross-attention, the vehicle's state vector and the environment's state vector are integrated into a new state vector, called the attention state vector.

[0096] Decision Network and target decision network The structure is consistent, the target decision network is the goal that the decision network wants to achieve, and the parameters of the decision network are periodically changed. Update the parameters of the target decision network .like Figure 3 As shown, the structures of the decision network and the target decision network contain stacked fully connected layers and ReLU activation functions. Figure 3 The example uses two stacked fully connected layers and a ReLU activation function, followed by a memory unit and a tanh activation function. The memory unit is used to retain the historical effective information of the decision network, which solves the problem of ignoring temporal dependencies in sequential decision problems and enhances the temporal dependency of the model.

[0097] The decision network The action, i.e., the longitudinal acceleration of the vehicle, is obtained based on the current attention state vector after cross-attention. Specifically, the decision network outputs... Combined with random Gaussian noise Obtain the longitudinal acceleration of the vehicle a : , This is the attention state vector input to the decision network. Random Gaussian noise. Obey the average value of 0 The target decision network outputs a Gaussian distribution with standard deviation denoted as . Combined with truncated random Gaussian noise Obtain the target motion, namely the longitudinal acceleration of the target vehicle. : 。 Truncated random Gaussian noise Where c is the cutoff threshold, This is the attention state vector input to the target decision network. This is a truncation function.

[0098] Maintain two independent first value networks Second Value Network and the corresponding first target value network Second target value network .like Figure 4 As shown, the first value network and the second value network, as well as the corresponding first target value network and second target value network, have the same structure, including: stacked fully connected layers and ReLU activation functions. Figure 4 The example uses two stacked fully connected layers and a ReLU activation function, followed by a memory cell and a ReLU activation function, and finally the value is obtained by mapping through a fully connected layer.

[0099] like Figure 5 As shown, one example memory unit employs a gated recurrent unit (GRU). The GRU uses update gates and reset gates to control the flow of information, resulting in lower computational cost. The GRU includes reset gates and update gates. These gates together determine how the hidden state at each time step is updated based on the current input and the hidden state from the previous time step. Specifically, the reset gate determines which information in the hidden state from the previous time step should be ignored. The calculation formula for the reset gate is: ,in, For the Sigmoid function, and The inputs at the current time t are respectively and the hidden state of the previous moment t-1 To the weight matrix of the reset gate, This is to reset the gate bias term. The purpose of the update gate is to determine how much information from the previous hidden state should be retained and passed to the current state. The formula for calculating the update gate is: ,in, and The inputs at the current time t are respectively and the hidden state of the previous moment t-1 To update the weight matrix of the gate, To update the gate bias term, candidate hidden states are calculated based on the current input and the previous hidden state after the gate reset adjustment: ,in, The hyperbolic tangent activation function is used. and The inputs at the current time t are respectively The hidden state of the previous moment t-1 after the door was reset and adjusted. The weight matrix to the candidate hidden state, This represents the Hadamard product, which is the element-wise multiplication. It updates the hidden state of the gate pair at the previous time step t-1. and candidate hidden state By performing a weighted combination, we obtain the final hidden state at time t: The gated recurrent unit (GRU) memorizes useful information from each process and passes it to the next GRU, preserving meaningful historical information. By introducing the GRU into the decision network and value network, it provides the model with a memory function, thereby improving the problems of slow convergence speed and low safe passage rate when facing the randomness and complexity of intersections.

[0100] Combination Figure 2 As shown, various simulation environments are used to train and improve the reinforcement learning decision-making model. The reinforcement learning training process of the improved reinforcement learning decision-making model is as follows:

[0101] Using stochastic model parameters , and To initialize the first value network Second value network and decision-making networks .

[0102] Initialize the first target value network, the second target value network, and the target decision network using the initialized first value network, second value network, and decision network. Specifically, the first value network... Model parameters Assigned to the first target value network Model parameters The second value network Model parameters Assigned to the second target value network Model parameters , decision network Model parameters Assigned to the target decision network Model parameters .

[0103] Initialize experience pool B. Experience pool B stores arrays of states, actions, next states, and rewards. During training, the improved reinforcement learning decision model receives a current state vector containing information about the reinforcement learning environment and a reward calculated based on the previous state to find a suitable action. The improved reinforcement learning decision model then executes the found action in the next state. The simulation environment sends the improved reinforcement learning decision model the next state and reward after the action execution to find the next action, and this process is repeated continuously. Each set of arrays of states, actions, next states, and rewards is stored in experience pool B to serve the training process of the improved reinforcement learning decision model.

[0104] Iteratively improve the training of the reinforcement learning decision model:

[0105] At any iteration step m, initialize the environment vehicle and the self vehicle to obtain the initial original state vector.

[0106] Using cross-attention mechanism to handle arbitrary time steps during left turn and merging. The original state vector is used as the query in the cross-attention mechanism, with the vehicle state vector in the original state vector as the key and the environmental vehicle state vector in the original state vector as the key and value. The query is used to calculate the relationship and correlation between the vehicle's state and the state of other environmental vehicles that need to be focused on. After cross-attention, the vehicle state vector and the environmental state vector are integrated into a new attention state vector. ;

[0107] attention state vector Decision network input to iteration step m .

[0108] According to the output of the decision network and random Gaussian noise To generate actions That is, the longitudinal acceleration of the vehicle: .

[0109] Based on the obtained action And reinforcement learning reward functions to obtain reward values and the original state vector for the next step t+1. For any time step... The original state vector, action Reward value The original state vector for the next step is stored in experience pool B, and the number of samples in experience pool B reaches the set number.

[0110] N samples are randomly selected from the experience pool B.

[0111] For each sample, the current target decision network Attention state vector based on the original state vector of the next step in the sample To calculate the target action:

[0112] Truncated random Gaussian noise , where c is the truncation threshold.

[0113] Take the current first target value network Second target value network Attention state vector based on the original state vector in the next step Calculate the current target Q value based on the target action: ,in, This is the discount factor.

[0114] Take the current first value network Second Value Network Attention state vector based on the current original state vector and current action Predicting value to minimize the mean squared error loss between the target Q value and the value. To update the first value network using the Adam optimizer. Second Value Network The parameters.

[0115] Based on the current first value network The model parameters θ of the decision network are updated using a deterministic policy gradient:

[0116] ,

[0117] in, Let be the decision objective function, representing the path along the first value network. The calculated value of the gradient direction of the action is used to update the model parameters of the decision network.

[0118] Update the target network regularly:

[0119] ,

[0120] ;

[0121] in, The target network update rate.

[0122] During reinforcement training, due to function approximation errors, the value network may overestimate the Q-values ​​of certain actions in some environments. The decision network then tends to select these overestimated actions, leading to the accumulation of errors and ultimately causing the algorithm to learn a suboptimal policy. To reduce Q-value overestimation, the minimum of the outputs of the two target value networks is taken when calculating the target Q-value. This minimum-value operation always tends to select the more conservative Q-value from the outputs of the two target value networks. This effectively suppresses Q-value overestimation and is beneficial for optimal decision-making.

[0123] To ensure more accurate Q-value estimation before updating the policy and reduce the risk of suboptimal updates when the value function is inaccurate, the decision network is updated only once after a predetermined number of updates to the value network. In the early stages of training, the value network itself is highly inaccurate. Frequent updates to the decision network at this stage would cause it to optimize in the wrong direction, introducing more error. Updating the decision network and the target decision network only once after a predetermined number of updates to the value network gives the value network more time to converge to a more accurate value estimate, which is then used to guide the improvement of the decision network.

[0124] An improved reinforcement learning decision-making model, after training, is used to make left-turn and merging decisions at unsignalized intersections.

[0125] Comparative experiments were conducted to compare this application with existing algorithms such as DDPG, D4PG, and TD3. The results of the comparative experiments show that this invention significantly outperforms existing algorithms in terms of security and robustness. The results of the comparative experiments are shown in the table below:

[0126]

[0127] Ablation experiments were conducted to remove memory units or cross-attention modules to determine their contribution to the proposed scheme.

[0128] Example 2

[0129] This invention provides a left-turn and merging decision-making device for unsignalized intersections, comprising: at least one memory and a processor; the memory stores a computer program executable on the processor, and the processor, when executing the program, implements the left-turn and merging decision-making method for unsignalized intersections, including:

[0130] A simulation environment for left turns and merging at unsignalized intersections is constructed to simulate diverse left-turn and merging scenarios. The simulation environment includes: the intersection, the vehicle to be controlled, and environmental vehicles that influence the vehicle's left-turn and merging decisions.

[0131] A reinforcement learning environment is constructed based on a simulation environment, and a state space and action space are defined to construct the reinforcement learning reward function. The state space includes the position information, velocity, and heading angle of the vehicle and the surrounding vehicles; the action space is the longitudinal acceleration of the vehicle; the reinforcement learning reward function consists of the sum of three parts: safety reward, efficiency reward, and timeout penalty.

[0132] Construct an improved reinforcement learning decision model that incorporates memory units and cross-attention mechanisms;

[0133] Utilize diverse simulation environments to train and improve reinforcement learning decision-making models;

[0134] An improved reinforcement learning decision-making model, after training, is used to make left-turn and merging decisions at unsignalized intersections.

[0135] Of course, the computer program stored in the memory of the unsignalized intersection left turn and merge decision device provided in the embodiments of the present invention is not limited to the operation of the method described above, and can also execute related operations in the unsignalized intersection left turn and merge decision method provided in any embodiment of the present invention.

[0136] Example 3

[0137] This invention provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the left-turn and merging decision-making method at an unsignalized intersection, including:

[0138] A simulation environment for left turns and merging at unsignalized intersections is constructed to simulate diverse left-turn and merging scenarios. The simulation environment includes: the intersection, the vehicle to be controlled, and environmental vehicles that influence the vehicle's left-turn and merging decisions.

[0139] A reinforcement learning environment is constructed based on a simulation environment, and a state space and action space are defined to construct the reinforcement learning reward function. The state space includes the position information, velocity, and heading angle of the vehicle and the surrounding vehicles; the action space is the longitudinal acceleration of the vehicle; the reinforcement learning reward function consists of the sum of three parts: safety reward, efficiency reward, and timeout penalty.

[0140] Construct an improved reinforcement learning decision model that incorporates memory units and cross-attention mechanisms;

[0141] Utilize diverse simulation environments to train and improve reinforcement learning decision-making models;

[0142] An improved reinforcement learning decision-making model, after training, is used to make left-turn and merging decisions at unsignalized intersections.

[0143] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for left turn and merging decision-making at an unsignalized intersection, characterized in that, include: A simulation environment for left turns and merging at unsignalized intersections is constructed to simulate diverse left-turn and merging scenarios. The simulation environment includes: the intersection, the vehicle to be controlled, and environmental vehicles that influence the vehicle's left-turn and merging decisions. A reinforcement learning environment is constructed based on a simulation environment, and a state space and action space are defined to construct the reinforcement learning reward function. The state space includes the position information, velocity, and heading angle of the vehicle and the surrounding vehicles; the action space is the longitudinal acceleration of the vehicle; the reinforcement learning reward function consists of the sum of three parts: safety reward, efficiency reward, and timeout penalty. An improved reinforcement learning decision-making model is constructed by introducing memory units and a cross-attention mechanism. The improved model includes: a cross-attention mechanism, a first value network, a second value network, a first target value network, a second target value network, a decision network, and a target decision network. The decision network and the target decision network have the same structure, consisting of stacked fully connected layers and ReLU activation functions, followed by memory units and tanh activation functions. The memory units are used to retain historical valid information of the decision network. The first and second value networks, as well as their corresponding first and second target value networks, have the same structure, including stacked fully connected layers and ReLU activation functions, followed by memory units and ReLU activation functions, and finally, a fully connected layer maps the values ​​Q of the state and action. The cross-attention mechanism uses the vehicle state vector in the original state vector as the query. Using the environment and vehicle state vectors in the original state vector as the key Sum By using queries to calculate the relationship and correlation between the vehicle's state and the states of other vehicles in the environment that require focused attention, the vehicle's state vector and the environment's state vector after cross-attention are integrated into a new attention state vector. Utilize diverse simulation environments to train and improve reinforcement learning decision-making models; An improved reinforcement learning decision-making model, after training, is used to make left-turn and merging decisions at unsignalized intersections.

2. The method for left turn and merging decision-making at an unsignalized intersection according to claim 1, characterized in that, A simulation environment for left turns and merging at unsignalized intersections was built based on the Carla platform to simulate various left turn and merging scenarios, including: Construct a coordinate system to describe objects in the simulation environment; In the constructed simulation environment, the environment vehicle and the driver vehicle are defined using a coordinate system and vehicle parameters. The environment vehicle and the driver vehicle are described by their respective states, which are determined by their positions, speeds, and heading angles. The positions are defined by the coordinate system in the simulation environment. The vehicle body range of the driver vehicle and the environment vehicle is represented by a box, where the center of the box is the vehicle position coordinate, the orientation of the box depends on the heading angle of the vehicle, and the size of the box is set according to the size of the vehicle. An intersection is defined as a bounded area in the coordinate system; The initial speed and position of vehicles in the environment are randomly generated, and they travel on the roads at intersections at set speeds to create diverse left-turn and merging environments.

3. The method for left turn and merging decision-making at an unsignalized intersection according to claim 1, characterized in that, The safety rewards calculated based on safety indicators are as follows: ； in, These are preset reward scores determined by safety indicators. The destination corresponds to the target range where the vehicle is located after turning left and merging. Whether the vehicle has reached the destination can be determined by detecting whether the vehicle has reached the target range. Whether the vehicle has collided is determined by whether the vehicle's bounding box and any other vehicle's bounding box in the environment intersect at any time. The efficiency reward is designed based on the vehicle's current speed. The efficiency reward is as follows: , As a limiting factor, This is the vehicle's current speed; A maximum simulation runtime threshold T is set, and a timeout penalty is defined using this maximum simulation runtime threshold T: ； in, These are preset reward scores determined by time indicators.

4. The method for left turn and merging decision-making at an unsignalized intersection according to claim 1, characterized in that, The reinforcement learning training process for the improved reinforcement learning decision-making model is as follows: Using stochastic model parameters , and To initialize the first value network Second value network and decision-making networks ; Using the initial first value network Second value network and decision-making networks Initialize the first target value network Second target value network and target decision network ; Initialize experience pool B, which is an array used to store states, actions, next states, and rewards; Iteratively improve the training of the reinforcement learning decision model: At any iteration step m, initialize the environment vehicle and the self vehicle to obtain the initial original state vector; Using cross-attention mechanism to handle arbitrary time steps during left turn and merging. The original state vector is used as the query in the cross-attention mechanism, with the vehicle state vector in the original state vector as the key and the environmental vehicle state vector in the original state vector as the key and value. The query is used to calculate the relationship and correlation between the vehicle's state and the state of other environmental vehicles that need to be focused on. After cross-attention, the vehicle state vector and the environmental state vector are integrated into a new attention state vector. ; attention state vector Decision network input to iteration step m ; According to the output of the decision network and random Gaussian noise To generate actions That is, the longitudinal acceleration of the vehicle: ; Based on the obtained action And reinforcement learning reward functions to obtain reward values and the original state vector for the next step t+1; any time step The original state vector, action Reward value The original state vector for the next step is stored in experience pool B, and the number of samples in experience pool B reaches the set number. N samples are randomly selected from experience pool B; For each sample, the current target decision network Attention state vector based on the original state vector of the next step in the sample To calculate the target action: truncated random Gaussian noise , where c is the cutoff threshold; Take the current first target value network Second target value network Attention state vector based on the original state vector in the next step Calculate the current target Q value based on the target action: ,in, Discount factor; Take the current first value network Second Value Network Attention state vector based on the current original state vector and current action Predicting value to minimize the mean squared error loss between the target Q value and the value. To update the first value network using the Adam optimizer. Second Value Network Parameters; Based on the current first value network Update the model parameters θ of the decision network using a deterministic policy gradient: ， in, Let be the decision objective function, representing the path along the first value network. The calculated value is used to update the model parameters of the decision network based on the gradient direction of the action. The goal decision and value network are updated regularly based on the two value networks of the decision network.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the left turn and merge decision method for unsignaled intersections as described in any one of claims 1-4.

6. A left-turn and merging decision-making device for a signalless intersection, comprising at least one memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the left turn and merge decision method for unsignaled intersections as described in any one of claims 1-4.

Images