Automatic driving signal-free intersection decision generation method based on hierarchical reinforcement learning

Abstract

The disclosure provides an automatic driving signal-free intersection decision generation method based on hierarchical reinforcement learning, and relates to the technical field of vehicle driving decision. The method comprises the following steps: acquiring a real driving environment in which a host vehicle and an oncoming vehicle are located; the driving environment comprises environment observation values and an action space; inputting the real driving environment into a path decision module to obtain a decision trajectory; inputting the decision trajectory into a speed decision module to obtain a decision speed; the path decision module and the speed decision module are trained based on a simulation driving environment, a host vehicle trajectory, an end condition and a reward function; and the host vehicle performs a left turn task according to the decision trajectory and the decision speed. Training the speed decision and the path decision respectively can make the speed decision model focus on learning how to select the best left turn acceleration according to the current environment and other factors, and the path decision model focuses on finding a suitable radius that can complete the crossing, thereby improving the model performance and the accuracy of the generated decision.

Description

Technical Field

[0001] This invention belongs to the field of vehicle driving decision technology, and in particular relates to a method for generating decisions for unsignaled intersections in autonomous driving based on hierarchical reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Among various driving environments, urban intersections are a hot topic and a major challenge in autonomous driving research. As a typical complex, dynamic, and time-varying scenario, studying decision-making and planning for crossing behavior at urban intersections is crucial for the practical application of autonomous driving. Left turns at unprotected intersections, a special type of intersection behavior, are among the most complex autonomous driving scenarios due to the shared passageway for both straight-ahead and left-turning vehicles and the lack of traffic lights. The interaction between left-turning vehicles and oncoming straight-ahead vehicles during the crossing process poses significant challenges to decision-making and planning. Statistics show that over 80% of autonomous driving failures occur at intersections, with unprotected left-turn decision-making being a core challenge. The interaction conflicts arising from the mixed traffic of manned and autonomous vehicles at urban intersections are even more pronounced, making the implementation of decision-making and planning at unprotected intersections even more challenging.

[0004] While rule-based decision-making methods are logically simple, easy to implement, and can directly draw on human driving experience, they often overlook the complexity of traffic, are typically overly conservative, and have poor generalization capabilities. Decision-making methods based on POMDP and game theory consider uncertainty and interaction, but are more difficult to solve and have poor real-time performance. Traditional reinforcement learning decision-making methods do not make high utilization of real-world driving data, are not timely in acceleration and deceleration, have rigid turning paths, and struggle to generate human-like decisions, resulting in low traffic efficiency. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a decision generation method for unsignaled intersections in autonomous driving based on hierarchical reinforcement learning. For left-turn tasks at unsignaled intersections, it considers the complexity of traffic by incorporating multiple intentions of oncoming vehicles, utilizes the AWAC and D3QN algorithms to improve computational efficiency, and employs real driving data to enhance the timely setting of acceleration and deceleration during turns, thus better aligning with human decision-making.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for generating decisions for unsignalized intersections in autonomous driving based on hierarchical reinforcement learning, comprising:

[0008] The actual driving environment of the vehicle and oncoming vehicles is obtained; the driving environment includes environmental observations and action space.

[0009] The real driving environment is input into the path decision module to obtain the decision trajectory; the decision trajectory is input into the speed decision module to obtain the decision speed; the path decision module and the speed decision module are trained based on the simulated driving environment, the vehicle trajectory, the termination condition and the reward function.

[0010] The vehicle performs the left turn task based on the decision trajectory and decision speed.

[0011] Preferably, while acquiring the real driving environment of the vehicle and the oncoming vehicle, coordinate transformation is performed to transform the real driving environment from the world coordinate system to the vehicle coordinate system;

[0012] The environmental observations include the motion parameters of the vehicle and oncoming vehicles, the intersection of the vehicle's trajectory and the oncoming vehicle's trajectory, and the vehicle's speed range.

[0013] Preferably, the oncoming vehicle includes a front vehicle and a rear vehicle, and the action space includes aggressive action, yielding action and normal action;

[0014] The radical action is as follows: the vehicle in front moves forward, and the vehicle behind follows and performs the same action as the vehicle in front; the vehicle itself does not perform a left turn.

[0015] The yielding action is as follows: the vehicle in front slows down to a stop at the intersection, and the vehicle behind follows the vehicle in front and performs the same action; the vehicle behind then performs a left turn to pass through the intersection, and oncoming vehicles wait.

[0016] The typical action is as follows: the vehicle in front moves forward, and the vehicle behind slows down to a stop at the intersection; the vehicle in front makes a left turn to pass through the intersection, and the vehicle behind waits.

[0017] Preferably, the training process of the speed decision module is as follows:

[0018] The first driving environment and the vehicle's preset trajectory are obtained and input into the speed decision module for training. When the first termination condition is triggered, the first reward is calculated. If the value of the first reward reaches the set threshold, the speed decision module training is complete.

[0019] The first driving environment includes first environmental observations and a motion space; the first environmental observations include motion parameters of the vehicle and oncoming vehicles, and the intersection of the vehicle's trajectory and the oncoming vehicle's trajectory; the first termination condition includes crossing, collision, and failing to pass within the time limit; the first reward includes a first no-collision reward, a speed reward, and a no-time-out reward.

[0020] Preferably, the training process of the path decision module is as follows:

[0021] The second driving environment and the variable trajectory of the vehicle are acquired and input into the path decision module for training. When the second termination condition is triggered, the second reward is calculated. If the value of the second reward reaches the set threshold, the path decision module training is complete.

[0022] The second driving environment includes second environmental observations and action space; the second environmental observations include motion parameters of the vehicle and oncoming vehicles, the intersection of the vehicle's trajectory and the oncoming vehicle's trajectory, and the vehicle's speed range; the second termination condition includes crossing and collision; the second reward includes a second no-collision reward, a speed reward, a path change reward, and a termination reward.

[0023] Preferably, the speed decision module is trained based on the AWAC algorithm and the TD3 algorithm.

[0024] Preferably, the path decision module is trained based on the D3QN algorithm.

[0025] Secondly, the present invention provides an autonomous driving decision generation system for unsignaled intersections based on hierarchical reinforcement learning, comprising:

[0026] An environment acquisition unit is used to acquire the real driving environment of the vehicle and oncoming vehicles; the driving environment includes environmental observations and action space.

[0027] The decision acquisition unit is used to input the real driving environment into the path decision module to obtain the decision trajectory; and input the decision trajectory into the speed decision module to obtain the decision speed; the path decision module and the speed decision module are trained based on the simulated driving environment, the vehicle trajectory, the termination condition and the reward function.

[0028] The decision execution unit is used by the vehicle to perform the left turn task based on the decision trajectory and decision speed.

[0029] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for generating decisions for unsignaled intersections for autonomous driving based on hierarchical reinforcement learning as described in the first aspect.

[0030] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the method for generating decision-making data for unsignaled intersections for autonomous driving based on hierarchical reinforcement learning as described in the first aspect.

[0031] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0032] This invention provides a method for generating decision-making parameters for unsignalized intersections in autonomous driving based on hierarchical reinforcement learning. The scheme targets left-turn scenarios at unsignalized intersections, decomposing the left-turn decision-making task into two parts: lateral decision-making and longitudinal decision-making. The longitudinal decision-making objective is to provide appropriate acceleration for the left-turn task, while the lateral decision-making objective is to select an appropriate left-turn radius. The upper-level path decision-making module is implemented using the D3QN algorithm, and the lower-level speed decision-making module is implemented using the AWAC algorithm. This approach enables more efficient and human-like passage in left-turn tasks at unsignalized intersections.

[0033] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0034] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0035] Figure 1 The main flowchart of an autonomous driving decision generation method for unsignaled intersections based on hierarchical reinforcement learning provided in Embodiment 1 of this disclosure;

[0036] Figure 2 A detailed flowchart of a method for generating decisions for unsignaled intersections in autonomous driving based on hierarchical reinforcement learning, provided in Embodiment 1 of this disclosure;

[0037] Figure 3 This is a schematic diagram of the training simulation environment described in Embodiment 1 of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment discloses a method for generating decisions for unsignalized intersections in autonomous driving based on hierarchical reinforcement learning, including the following steps:

[0041] S1: Obtain the actual driving environment of the vehicle and oncoming vehicles; the driving environment includes environmental observations and action space;

[0042] S2: Input the real driving environment into the path decision module to obtain the decision trajectory; input the decision trajectory into the speed decision module to obtain the decision speed; the path decision module and the speed decision module are trained based on the simulated driving environment, the vehicle trajectory, the termination condition and the reward function;

[0043] S3: The vehicle performs a left turn based on the decision trajectory and speed.

[0044] This embodiment decomposes the left-turn decision-making task at an unsignalized intersection into two parts: lateral decision-making and longitudinal decision-making. The goal of longitudinal decision-making is to provide appropriate acceleration for the left-turn task, while the goal of lateral decision-making is to select an appropriate left-turn radius for the left-turn task.

[0045] In the hierarchical reinforcement learning of this invention, such as Figure 2 As shown, the upper-layer path decision module is implemented using the D3QN algorithm, while the lower-layer speed decision module is implemented using the AWAC algorithm. During training, the AWAC algorithm is first trained using a real dataset and TD3 augmented data. The TD3 augmented data includes randomly generated turning radii, which not only allows the algorithm to learn speed decisions under various conditions but also better couples with the real dataset. Then, with randomly generated turning radii in a fixed environment, the D3QN algorithm, under the speed control of the trained AWAC algorithm, identifies the intention of oncoming vehicles and outputs the appropriate timing for a large-radius left turn. After both the speed and path decision modules are trained, environmental information is first input into the path decision module to determine whether to change the turning radius. The result is then input into the speed decision module, which outputs the appropriate acceleration, completing the overall decision.

[0046] (I) Data Extraction and Simulation Environment Generation

[0047] 1. Data Extraction from the Dataset

[0048] This invention selects the Interaction dataset as the source of real driving data, specifically choosing scenarios numbered DR_USA_Interaction_EP0 and DR_USA_Interaction_EP1. By effectively visualizing segmented road data from different scenarios within the dataset, the left-turn data at unsignalized intersections in these scenarios was filtered.

[0049] After extracting vehicle trajectory data, to standardize the data for easier use and to facilitate the establishment of a simulation environment, a coordinate system transformation is performed to obtain the trajectory data coordinate system. The origin of the coordinate system is set at the road center point O in the environment, with the X-axis perpendicular to the vehicle lane and pointing to the right, and the Y-axis representing the vehicle lane's forward direction. The extracted data for the vehicle and oncoming vehicles are rotated in the coordinate system; if the data does not exist, it is padded with zeros.

[0050] 2. Simulation Environment Generation

[0051] First, the trajectories and positions of the driver and other vehicles in the simulation environment are designed. The coordinate system of the simulation environment is the same as the coordinate system of the trajectory data after transformation. In the coordinate transformation, the center point O of the road is used as the origin of the coordinate system, and the X-axis and Y-axis have the same direction. The driver is set as a left-turning vehicle, and two environmental vehicles are set in the opposite lane, where surround1 is the vehicle in front of the oncoming vehicle and surround2 is the vehicle behind the oncoming vehicle.

[0052] Since this invention primarily addresses the decision-making process for left turns, the initial trajectory is set by combining a straight line with a quarter circle. Figure 3 The red line represents the driving reference trajectory of the self-driving vehicle, consisting of three segments (entrance segment: straight ahead ab; left-turn segment: semicircle bc with radius R; exit segment: straight ahead cd). The blue line represents the driving trajectories of two oncoming vehicles. The point where the self-driving vehicle's trajectory intersects with the oncoming vehicle's trajectory is point P in the diagram. The straight line corresponding to A in the diagram is the starting line of the self-driving vehicle's road, and the straight line corresponding to B is the starting line of the oncoming vehicle's road.

[0053] After setting up the environment, this embodiment adds the interaction process between the vehicle and other vehicles in the environment. Based on the analysis of left-turn data in the Interaction dataset, this embodiment summarizes the possible reactions of oncoming vehicles in the opposite lane into the following three categories:

[0054] (1) Aggressive type: In order to enable the algorithm to prevent the vehicle from making a left turn in the environment and to avoid dangerous driving behavior of trying to force its way through. In this scenario, surround1 is set to move forward regardless of what behavior the vehicle does, and surround2 follows surround1.

[0055] (2) Yielding type: Based on the scenario of yielding to oncoming vehicles in the real environment, surround1 will slow down to a stop before intersection A when there are left-turning vehicles, and wait for the left-turning vehicles to pass. surround2 follows surround1.

[0056] (3) Normal type: Through observation of real data, it was found that when a left-turning vehicle approaches or exits the lane line and has forward speed, that is, when it shows the intention to cross, surround2 usually yields. Therefore, this environment type is set. When the vehicle's position is beyond the intersection A and has a certain speed, if surround2 can decelerate to a stop before the intersection A, it will make the vehicle yield to achieve a safe left turn. Otherwise, surround2 will drive normally.

[0057] The initial positions and velocities of surround1 and surround2 in the environment are randomly generated within a certain range to improve the applicability of the algorithm.

[0058] (II) Lower-level velocity decision model based on AWAC

[0059] 1. AWAC Algorithm

[0060] The AWAC algorithm still uses the method of minimizing the Bellman error to update the commentator, as shown in the following equation:

[0061] (1)

[0062] (2)

[0063] AWAC adds implicit constraints to the actors through the following process. First, it adopts a similar approach to other offline reinforcement learning algorithms, namely, constraining the policy to be similar to the policy in the prior dataset. Therefore, AWAC optimizes the policy so that the estimated Q-function at each state... The goal is to maximize the advantage function while ensuring it closely approximates the actions observed in the data. Since optimizing the advantage function achieves the same effect as the Q function, the algorithm chooses the advantage function A for optimization, resulting in the following final optimized function:

[0064] (3)

[0065] (4)

[0066] In the formula, k represents the k-th iteration. The distribution of possible strategies in the data is represented by β, which refers to the replay empirical region. The constraint below is the KL divergence.

[0067] Since directly adding hard constraints would cause offline reinforcement learning to lose its ability to fine-tune, this embodiment uses the KKT method to obtain the solution to the above equation, and its Lagrange equation is as follows:

[0068] (5)

[0069] The analytical solution to this problem can be written as In the form of. By minimizing The policy is learned by comparing the KL divergence with the analytical solution, as shown in the following formula:

[0070]

[0071] (6)

[0072] in, Let be the parameter vector. Choosing the reverse KL divergence here would result in an explicit penalty. Therefore, the algorithm here chooses forward KL, directly sampling from β to calculate the policy update, with the update formula as follows:

[0073] (7)

[0074] 2. Model Design

[0075] The training of the lower-level velocity decision module needs to consider its coordination with the subsequent training of the upper-level path planning module. Therefore, in this embodiment, segment bc from the initial reference trajectory generated by the training environment each time during the training process is used, such as... Figure 3 The radius R shown is randomly generated within the range of [5, 13.5] m, and the position of point P in the diagram will also change accordingly. Here, the maximum value of the reference trajectory radius range is chosen to be 13.5 m, which is 1 m before the starting line of the road when the left turn begins. The minimum value is chosen to be 5 m, which is derived from the following formula for the vehicle's constant-speed circular motion.

[0076]

[0077] Where R is the radius of the circular motion, and V is the vehicle speed. Where yaw rate is the vehicle's angular velocity, and K is the stability factor. For the car wheelbase, This refers to the front wheel steering angle. Calculations show that the minimum turning radius of the vehicle at the maximum speed of 8m / s set in this embodiment is approximately 5m. Therefore, 5m is chosen as the minimum value here.

[0078] By having the speed decision model turn left under different reference lines, more left-turn data with different radii can be obtained, enabling the lower-level speed decision model to cope with different turning radii selected by the upper-level path selection model during subsequent path selection training.

[0079] (1) State-space design

[0080] Assuming the vehicle knows relevant information about all oncoming vehicles in the current environment, and in order to enable the vehicle to obtain its own state and the states of both oncoming vehicles, as well as the point of conflict between the vehicle and the oncoming vehicles, the first environmental observation value designed in this embodiment is: . include , representing the x and y positions of the vehicle in the world coordinate system. and These represent the x-axis and y-axis components of the vehicle's velocity, respectively. yaw is the vehicle's heading angle.

[0081] Points include , in and These are the x and y positions of the intersection of the vehicle's reference line and the center line of the oncoming road. For example... Figure 3 The midpoint P is shown. This parameter provides the model with a potential left-turn conflict point, and this parameter also changes with the generated turning radius.

[0082] include , representing the x position, y position, heading angle, x-axis velocity component, and y-axis velocity component of the oncoming vehicle surround1. The parameters representing the oncoming vehicle's `surround2`, specifically their types and... same.

[0083] (2) Action space design

[0084] The model generates the vehicle's current acceleration. To ensure comfort, this embodiment sets the acceleration output range to [-1.5, 1.5] m / s². Since the vehicle's driving behavior is a left turn, its speed cannot be too high to meet vehicle kinematic requirements. Based on data analysis of left-turning vehicles in the extracted real driving data, this embodiment sets the maximum speed during a left turn to 8 m / s. During training, the vehicle moves along a preset reference line according to the following formula.

[0085]

[0086] In the formula, 'a' represents the acceleration obtained by the algorithm. The distance traveled along the preset path includes the straight section at the entrance, the semicircular left turn section, and the straight section at the exit; ∆t is the time interval between two frames, which is 0.1s here.

[0087] (3) Setting the end conditions of the training environment

[0088] Set the vehicle to pass Figure 3 The training process ends at point d, the endpoint of the reference line. A collision during driving will also end the training process, and the environment will automatically end if the running time exceeds 20 seconds.

[0089] (4) Reward function design

[0090] In this embodiment, the reward function design of the algorithm mainly considers vehicle efficiency and safety, and promotes algorithm convergence. A collision-free reward is set, while a collision results in a negative reward.

[0091]

[0092] To encourage vehicles to pass through intersections quickly, a simple speed reward system is implemented. The specific speed reward settings are as follows:

[0093]

[0094] Where v is the vehicle speed in each frame. The efficiency reward coefficient is set to 1 here.

[0095] To prevent vehicles from remaining stationary, this embodiment includes a timeout reward at the end of the environment. If a vehicle crosses the finish line within 20 seconds, it receives a reward of 10; if it fails to complete the crossing within the time limit, it receives a penalty of -100. Details are as follows:

[0096]

[0097] The final overall reward design is as follows:

[0098]

[0099] The interaction between the vehicle and oncoming vehicles in the training environment is as described in the action space, where aggressive, yielding, and normal actions are set in a 1:1:2 ratio. The training dataset consists of 14,620 training data sets extracted from the Interaction dataset. Simultaneously, to improve the training effect and efficiency of AWAC, and to compensate for the differences between real data and the environment set in this embodiment, the dataset was expanded using training data obtained from training in the simulation environment using TD3, resulting in a final training dataset of 100,000 sets.

[0100] The training process of the lower-level velocity decision module based on AWAC in this embodiment is as follows: the current environmental observation value is used as input, and then the commentator is updated according to the above equations (1) and (2). Then the strategy is updated according to equation (7), and the output is the vehicle acceleration.

[0101] (III) Upper-layer path selection model based on D3QN

[0102] 1. D3QN Algorithm

[0103] The D3QN algorithm, short for Dueling Double DQN, combines the features of Dueling DQN and Double DQN on the basis of DQN, achieving better results.

[0104] The Double DQN algorithm first selects the action corresponding to the maximum Q-value by estimating the network, as expressed by the following formula:

[0105] (8)

[0106] Next, the Q-value of this action is calculated using the target network, as follows:

[0107] (9)

[0108] The final formula for Q is as follows:

[0109] (10)

[0110] Where θ represents the estimated network parameters. For the target network parameters, the values ​​of the estimated network are copied to the target network at regular intervals. This update method ensures that the estimate of the true value does not change too quickly as the estimated network is continuously updated, effectively curbing the overestimation of the Q value.

[0111] Dueling DQN splits the original Q-function output into two parts: the state value function V and the action advantage function A. The state value function V is defined as follows:

[0112] (11)

[0113] The action advantage function A is defined as follows:

[0114] (12)

[0115] Directly updating V and A using the above formula poses a risk: once the neural network is trained to a fixed value of 0, the network structure loses its original purpose and becomes a simple DQN network. Therefore, normalizing A ensures that the trained V value is the average of all action Q values ​​in this state, thus avoiding the aforementioned problem. This can be achieved by subtracting the average value from the A value, as shown in the following formula:

[0116] (13)

[0117] In the formula, θ represents the neural parameters of the first part of the network, α and β represent the network parameters of the branch for calculating A and the branch for calculating V, respectively, and N is the number of selectable actions.

[0118] 2. Model Design

[0119] First, it's necessary to identify the circumstances under which a vehicle might change its travel path. Observation of real left-turn data shows that if oncoming vehicles yield during a left turn, the vehicle should quickly recognize this and make the left turn as early as possible. In this case, a large-radius left turn can reduce the overall travel distance and increase the turning radius, allowing for faster passage through the intersection. However, in other scenarios, a large-radius left turn could constitute a cutting-off, which is unsafe. Changing the left-turn radius during a turn can also make it difficult for oncoming vehicles to recognize the vehicle's intentions and hinder coordination with the speed decision model. Therefore, to ensure safety, changing the reference route is not recommended. This embodiment, to improve the adaptability and robustness of the autonomous driving system and optimize speed and path planning, designs a training environment with a variable reference route to train the D3QN algorithm.

[0120] (1) State Space Design: This embodiment assumes that the vehicle is aware of the relevant information of all oncoming vehicles in the current environment. The initial reference line segment radius in the environment is set to 7m, and the vehicle's forward acceleration is controlled by a trained AWAC-based lower-level velocity decision model. The second environmental observation consists of a 19-dimensional vector: The first four observations are the same as the AWAC settings mentioned earlier. and This indicates that when the vehicle's center is within this y-axis coordinate range, the model's output can change the reference line, and a large-radius left turn is more in line with the reality that the vehicle wants to pass quickly. Here, based on environmental constraints and experimental results, it is set to [-1, 4].

[0121] (2) Motion space design: The model outputs discrete variables 0 and 1. When the model outputs 1 and the vehicle is within the specified range ( < < The reference line will be regenerated 1m ahead of the vehicle's current position along the y-axis. If the model output is 0 or the vehicle has gone out of the specified range, the reference line will remain unchanged.

[0122] (3) Training environment termination condition setting: The training process ends when the vehicle reaches the reference line endpoint point d. If a collision occurs during the driving process, the training process will also terminate. Since the speed in this training is controlled by the trained lower-level speed decision model, there is no stagnation situation, and no timeout termination flag is set.

[0123] (4) Reward function design: The purpose of this algorithm is to enable vehicles to yield to oncoming vehicles and turn left in advance, and to quickly make a large-radius left turn when there are still oncoming vehicles refusing to yield, thereby improving the traffic efficiency. Therefore, the reward function design in this embodiment includes the efficiency of scene recognition and traffic safety, and sets a collision-free reward for task completion, where a collision will result in a negative reward.

[0124]

[0125] In this embodiment, speed is included as a small reward to help the upper-level algorithm change the left-turn radius at the most appropriate time, thereby obtaining the maximum speed reward, i.e., the highest efficiency value.

[0126]

[0127] Where v is the vehicle speed in each frame. The efficiency reward coefficient is set to 0.1 here.

[0128] To enable vehicles to identify the intentions of oncoming vehicles early and change their turning radius in dynamic environments, the rewards for vehicles in different environments need to be designed accordingly. When the vehicle is in a yielding environment, it is believed that it should be encouraged to identify the current state and change its path early, so it receives a small reward at each moment after changing its path. In other environments, while path changes are not encouraged, if the speed gain can increase its total reward, it is allowed to explore freely. Therefore, in other environments, if the vehicle chooses to change its path, it receives a negative reward at the moment of the change, and is not penalized at other times. The specific formula is as follows:

[0129]

[0130] At the same time, a completion reward is set at the end of the environment. If the vehicle can successfully reach the finish line, a simple arrival reward will be given.

[0131]

[0132] The final overall reward design is as follows:

[0133]

[0134] To increase the proportion of yielding environments in the training environment and improve training efficiency, the ratio of aggressive, yielding, and normal environments in the training environment is set at 1:2:1.

[0135] The process of training the upper-layer path selection module based on the D3QN algorithm in this embodiment is as follows: input the second environmental observation value, calculate the Q value of each action according to the formula (13); then select the action corresponding to the maximum Q value through the estimation network according to the formula (8), execute the action, and obtain the new state and reward; update the Q network according to the formula (10), and output whether to change the turning radius (discrete variables 0 and 1).

[0136] During training, the AWAC algorithm is first trained using a real dataset and TD3 augmented data. The TD3 augmented data includes randomly generated turning radii, which not only enables the algorithm to learn speed decisions under various conditions but also allows for better coupling with the real dataset. Then, using randomly generated turning radii in a fixed environment, the D3QN algorithm is trained to recognize the intentions of oncoming vehicles and output appropriate large-radius left-turn timings under speed control by the trained AWAC algorithm.

[0137] After the speed and path decision modules are trained, the real driving environment is first input into the path decision module to determine whether to change the turning radius. Then the result is input into the speed decision module to output the appropriate acceleration, thus completing the overall decision. The vehicle then performs the left turn task based on the decision.

[0138] In this embodiment, the velocity decision and path decision are trained separately, making the model more modular. Each model can focus on its specific task, improving its performance on that task. Specifically, the velocity decision model focuses on learning how to select the optimal left-turn acceleration based on the current environment and other factors, while the path decision model focuses on finding the appropriate radius for a successful crossing. Because the two models are trained separately, they can be tuned and improved independently. When one model encounters a problem, the other can still maintain a certain level of performance, enhancing the robustness of the entire system.

[0139] During the model training phase, training the speed decision model first provides the path decision model with more accurate speed information. This allows for more reasonable and accurate path planning by considering driving conditions at different speeds. Speed ​​and path decisions may vary in different driving scenarios. Training the speed decision model first enables it to better adapt to speed changes in different scenarios, allowing the path decision model to plan the optimal path based on these speed variations.

[0140] In decision-making and prediction, predicting the path first can provide a general direction and destination, and then predicting the speed based on this path can make speed prediction more targeted. This helps reduce unnecessary computation and improve prediction efficiency.

[0141] Example 2

[0142] This embodiment provides an autonomous driving decision generation system for unsignalized intersections based on hierarchical reinforcement learning, including:

[0143] An environment acquisition unit is used to acquire the real driving environment of the vehicle and oncoming vehicles; the driving environment includes environmental observations and action space.

[0144] The decision acquisition unit is used to input the real driving environment into the path decision module to obtain the decision trajectory; and input the decision trajectory into the speed decision module to obtain the decision speed; the path decision module and the speed decision module are trained based on the simulated driving environment, the vehicle trajectory, the termination condition and the reward function.

[0145] The decision execution unit is used by the vehicle to perform the left turn task based on the decision trajectory and decision speed.

[0146] Example 3

[0147] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the method for generating decisions for unsignaled intersections for autonomous driving based on hierarchical reinforcement learning as described in Embodiment 1 above.

[0148] Example 4

[0149] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the method for generating decisions for unsignaled intersections for autonomous driving based on hierarchical reinforcement learning as described in Embodiment 1 above.

[0150] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0151] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hierarchical reinforcement learning-based automatic driving signal-free intersection decision generation method, characterized in that, The method comprises the following steps: obtaining a real driving environment in which a host vehicle and an oncoming vehicle are located; the driving environment comprises environment observation values and an action space; inputting the real driving environment into a path decision module to obtain a decision trajectory, and inputting the decision trajectory into a speed decision module to obtain a decision speed; the path decision module and the speed decision module are trained based on a simulated driving environment, a host vehicle trajectory, an end condition and a reward function; the host vehicle performs a left-turn task according to the decision trajectory and the decision speed; the oncoming vehicle comprises a front vehicle and a rear vehicle, and the action space comprises an aggressive action, a yielding action and a normal action; the aggressive action is that the front vehicle drives forward, and the rear vehicle follows the front vehicle to perform the same behavior as the front vehicle; the host vehicle does not perform a left-turn behavior; the yielding action is that the front vehicle slows down to a stop at a road intersection, and the rear vehicle follows the front vehicle to perform the same behavior as the front vehicle; the host vehicle performs a left-turn behavior to pass through the road intersection, and the oncoming vehicle waits; the normal action is that the front vehicle drives forward, and the rear vehicle slows down to a stop at the road intersection; the host vehicle performs a left-turn behavior to pass through the road intersection, and the rear vehicle waits; a training process of the path decision module comprises the following steps: obtaining a second driving environment and a variable trajectory of the host vehicle, and inputting the second driving environment and the variable trajectory into the path decision module to train the path decision module; when a second end condition is triggered, a second reward is calculated; if a value of the second reward reaches a set threshold, the training of the path decision module is completed; 2. The automatic driving signalized intersection decision generation method based on hierarchical reinforcement learning according to claim 1, wherein, wherein the second driving environment comprises second environment observation values and an action space; the second environment observation values comprise motion parameters of the host vehicle and the oncoming vehicle, an intersection of a host vehicle trajectory and an oncoming vehicle trajectory, and a host vehicle speed driving range; the second end condition comprises crossing and collision; and the second reward comprises a second collision-free reward, a speed reward, a path change reward and an end reward. coordinate transformation is performed simultaneously when the real driving environment in which the host vehicle and the oncoming vehicle are located is obtained, so that the real driving environment is transformed from a world coordinate system to a host vehicle coordinate system; 3.The automatic driving signalized intersection decision generation method based on hierarchical reinforcement learning of claim 1, wherein, the environment observation values comprise motion parameters of the host vehicle and the oncoming vehicle, an intersection of a host vehicle trajectory and an oncoming vehicle trajectory, and a host vehicle speed driving range. a training process of the speed decision module comprises the following steps: obtaining a first driving environment and a preset trajectory of the host vehicle, and inputting the first driving environment and the preset trajectory into the speed decision module to train the speed decision module; when a first end condition is triggered, a first reward is calculated; if a value of the first reward reaches a set threshold, the training of the speed decision module is completed; 4.The automatic driving signalized intersection decision generation method based on hierarchical reinforcement learning of claim 1, wherein, wherein the first driving environment comprises first environment observation values and an action space; the first environment observation values comprise motion parameters of the host vehicle and the oncoming vehicle, and an intersection of a host vehicle trajectory and an oncoming vehicle trajectory; the first end condition comprises crossing, collision and timeout failure; and the first reward comprises a first collision-free reward, a speed reward and a non-timeout reward.

5. The automatic driving signalized intersection decision generation method based on hierarchical reinforcement learning according to claim 1, wherein, the speed decision module is trained based on an AWAC algorithm and a TD3 algorithm.

6. An automatic driving intersection decision generation system based on hierarchical reinforcement learning, adopting the automatic driving intersection decision generation method based on hierarchical reinforcement learning according to any one of claims 1-5, characterized in that, the path decision module is trained based on a D3QN algorithm. The method comprises the following steps: an environment acquisition unit is configured to obtain a real driving environment in which a host vehicle and an oncoming vehicle are located; the driving environment comprises environment observation values and an action space; The decision acquisition unit is configured to input a real driving environment into a path decision module to obtain a decision trajectory, and input the decision trajectory into a speed decision module to obtain a decision speed, wherein the path decision module and the speed decision module are trained based on a simulated driving environment, a trajectory of the ego vehicle, an end condition and a reward function. The decision execution unit is configured to execute a left-turn task of the ego vehicle according to the decision trajectory and the decision speed.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps in the automatic driving decision generation method for a signal-free intersection based on hierarchical reinforcement learning according to any one of claims 1-5.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the automatic driving decision generation method for a signal-free intersection based on hierarchical reinforcement learning according to any one of claims 1-5.

Images