A hierarchical vehicle driving decision-making method based on multiple time scales

Abstract

The present application relates to a kind of hierarchical vehicle driving decision-making methods based on multi-time scale, depth hierarchical reinforcement learning decision model combined by high-level model and low-level model, the driving behavior of target vehicle is planned according to the environmental observation information by high-level model, then corresponding low-level model is selected based on driving behavior, and vehicle posture is adjusted according to the decision of driving behavior and environmental observation information, complex driving decision-making task is decomposed into diversified driving behavior, different traffic road environment is adapted, so that automatic driving vehicle realizes hierarchical control, better satisfy the diversification demand of automatic driving in safety and efficiency;Meanwhile, pre-training low-level model is used, then the training mode of high-level model and low-level model is combined, effectively avoid the option collapse problem in the joint training process of high-level model and low-level model, ensure the stability of entire decision system.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and more specifically, to a hierarchical vehicle driving decision-making method based on multiple time scales. Background Technology

[0002] The development and promotion of autonomous vehicles have provided new ideas and methods for improving road safety, increasing traffic efficiency, and building intelligent transportation systems.

[0003] In autonomous driving technology, connected vehicles utilize perception information to comprehensively assess their own driving status and external risks, thereby deciding on driving behaviors appropriate to the current traffic scenario. This is the technological foundation for achieving safe and reliable operation of autonomous vehicles. Therefore, how to make real-time and accurate driving decisions based on real-time changes in the traffic environment has become a hot topic and challenge in current autonomous driving technology research.

[0004] Currently, driving decision-making methods can be broadly categorized into two types: rule-based and learning-based methods.

[0005] Rule-based methods refer to autonomous vehicles making driving decisions based on fixed rules and conditions. Finite State Machines (FSMs) are a typical example. In an FSM model, an autonomous vehicle can be in one of a finite number of states, such as driving, parking, or turning. The transitions between states are determined by the current input (such as sensor data, traffic signals, etc.) and the previous state. Once a specific condition is met, the vehicle transitions from the current state to the next state and performs the corresponding driving action. Thanks to the clear state definitions and transition rules, finite state machines perform well in ideal scenarios (such as good road conditions and stable traffic flow).

[0006] With the rapid development of machine learning (ML) and deep learning (DL) algorithms, learning-based driving decision-making methods have become a research hotspot in the field of autonomous driving. Deep learning [2] can effectively extract high-level abstract features from data by constructing deep neural network models, providing powerful data processing capabilities and accurate pattern recognition capabilities for autonomous driving systems.

[0007] On the other hand, reinforcement learning (RL), based on the theory of Markov Decision Process (MDP), solves the decision-making problem of intelligent agents in continuous time series. Through trial and error with the environment, reinforcement learning enables autonomous vehicles to continuously optimize their driving strategies and achieve autonomous learning of driving decisions.

[0008] However, traditional reinforcement learning has limitations in its application to high-dimensional state spaces. To overcome this challenge, deep reinforcement learning (DRL) combines the perceptual capabilities of deep learning with the decision-making capabilities of reinforcement learning, using neural networks to efficiently extract and process continuous high-dimensional state information, such as road environment, vehicle dynamics, and traffic signals. This enables autonomous vehicles to make more accurate driving decisions.

[0009] Despite significant progress in rule-based and learning-based methods for vehicle driving decision-making, several challenges remain in practical applications:

[0010] First, real-world traffic scenarios are highly dynamic and uncertain. Rule-based methods, constrained by fixed rules and conditions, struggle to create a safety rule base that covers all possible traffic scenarios. Furthermore, these methods lack flexibility, only addressing pre-defined situations and failing to handle exceptional circumstances. Therefore, rule-based methods have limitations when dealing with uncertain driving scenarios and may fail to make timely and accurate decisions.

[0011] On the other hand, deep reinforcement learning algorithms typically employ a single driving strategy in practical applications. However, due to the diverse efficiency and safety requirements of real-world traffic scenarios, a single strategy often struggles to meet the needs of all scenarios. This leads to a decline in algorithm performance, which in turn affects the stability and reliability of the autonomous driving system. Summary of the Invention

[0012] The technical problem this invention aims to solve is how to enable autonomous vehicles to make more intelligent and safer driving decisions based on traffic conditions, thereby improving the adaptability of autonomous vehicles to the environment and the reliability of their decisions.

[0013] This invention provides a hierarchical vehicle driving decision-making method based on multiple time scales, comprising:

[0014] Step 1. Obtain the coordinates of the destination area for the target vehicle's driving mission. And acquire environmental observation information of the target vehicle;

[0015] Step 2. Construct a deep hierarchical reinforcement learning decision model, which includes a high-level model and a low-level model. The high-level model is used to make the optimal driving behavior according to different environments. Each driving behavior corresponds to a low-level model, which is used to perform attitude control on the target vehicle.

[0016] Step 3. Train the low-level model using the TD3 algorithm;

[0017] Step 4. Use the OC algorithm to jointly train the high-level model and the low-level model trained in Step 3;

[0018] Step 5. Based on the environmental observation information obtained by the target vehicle, obtain the driving behavior decision of the target vehicle through a high-level model;

[0019] Step 6. The target vehicle loads the corresponding low-level model based on the driving behavior decision output in Step 5, and combines it with environmental observation information to perform attitude control on the target vehicle through the low-level model.

[0020] Compared with existing technologies, this application has the following advantages: It adopts a deep hierarchical reinforcement learning decision model composed of a high-level model and a low-level model. The high-level model macroscopically plans the driving behavior of the target vehicle based on environmental observation information, and then selects the corresponding low-level model based on the driving behavior. The vehicle posture is adjusted according to the decision of the driving behavior and environmental observation information, decomposing the complex driving decision task into diversified driving behaviors to adapt to different traffic road environments. This enables the autonomous vehicle to achieve hierarchical control and better meet the diversified needs of autonomous driving in terms of safety and efficiency. At the same time, the method of pre-training the low-level model and then jointly training the high-level and low-level models effectively avoids the option collapse problem that occurs during the joint training of the high-level and low-level models, ensuring the stability of the entire decision system.

[0021] In one possible implementation, the target vehicle environmental observation information obtained in step 1 is represented as follows:

[0022] ;

[0023] In the formula, These represent the current coordinates of the target vehicle. and Indicates the current time the target vehicle is in and Velocity components in two directions; , indicating environmental vehicles The state vector, This indicates the number of vehicles in the environment surrounding the target vehicle. Indicates environmental vehicles The coordinates of the location; This indicates the horizontal distance between the target vehicle and the center line of its current lane; This indicates the distance between the target vehicle and the nearest ambient vehicle in each lane.

[0024] In one possible implementation, the state space of the high-level model is represented as:

[0025] ;

[0026] The action space of the high-level model represents the driving behavior of the target vehicle at the current moment.

[0027] ;

[0028] In the formula, Indicates aggressive driving behavior. This represents conservative driving behavior, and the driving behavior decision output by the high-level model is either aggressive or conservative driving behavior.

[0029] The reward function of the high-level model is the environmental reward function, which includes collision penalty. Penalties for exceeding road boundaries Advance distance reward and penalties and rewards for deviating from the center line Specifically, it is expressed as:

[0030] ;

[0031] In the formula, collision penalty This represents the collision penalty between the target vehicle and other vehicles in the environment; specifically:

[0032] ;

[0033] Penalty for exceeding road boundaries This represents the collision penalty between the target vehicle and the road boundary barrier, specifically as follows:

[0034] ;

[0035] Forward Distance Bonus The reward or penalty is expressed as the effective distance the target vehicle travels along the road direction, specifically as follows:

[0036] ;

[0037] in, Indicates the time it takes for the target vehicle to travel along the road;

[0038] Off-center line penalties and rewards Penalty for a target vehicle deviating from the center line of the road:

[0039] ;

[0040] in, The width of the lane where the target vehicle is located.

[0041] In one possible implementation, the high-level model further includes a termination network and a Q-network. The termination network is used to calculate and output the termination probability of the target vehicle's driving behavior at the current moment based on the state space at the current moment, and the Q-network is used to calculate and output the Q value of each driving behavior of the target vehicle based on the state space at the current moment.

[0042] The low-level model includes a policy network module and a dual-objective evaluation network module. The policy network module generates actions based on the state space with the assistance of the dual-objective evaluation network module. The policy network module includes a policy network and a target policy network. The dual-objective evaluation network module includes a first evaluation network, a first target evaluation network, a second evaluation network, and a second target evaluation network.

[0043] In one possible implementation, the state space of the low-level model is represented as:

[0044] ;

[0045] The action space of the low-level model is the current speed of the target vehicle:

[0046] ;

[0047] In the formula, and These respectively indicate that the target vehicle is in and Decision velocity components in two directions, Indicates that the target vehicle is in Maximum speed in this direction Indicates that the target vehicle is in Maximum speed in the direction of travel;

[0048] The reward function of the low-level model includes an environmental reward function that reflects driving safety and an auxiliary reward function that reflects driving efficiency.

[0049] In one possible implementation, the environmental reward function of the low-level model is consistent with the environmental reward function of the high-level model, expressed as:

[0050] ;

[0051] The auxiliary reward function includes lane change reward. Safe distance penalty Early arrival reward and the range of motion punishment ,in:

[0052] Lane change reward Represented as:

[0053] ;

[0054] In the formula, The target lane that the target vehicle needs to change to. A constant threshold for determining whether lane-changing behavior is appropriate;

[0055] Safe distance penalty Represented as:

[0056] ;

[0057] In the formula, The lane where the target vehicle is located. It is a safe distance preset based on traffic rules and actual conditions;

[0058] Early arrival bonus Represented as:

[0059] ;

[0060] In the formula, t For the current moment, Maximum control duration;

[0061] Punishment for range of motion Represented as:

[0062] ;

[0063] The reward function of the low-level model Represented as:

[0064] ;

[0065] In the formula, These are the weighting coefficients.

[0066] In one possible implementation, step 3, training the low-level model using the TD3 algorithm, specifically includes:

[0067] Step 301. Initialize policy network parameters First evaluation network parameters Second evaluation network parameters Experience replay pool and maximum control duration ;

[0068] Step 302. Input the current time into the low-level model. status The policy network is based on the state Generate Actions ;

[0069] Step 303. Action-based Interacting with the driving environment to obtain the next moment status And through the reward function of the low-level model Receive the reward at the current moment ; Obtain empirical datasets Save it to the experience replay pool ,and , ;

[0070] Step 304. Determine the experience replay pool The amount of empirical data in Is it greater than the preset sampling experience batch size? If yes, proceed to step 305; otherwise, return to step 302.

[0071] Step 305. From the experience replay pool A low-level model is trained by sampling a batch of empirical datasets and minimizing the actions. The mean square error of the Q-value is used to update the parameters of the first and second evaluation networks; the expression is:

[0072] ;

[0073] ;

[0074] In the formula, As a discount factor, The total amount of empirical data sampled. This is motion noise;

[0075] Step 306. Update the policy network parameters using policy gradients. The expression is:

[0076] ;

[0077] Step 307. Update the target policy network parameters, the first target evaluation network parameters, and the second target evaluation network parameters using a soft update method, with the following expression:

[0078] ;

[0079] In the formula, To update the parameter weights;

[0080] Step 308. Determine if the training termination condition is met. If yes, end the training; otherwise, return to step 302.

[0081] In one possible implementation, the training termination condition in step 308 is:

[0082] Condition A1. This indicates that the target vehicle has left the road boundary;

[0083] Condition A2. This indicates that the target vehicle collided with an environmental vehicle.

[0084] Condition A3. , indicating the current time Reaching maximum control duration ;

[0085] Condition A4. This indicates that the target vehicle has arrived at the destination area;

[0086] Training ends when any of the above conditions are met.

[0087] In one possible implementation, step 4, which involves using the OC algorithm to jointly train the high-level model and the low-level model trained in step 3, specifically includes:

[0088] Step 401. Initialize the termination network parameters of the high-level model. Q network parameters Ω, driving behavior, experience replay pool and maximum control duration ;

[0089] Step 402. Set the current time status Input high-level model, the termination network is state-based Calculate the termination probability for each driving action;

[0090] Step 403. Based on the termination probability of each driving behavior, determine whether the current driving behavior will terminate in the next moment. If yes, proceed to step 404; otherwise, maintain the current driving behavior. Proceed to step 405;

[0091] Step 404. The Q-network is state-based. Calculate the Q-value for each driving action, and select the driving action with the largest Q-value based on a greedy strategy. Output;

[0092] Step 405. Based on driving behavior Load the low-level model trained in step 3, the low-level model being state-based. Generate Actions ;

[0093] Step 406. Action-based Interacting with the driving environment to obtain the next moment state space And through the reward function of the high-level model Receive the reward at the current moment ; Obtain empirical datasets ; and save it to the experience replay pool ,and , ;

[0094] Step 407. Determine the experience replay pool The amount of empirical data in If the sample size is greater than the preset sample experience batch size P, proceed to step 408; otherwise, return to step 402.

[0095] Step 408. From the experience replay pool A high-level model is trained using a medium-batch collection of empirical datasets; the Q-network is updated based on minimizing the mean squared error of the Q-value; the update formula for the Q-network is:

[0096] ;

[0097] In the formula, For the current moment Option value functions; , The driving behavior at the current moment exist The probability of termination at that time; Indicates the learning rate;

[0098] Step 409. Update the termination network using policy gradient; the update formula for the termination network is:

[0099] ;

[0100] In the formula, The dominant function representing the termination of the network. Indicates the cost parameters to consider; Indicates driving behavior at the current moment. In state The probability of termination at that time; Represents partial derivatives; Represents the loss function;

[0101] Step 410. Determine if the training termination condition is met. If yes, end the training; otherwise, return to step 402.

[0102] In one possible implementation, the training termination condition in step 410 is:

[0103] Step B1. This indicates that the target vehicle has left the road boundary;

[0104] Condition B2. This indicates that the target vehicle collided with an environmental vehicle.

[0105] Condition B3. , indicating the current time Reaching maximum control duration ;

[0106] Condition B4. This indicates that the target vehicle has arrived at the destination area;

[0107] Training ends when any of the above conditions are met. Attached Figure Description

[0108] Figure 1 This is a flowchart of the present invention;

[0109] Figure 2 This is a training framework diagram of the deep hierarchical reinforcement learning decision model of the present invention;

[0110] Figure 3 This is a training framework diagram of the low-level model of the present invention;

[0111] Figure 4 This is a flowchart illustrating the training process of the low-level model of this invention.

[0112] Figure 5 This is a flowchart illustrating the joint training process of the high-level model and the low-level model of this invention.

[0113] Figure 6 This is a comparative analysis of the cumulative distribution function curve of environmental rewards in a specific embodiment of the present invention;

[0114] Figure 7 This is a comparative analysis diagram of the distribution of different driving behavior choices in a specific embodiment of the present invention;

[0115] Figure 8 This is a comparative analysis experimental driving behavior switching view in a specific embodiment of the present invention. Detailed Implementation

[0116] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0117] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0118] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0119] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0120] To facilitate the description of vehicle driving decision-making problems, this application refers to the autonomous vehicle being studied as the target vehicle, and the manually driven or intelligent vehicles near the target vehicle as environmental vehicles. The target vehicle can obtain environmental observation information through Vehicle-to-Everything (V2X). The environmental observation information includes traffic information and environmental vehicle information within the current road segment. The driving task of the target vehicle is predefined, and the destination area is determined.

[0121] In this specific embodiment, the target vehicle is set to have two typical driving behaviors: conservative driving behavior and aggressive driving behavior. Conservative driving behavior aims to maximize driving safety by controlling the vehicle's attitude and is relatively simple to adapt to road environment conditions. Aggressive driving behavior aims to maximize driving efficiency by controlling the target vehicle's attitude and is more complex to adapt to road environment conditions.

[0122] See Figure 1 As shown in the figure, this application discloses a hierarchical vehicle driving decision-making method based on multiple time scales, including:

[0123] Step 1. Obtain the coordinates of the destination area for the target vehicle's driving mission. And obtain environmental observation information of the target vehicle.

[0124] Step 2. Construct a deep hierarchical reinforcement learning decision model, which includes a high-level model and a low-level model. The high-level model is used to determine the optimal driving behavior based on different environments; each driving behavior corresponds to a low-level model, which is used for attitude control of the target vehicle. In this specific implementation, corresponding low-level models are constructed for conservative driving behavior and aggressive driving behavior distributions; wherein:

[0125] State space design for high-level models:

[0126] The state space of the high-level model is designed as a holistic description of the environmental observation information of the target vehicle, specifically represented as the vehicle state matrix at the current moment:

[0127] ;

[0128] In the formula, These represent the current coordinates of the target vehicle. and Indicates the current time the target vehicle is in and Velocity components in two directions; , indicating environmental vehicles The state vector, This indicates the number of vehicles in the environment surrounding the target vehicle. Indicates environmental vehicles The coordinates of the location; This indicates the horizontal distance between the target vehicle and the center line of its current lane; This indicates the distance between the target vehicle and the nearest ambient vehicle in each lane.

[0129] Action space design for high-level models:

[0130] The action space of the high-level model is designed to represent the driving behavior of the target vehicle, as follows:

[0131] ;

[0132] In the formula, Indicates aggressive driving behavior. This indicates conservative driving behavior, and the driving behavior decision output by the high-level model is either aggressive or conservative driving behavior.

[0133] Reward function of high-level models design:

[0134] The reward function of the high-level model is designed as an environmental reward function to reflect driving safety, specifically including collision penalties. Penalties for exceeding road boundaries Advance distance reward and penalties and rewards for deviating from the center line , represented as: ;

[0135] Among them, collision penalty This indicates that the target vehicle's trajectory must not intersect with the trajectory of any surrounding vehicle at any time; if an intersection occurs, a collision penalty is imposed. Represented as:

[0136] ;

[0137] Penalty for exceeding road boundaries This indicates that the target vehicle must not cross the road boundary at any time; otherwise, it will collide with the roadside barriers, affecting driving safety. Crossing the road boundary incurs a penalty. Represented as:

[0138] ;

[0139] Forward Distance Bonus The immediate reward for the effective distance traveled by the target vehicle within a unit of time period; distance traveled bonus. Represented as:

[0140] ;

[0141] This indicates the time it takes for the target vehicle to travel along the road direction; the greater the effective travel distance within a unit of time, the higher the driving efficiency within that unit of time, and the greater the immediate reward obtained from the distance the target vehicle travels along the road direction (i.e., the X-axis direction).

[0142] Off-center line penalties and rewards The purpose is to ensure that the target vehicle stays as close to the center line of the current road as possible, with penalties and rewards for deviating from the center line. Represented as:

[0143] ;

[0144] in, The width of the lane where the target vehicle is located.

[0145] The state space of the low-level model is designed as follows: .

[0146] The action space of the low-level model is designed as the set of actions that the target vehicle can execute. In this specific embodiment, the low-level model needs to make decisions on the attitude control of the target vehicle. Therefore, in this specific embodiment, the action vector of the low-level model is set as the driving speed of the target vehicle at the current moment, and the action space of the low-level model is represented as follows:

[0147] ;

[0148] In the formula, and These respectively indicate that the target vehicle is in and Decision velocity components in two directions, Indicates that the target vehicle is in Maximum speed in this direction Indicates that the target vehicle is in Maximum speed in the direction of travel;

[0149] Reward function of low-level model Designed to include an environmental reward function and auxiliary reward function; where:

[0150] The environmental reward function of the low-level model Includes collision penalty Penalties for exceeding road boundaries Advance distance reward and penalties and rewards for deviating from the center line , represented as: ;

[0151] The auxiliary reward function includes lane change reward. Safe distance penalty Early arrival reward and the range of motion punishment ;

[0152] The lane change bonus The aim is to encourage target vehicles to learn lane-changing skills and avoid wasting time in congested lanes. When a target vehicle successfully changes lanes and maintains a safe distance from the nearest surrounding vehicle, a lane-changing reward is given. Represented as:

[0153] ;

[0154] In the formula, Indicates the target lane. A constant threshold for determining whether lane-changing behavior is appropriate;

[0155] The safety distance penalty The purpose is to guide the target vehicle to learn and maintain a safe driving distance. When the distance between the target vehicle and the nearest environmental vehicle in the current lane is less than the preset safe distance, a safety distance penalty is imposed, as shown below:

[0156] ;

[0157] In the formula, The lane where the target vehicle is located. It is a safe distance preset based on traffic rules and actual conditions;

[0158] The reward for arriving at the finish line early The reward given to a target vehicle for reaching the finish line ahead of schedule within the maximum control time is represented as:

[0159] ;

[0160] In the formula, Indicates the current moment. Indicates the maximum control duration;

[0161] The punishment for the range of motion Used to limit the target vehicle's movement within a unit of time. The range of motion changes within the vehicle is controlled to ensure smoothness and comfort during driving; penalty for excessive motion. Represented as:

[0162] ;

[0163] The reward function of the low-level model Represented as:

[0164] ;

[0165] In the formula, These are weighting coefficients, with specific values ​​set based on various typical driving behaviors. Table 1 shows the weighting coefficients for conservative and aggressive driving behaviors in this specific embodiment.

[0166] Table 1 Weight Coefficients of Reward Function in Low-Level Model

[0167] ;

[0168] As shown in Table 1, the goal of aggressive driving behavior is to enable the target vehicle to master lane-changing skills when traffic density is high and reach the destination as quickly as possible; while the goal of conservative driving behavior is to enable the target vehicle to maintain a large safe distance and ensure good moderation.

[0169] The high-level model in this specific embodiment includes a termination network and a Q-network. The termination network is used to calculate and output the termination probability of the target vehicle's driving behavior at the current moment based on the state space at the current moment. The Q-network is used to calculate and output the Q value of each driving behavior of the target vehicle based on the state space at the current moment.

[0170] The low-level model includes a policy network module and a dual-objective evaluation network module. The policy network module generates actions based on the state space with the assistance of the dual-objective evaluation network module. The policy network module includes a policy network and a target policy network. The dual-objective evaluation network module includes a first evaluation network, a first target evaluation network, a second evaluation network, and a second target evaluation network.

[0171] Step 3. Train the low-level model using the TD3 algorithm, such as... Figures 2-3 As shown; this specific embodiment uses the TD3 algorithm to address the problem of Q-value overestimation in the action vectors of the low-level model. Through a dual evaluation module, the target Q-value of the action vectors output by the low-level model is jointly calculated. A smaller target Q-value is used to participate in the parameter updates of the first and second evaluation networks, which can alleviate the Q-value overestimation problem. The training of the low-level model is as follows: Figure 4 As shown, it specifically includes:

[0172] Step 301. Initialize policy network parameters First evaluation network parameters Second evaluation network parameters Experience Replay Pool and maximum control duration ;

[0173] Step 302. Input the current time into the low-level model. status The policy network is based on the state Generate Actions ;

[0174] Step 303. Action-based Interacting with the driving environment to obtain the next moment status And through the reward function of the low-level model Receive the reward at the current moment ; Obtain empirical datasets Save it to the experience replay pool ,and , ;

[0175] Step 304. Determine the experience replay pool The amount of empirical data in Is it greater than the preset sampling experience batch size? If yes, proceed to step 305; otherwise, return to step 302.

[0176] Step 305. From the experience replay pool A low-level model is trained by sampling a batch of empirical datasets and minimizing the actions. The mean square error of the Q-value is used to update the parameters of the first and second evaluation networks; the expression is:

[0177] ;

[0178] ;

[0179] In the formula, As a discount factor, The total amount of empirical data sampled. This is motion noise;

[0180] Step 306. Update the policy network parameters using policy gradients. The expression is:

[0181] ;

[0182] Step 307. Update the target policy network parameters, the first target evaluation network parameters, and the second target evaluation network parameters using a soft update method, with the following expression:

[0183] ;

[0184] In the formula, For soft update parameters;

[0185] Step 308. Determine if any of the following training termination conditions are met:

[0186] Condition A1. This indicates that the target vehicle has left the road boundary;

[0187] Condition A2. This indicates that the target vehicle collided with an environmental vehicle.

[0188] Condition A3. , indicating the current time Reaching maximum control duration ;

[0189] Condition A4. This indicates that the target vehicle has arrived at the destination area;

[0190] If any of the above conditions are met, then the training ends; otherwise, then... Then return to step 302.

[0191] Step 4. Use the Objective-C algorithm to jointly train the high-level model and the low-level model trained in Step 3; for example... Figure 5 As shown, it specifically includes:

[0192] Step 401. Initialize the termination network parameters of the high-level model. Q network parameters Ω, driving behavior, experience replay pool and maximum control duration ;

[0193] Step 402. Set the current time status Input high-level model, the termination network is state-based Calculate the termination probability for each driving action;

[0194] Step 403. Based on the termination probability of each driving behavior, determine whether the current driving behavior should terminate at the current moment. If yes, proceed to step 404; otherwise, maintain the current driving behavior. Proceed to step 405;

[0195] Step 404. The Q-network is state-based. Calculate the Q-value for each driving action, and select the driving action with the largest Q-value based on a greedy strategy. Output;

[0196] Step 405. Based on driving behavior Load the low-level model trained in step 3, the low-level model being state-based. Generate Actions ;

[0197] Step 406. Action-based Interacting with the driving environment to obtain the next moment state space And through the reward function of the high-level model Receive the reward at the current moment ; Obtain empirical datasets ; and save it to the experience replay pool ,and , ;

[0198] Step 407. Determine the experience replay pool The amount of empirical data in If the sample size is greater than the preset sample experience batch size P, proceed to step 408; otherwise, return to step 402.

[0199] Step 408. From the experience replay pool A high-level model is trained using a medium-batch collection of empirical datasets; the Q-network is updated based on minimizing the mean squared error of the Q-value; the update formula for the Q-network is:

[0200] ;

[0201] In the formula, For the current moment Option value functions; , The driving behavior at the current moment exist The probability of termination at that time; Indicates the learning rate;

[0202] Step 409. Update the termination network using policy gradient; the update formula for the termination network is:

[0203] ;

[0204] In the formula, The dominant function representing the termination of the network. Indicates the cost parameters to consider; Indicates driving behavior at the current moment. In state The probability of termination at that time; Represents partial derivatives; Represents the loss function;

[0205] Step 410. Determine if any of the following training termination conditions are met:

[0206] Step B1. This indicates that the target vehicle has left the road boundary;

[0207] Condition B2. This indicates that the target vehicle collided with an environmental vehicle.

[0208] Condition B3. , indicating the current time Reaching maximum control duration ;

[0209] Condition B4. This indicates that the target vehicle has arrived at the destination area;

[0210] If any of the above training termination conditions are met, then training ends; otherwise, then... Then return to step 402.

[0211] Step 5. Based on the environmental observation information obtained by the target vehicle, obtain the driving behavior decision of the target vehicle through a high-level model;

[0212] Step 6. The target vehicle loads the corresponding low-level model based on the driving behavior decision output in Step 5, and combines it with environmental observation information. Through the low-level model, the target vehicle's attitude is controlled. In this specific embodiment, the target vehicle's attitude control is the target vehicle's driving speed.

[0213] To further evaluate the performance of the technical solution of this application, it is compared with two driving decision-making methods: single driving behavior: aggressive and single driving behavior: conservative. The single driving behavior: aggressive and single driving behavior: conservative methods are based on a single time scale, specifically, the vehicle's attitude is controlled at each control moment. The resulting cumulative distribution function (CDF) curve is shown below. Figure 6 As shown, the driving decision-making method proposed in this invention is trained based on a single driving behavior-based driving decision-making method, therefore its environmental rewards are mainly concentrated in a higher range. Furthermore, compared to driving decision-making methods using a single driving behavior, the driving decision-making method proposed in this invention can obtain higher environmental rewards.

[0214] To verify the rationality of the high-level model's decision-making in this application, the selection of different types of driving behaviors was visualized during the testing phase: Figure 7 This reflects the selection of different types of driving behaviors. Data shows that on the test roads, the target vehicles tended to choose more aggressive driving behaviors, likely due to the generally low traffic density on these roads. Figure 8 Yes Figure 7 Further explanation follows. In the diagram, the blue line segment depicts the target vehicle's trajectory, while the yellow "×" marks indicate that the target vehicle adopted conservative driving behavior at that location. When the ambient traffic density is high, the target vehicle adopts conservative driving behaviors such as slowing down to ensure driving safety. Conversely, when the ambient traffic density is low, the target vehicle tends to choose aggressive driving behaviors, such as changing lanes, to improve driving efficiency.

[0215] Therefore, the technical solution of this application has the following advantages: The multi-timescale hierarchical vehicle driving decision-making method is divided into two levels. First, a high-level model is used to macroscopically plan the vehicle's driving behavior based on the driving task and road environment over a longer time range. Then, a low-level model controls each control moment and performs fine vehicle attitude control and adjustment based on the decision results of the high-level model and real-time environmental observation information. The collaborative work of the high-level model and the low-level model realizes adaptive driving decision-making, which improves the safety and efficiency of the target vehicle's autonomous driving in dynamic road environments and more effectively completes the driving decision-making task.

[0216] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0217] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0218] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-time scale based hierarchical vehicle driving decision making method, characterized in that, Comprise: Step 1. Obtain the end area coordinates of the target vehicle driving task ; and obtain the environment observation information of the target vehicle; Step 2. Construct a deep hierarchical reinforcement learning decision model comprising a high-level model and a low-level model, the high-level model being used to determine the optimal driving behavior according to different environmental decisions; each driving behavior corresponds to a low-level model, the low-level model being used to control the posture of the target vehicle; Step 3. Train the low-level model using the TD3 algorithm; Step 4. Jointly train the high-level model and the low-level model trained in step 3 using the OC algorithm; specifically comprising: Step 401. Initialize termination network parameters of high-level model , Q-network parameters Ω, driving behavior, experience replay pool and maximum control length ; Step 402. inputting the current time of state of the high-level model, the termination network calculates the termination probability of each driving behavior based on the state ; Step 403. Determine whether the current driving behavior is terminated at the current time based on the termination probability of each driving behavior, if yes, go to step 404, if no, keep the driving behavior at the current time , go to step 405; Step 404. The Q network is based on the state The Q value of each driving behavior is calculated, and the driving behavior with the maximum Q value is selected based on the greedy strategy Output; Step 405. Based on driving behavior Load the low-level model trained through step 3, which is based on state Generate action ; Step 406. Action-based Interacting with the driving environment to obtain the next moment state space And through the reward function of the high-level model Receive the reward at the current moment ; Obtain empirical datasets ; and save it to the experience replay pool ,and , ; Step 407. Determine the experience replay pool The amount of empirical data in Is it greater than the preset sampling experience batch P? If yes, proceed to step 408; otherwise, return to step 402. Step 408. Collect experience data set from experience replay pool Train high-level model from experience data set in batch; update Q network based on minimizing mean square error of Q value; update formula of Q network is: ； In the formula, is the option value function at the current time ; , is the termination probability of the driving behavior at the current time ; ; denotes the learning rate; Step 409. Update the termination network using policy gradient; the update formula of the termination network is: ； wherein denotes the advantage function of terminating the network, denotes the thinking cost parameter; denotes the driving behavior at the current time denotes the termination probability at state at time denotes the partial derivative; denotes the loss function; Step 410. Determine whether the training end condition is met, if yes, end the training, if no, return to step 402; Step 5. Obtain the driving behavior decision of the target vehicle through the high-level model according to the environmental observation information obtained by the target vehicle; Step 6. The target vehicle loads the corresponding low-level model according to the driving behavior decision output by step 5, and jointly controls the posture of the target vehicle through the low-level model according to the environmental observation information.

2. The multi-time scale based hierarchical vehicle driving decision making method according to claim 1, wherein, The environmental observation information of the target vehicle obtained in step 1 is represented as: ； In the formula, These represent the current coordinates of the target vehicle. and Indicates the current time the target vehicle is in and Velocity components in two directions; , indicating environmental vehicles The state vector, This indicates the number of vehicles in the environment surrounding the target vehicle. Indicates environmental vehicles The coordinates of the location; This indicates the horizontal distance between the target vehicle and the center line of its current lane; This indicates the distance between the target vehicle and the nearest ambient vehicle in each lane.

3. The multi-time scale based hierarchical vehicle driving decision making method according to claim 2, wherein, The state space of the high-level model is represented as: ； The action space of the high-level model is the driving behavior of the target vehicle at the current time: ； In the formula, represents aggressive driving behavior, represents conservative driving behavior, and the driving behavior decision output by the high-level model is aggressive driving behavior or conservative driving behavior. The reward function of the high-level model is an environment reward function, and the environment reward function includes a collision penalty , an off-road boundary penalty , a forward distance reward , and a deviation from the center line penalty reward , and is specifically represented as: ； where the collision penalty is expressed as a collision penalty between the target vehicle and the environment vehicle; specifically expressed as: ； Exceeding road boundary penalty is expressed as a penalty for collision between the target vehicle and a road boundary barrier, and is specifically expressed as: ； forward distance reward a reward expressed as a target vehicle's effective forward distance in the direction of the road, specifically expressed as: ； wherein, represents the length of time the target vehicle advances in the direction of the road; Off-centerline penalty reward Penalty representing the target vehicle's deviation from the road centerline: ； wherein, is the width of the lane in which the target vehicle is located.

4. The multi-time scale based hierarchical vehicle driving decision making method according to claim 3, wherein, The high-level model further comprises a termination network and a Q network, the termination network being used to calculate and output the termination probability of the driving behavior of the target vehicle at the current time according to the state space at the current time, and the Q network being used to calculate and output the Q value of each driving behavior of the target vehicle according to the state space at the current time; The low-level model comprises a policy network module and a double-target evaluation network module, the policy network module generating actions according to the state space with the assistance of the double-target evaluation network module, the policy network module comprising a policy network and a target policy network, and the double-target evaluation network module comprising a first evaluation network, a first target evaluation network, a second evaluation network and a second target evaluation network.

5. The multi-time scale based hierarchical vehicle driving decision making method according to claim 4, wherein, The state space of the low-level model is represented as: ； The action space of the low-level model is the driving speed of the target vehicle at the current time: ； wherein and respectively represent the decision speed components of the target vehicle in and two directions, represents the maximum travel speed of the target vehicle in direction, represents the maximum travel speed of the target vehicle in direction; The reward function of the low-level model comprises an environmental reward function reflecting driving safety and an auxiliary reward function reflecting driving efficiency.

6. The multi-time scale based hierarchical vehicle driving decision making method according to claim 5, wherein, The environmental reward function of the low-level model is consistent with the environmental reward function of the high-level model, and is represented as: ； The auxiliary reward function includes a lane change reward , a safety distance penalty , an early arrival reward , and an action magnitude penalty , wherein: Lane change reward is represented as: ； In the formula, a target lane for which the target vehicle needs to change, a constant threshold for judging whether the lane-changing behavior is appropriate. Safety distance penalty is represented as: ； In the formula, is the lane in which the target vehicle is located, is a safety distance preset according to traffic rules and actual conditions; Early arrival reward is represented as: ； In the formula, t is the current time, is the maximum control duration; Action magnitude penalty is represented as: ； reward function of the low-level model is represented as: ； In the formula, are weight coefficients.

7. The multi-time scale based hierarchical vehicle driving decision making method according to claim 6, wherein, The low-level model is trained using the TD3 algorithm in step 3, specifically comprising: Step 301. Initialize policy network parameters , first evaluation network parameters , second evaluation network parameters , experience replay pool , and maximum control length ; Step 302. Input the state of the current time to the lower layer model , the policy network generates an action according to the state ;​​ Step 303. Action-based Interacting with the driving environment to obtain the next moment status And through the reward function of the low-level model Receive the reward at the current moment ; Obtain empirical datasets Save it to the experience replay pool ,and , ; Step 304. Determine the experience replay pool The amount of empirical data in Is it greater than the preset sampling experience batch size? If yes, proceed to step 305; otherwise, return to step 302. Step 305. Sample a batch of experience datasets from the experience replay pool Train the low-level model from the batch of experience datasets by minimizing the mean squared error of the Q-values of the actions Update the parameters of the first and second critic networks; the expression is: ； ； wherein is a discount factor, is the total amount of empirical data sampled, is the action noise; Step 306. Update the policy network parameters by policy gradient , the expression is: ； Step 307. Update the target policy network parameters, the first target evaluation network parameters and the second target evaluation network parameters respectively using the soft update method, and the expression is: ； In the formula, is an updated parameter weight; Step 308. Determine whether the training end condition is met, if yes, end the training, if no, return to step 302.

8. The multi-time scale based hierarchical vehicle driving decision making method according to claim 7, wherein, The training end condition in step 308 is: Condition A1. , indicating that the target vehicle is exiting a road boundary; Condition A2. , indicating that the target vehicle collides with the environment vehicle; Condition A3. , indicates the current time reaches the maximum control duration ; Condition A4. , indicating that the target vehicle has reached the end zone; When any of the above conditions is met, the training is ended.

9. The multi-time scale based hierarchical vehicle driving decision making method of claim 1, wherein, The training end condition in step 410 is: Step B1. , indicating that the target vehicle is driving out of the road boundary; Condition B2. , indicating that the target vehicle collides with the environment vehicle; Condition B3. , indicates the current time reaches the maximum control duration ; Condition B4. , indicating that the target vehicle has reached the end zone; When any of the above conditions is met, the training is ended.

Images