A decision-making method for intelligent car driving around an island based on behavioral risk reinforcement learning

Abstract

This invention belongs to the field of automotive technology, specifically a method for making roundabout driving decisions for intelligent vehicles based on behavioral risk reinforcement learning. First, the method analyzes and establishes a vehicle's coupled lateral and longitudinal action space, a multi-scale state space encompassing driving task, surrounding environment, and vehicle information, and a multi-objective reward function that considers safety, task, and driving behavior. Second, by applying a self-learning behavioral risk classifier based on a multi-layer perceptron, the agent's behavioral risks are classified and assessed. Based on the behavioral risk classification results, a behavioral risk-based strategy optimization objective is designed to optimize the reinforcement learning driving decision-making algorithm. This method addresses the impact of inadequate reward function design on driving strategy optimization and guides driving decisions toward a safer convergence.

Description

Technical Field

[0001] The present invention belongs to the field of automobile technology, and specifically relates to an intelligent automobile roundabout driving decision-making method based on behavioral risk reinforcement learning. Background Art

[0002] As a crucial component of complex urban traffic, roundabouts play a crucial role in diverting vehicles, reducing conflicts, and improving driving efficiency. Compared to simple driving tasks like longitudinal cruise following and high-speed lateral lane changes, smart cars face the high complexity and interactivity of dynamic and diverse driving tasks such as merging, circling, and exiting in roundabouts. This poses a significant challenge to the safety of smart car driving decisions.

[0003] Deep reinforcement learning, which combines the feature extraction capabilities of deep neural networks with the advantages of reinforcement learning-environment interaction, is considered an effective solution for designing intelligent vehicle driving decision-making strategies. It has been widely used in simple urban traffic conditions such as lane keeping, lane changing, and adaptive cruise control. However, before applying deep reinforcement learning to vehicle driving decisions, the environment needs to be properly modeled to improve the quality of strategy training. However, these studies on simple urban traffic conditions lack comprehensive consideration of the vehicle's lateral and longitudinal coupled action space, dynamic and diverse driving tasks, complex driving condition constraints, and strong vehicle interactions. This leads to certain safety risks when directly applied to intelligent vehicle driving decisions in complex urban traffic scenarios such as roundabouts.

[0004] Furthermore, the reward function is crucial for the optimization and convergence of deep reinforcement learning-based driving decision-making strategies. Reward functions designed based on expert experience for complex urban traffic scenarios may have certain empirical blind spots, causing deep reinforcement learning-based driving decision-making strategies to fail to converge or become trapped in local optima. Consequently, methods such as imitation learning and inverse reinforcement learning have been applied in recent years to design driving decision-making strategies for complex urban traffic scenarios. However, imitation learning relies heavily on effective expert demonstration data and has poor generalization capabilities to scenarios not covered by the data. Inverse reinforcement learning also relies on expert demonstration data, and the heterogeneous driving characteristics of different experts can also lead to bias in the fitted reward function or difficulty in convergence. Recently, reinforcement learning methods based on human preferences, such as human feedback reinforcement learning, have been widely used in the field of large language models. Their core idea is to compare superior and inferior strategies, aiming to output superior strategies rather than inferior strategies. Therefore, reinforcement learning methods that understand human preferences can also be applied to driving decision-making, helping intelligent vehicles to output safer driving maneuvers in complex traffic scenarios. Human preferences are represented by establishing objective reward functions. However, the training of this reward function requires a large amount of driving decision data, which to some extent hinders the application of reinforcement learning algorithms based on human preferences in complex urban traffic scenarios. Summary of the Invention

[0005] To solve the above problems, the present invention provides an intelligent car island driving decision-making method based on behavioral risk reinforcement learning, which can adapt to the complex interactive scenarios of the island, ensure safe decision-making during the training process of the deep reinforcement learning algorithm, and realize the rapid implementation of the reinforcement learning algorithm.

[0006] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings:

[0007] A reinforcement learning-based intelligent vehicle roundabout driving decision-making method includes the following steps:

[0008] Step 1: Build a roundabout scene model;

[0009] Step 2: Build a self-learning behavioral risk classifier.

[0010] Step 3: Model the behavioral risk reinforcement learning driving decision algorithm to enable smart cars to make decisions on roundabout driving.

[0011] Furthermore, the specific method of step one is as follows:

[0012] 11) Design action space;

[0013] 12) Design state space;

[0014] 13) Design a multi-objective reward function for intelligent vehicles.

[0015] Furthermore, the specific method of step 11) is as follows:

[0016] Combine intelligence and drive into the normalized signal p required for simulating vehicle longitudinal acceleration and deceleration control x ∈[-1, 1], and the front wheel active steering signal δ e ∈[-1, 1], construct the horizontal and vertical coupling action space A of the intelligent car e :

[0017] A e =[p ex ,δ e ] (1).

[0018] Furthermore, the specific method of step 12) is as follows:

[0019] Design a driving task vector V consisting of key tasks and driving tasks T ,as follows:

[0020]

[0021] Where ΔP ENDis the coordinate of the vehicle’s current position relative to the end point; and are the vehicle's relative forward path point P i The distance and orientation deviation, P i Progressive as the vehicle moves forward;

[0022] Design the surrounding environment state vector V composed of surrounding vehicle information E ; The vehicle's surroundings are divided into seven detection intervals from P0 to P6 by using the region cutting coding method; the high interaction detection intervals P0, P1, P4, and P5 are expanded; when the external perception module detects another vehicle that is traveling in the same direction as the vehicle and whose center point projection is within the detection interval, it is marked as the perceived vehicle, and the corresponding vehicle information V i Expressed as:

[0023] V i =[F,ΔL,d -1 ,Δv,δθ,I],i={0,...,6}, (3)

[0024] Where F∈{0,1} is the flag of whether there is a vehicle; ΔL,d,Δv=[Δv x , Δv y ] and δθ are the relative lane offset, distance, speed and relative orientation between the other vehicle and the vehicle itself, respectively; I∈[-1, 1] is the possibility of the other vehicle changing lanes;

[0025] When there is no other vehicle in the detection range, the corresponding setting Finally, the surrounding environment vector V is obtained E for:

[0026] V E =[V0,V1,V2,V3,V4,V5,V6]. (4)

[0027] Design includes the vehicle speed information v e =[v ex , v ey ], acceleration information a e =[a ex , a ey ], driving direction information h e =[cosθ e ,sinθ e ] and yaw rate information ω e The vehicle state vector, with the future target point preview information added To comprehensively establish the state vector V e :

[0028] V e =[v e,a e ,h e ,ω e ,Δθ e ,A e ], (5)

[0029] Where θ e is the heading angle of the ego vehicle; is the angular difference between the directions of the three future target points and the heading angle of the vehicle;

[0030] In summary, the state space is formed:

[0031] S=[V T ,V e ,V E ] T . (6).

[0032] Furthermore, the specific method of step 13) is as follows:

[0033] Design collision reward function r c for:

[0034]

[0035] Where, min{d i} is the shortest distance between all vehicles and the self-vehicle in the detection interval; D c is the dangerous collision distance threshold;

[0036] Through vertical and horizontal safety rewards f and r l Boot as follows:

[0037] r f =-0.5|min(0,Δv fx )|(D f -d f ), (8)

[0038] r l =-2.5|min(0,Δv ky )|(D l -d k ) 2 , (9)

[0039] Where Δv fx and d f ∈(D c , D f ) is the relative longitudinal speed and distance between the preceding vehicle and the vehicle; Δv ky and d k ∈(D c , D l) is the relative lateral speed and distance between the nearest vehicle next to the vehicle and the vehicle; D f and D l The longitudinal and lateral safe vehicle distances;

[0040] Design roundabout entrance safety reward RE , as shown below:

[0041] r RE =-0.25||Δv i ||(D RE -d i ), (10)

[0042] Where, d i ∈(D c , D RE ) and Δv i are the distance and relative speed vector between the ego vehicle and the high-risk vehicle respectively; D RE Safe distance for high-risk interactions;

[0043] The security reward function is obtained as follows:

[0044] R S =r c +r f +r l +r RE , (11)

[0045] Establish the reward r for reaching the end point succ , as shown below:

[0046] r succ =100,||ΔP End ||≤D P , (12)

[0047] Where D P is the distance threshold between the vehicle and the path point P;

[0048] Introducing path tracking reward r tag , as shown below:

[0049]

[0050] Where, is the driving distance of the vehicle between the two target points;

[0051] Build Speed ​​Bonus sp , as shown below:

[0052]

[0053] Design Course Rewards δθGuide the vehicle to move in the specified road direction as shown below:

[0054]

[0055] Where V lim is the road speed limit; δθ e is the deviation between the vehicle heading and the road direction; Θ δθ1 ,Θ δθ2 is the vehicle driving direction deviation threshold;

[0056] Design the driving task reward function as follows:

[0057] R T =r succ +r tag +r δθ +r sp (16)

[0058] Design Turn Rewards st , as shown below:

[0059] r st =-2|δ e |v x (17)

[0060] Designing lane keeping rewards lk , as shown below:

[0061]

[0062] Where δd is the lateral offset of the vehicle's current position relative to the road centerline; L is the road width; when the vehicle crosses the road boundary, that is, when |δd| ≥ L, r lk =-20 ends the current round and resets the environment;

[0063] Define the control instruction reward r control , as shown below:

[0064] r control =-4||A cur -A per ||2 (19)

[0065] Where A per , A cur are the action vectors at the previous and next moments respectively;

[0066] Establishing yaw comfort bonus ω , guiding the car's yaw rate to change smoothly, as shown below:

[0067] r ω =-((ω-Ω ω ) / 5)2 ,Ω ω <ω (20)

[0068] Where, Ω ω is the yaw rate threshold;

[0069] Design driving behavior reward function R C , as shown below:

[0070] R C =r lk +r ω +r st +r control (twenty one)

[0071] Finally, the multi-objective reward function of the intelligent car is obtained as follows:

[0072] R=R S +R T +R C (twenty two).

[0073] Furthermore, the specific method of step 2 is as follows:

[0074] 21) Construct a multi-layer perceptron with the input of state space s and action space a, and the output of the machine is the probability of three behavioral risk categories: safe, low risk, and high risk.

[0075] 22) Train the classifier.

[0076] Furthermore, the specific method of step 22) is as follows:

[0077] 1) Data Collection and Labeling: Based on the existing roundabout decision-making strategy, simulated roundabout driving data is collected. State-action pairs {s, a} are randomly sampled from the collected data and labeled with behavioral risk categories to assess the risk of driving decision-making behaviors. Three behavioral risks are determined based on the following criteria: high-risk behaviors are those that, if executed, will further deteriorate driving safety under dangerous conditions such as collision, lane departure, or high-speed steering; safe behaviors are those that follow or change lanes under safe conditions, or those that avoid further deterioration of the situation under dangerous conditions; and low-risk behaviors are those that, under safe conditions, will deteriorate driving safety.

[0078] 2) Pre-training: Use labeled data to pre-train the classifier based on supervised learning, so that it has the initial ability to distinguish different risk categories;

[0079] 3) Self-learning Training: During the prediction phase, the classifier first makes predictions on unlabeled roundabout data. High-confidence screening is performed, and state-action pairs with a risk category prediction confidence exceeding 90% are labeled as corresponding predicted categories. These data are then added to the next round of training, and the training cycle repeats the self-learning process to continuously update the classifier.

[0080] 4) Termination of training: When the classification accuracy of the classifier on the test set no longer improves, the training ends;

[0081] A joint loss consisting of focal loss and label distribution-aware margin loss is used to address the problems of class imbalance and difficult sample mining in pre-training and self-learning training. The joint loss function is constructed as follows:

[0082]

[0083] Where γ is the hyperparameter for adjusting the weight factor; p y ∈[0,1] is the predicted value; α y =3∑ j n j / n y is the weight parameter for balancing the number of samples, where n j is the number of samples in each category;

[0084] Design the parameter α related to the category proportion in the loss function y , n y As the training changes dynamically, the loss function can be adaptively adjusted.

[0085] Furthermore, the specific method of step three is as follows:

[0086] 31) By introducing policy entropy To encourage the policy to produce more diverse exploration behaviors, as shown below:

[0087]

[0088] Where, π φ is the policy function;

[0089] 32) Reinforcement learning algorithms consider policy entropy through Bellman regression operators and temporal difference pairs The Q-value function is iteratively solved as follows:

[0090]

[0091] The cost function of the algorithm value network is defined as

[0092]

[0093] Where s, a, r, s′ are the current state, action, reward, and next state of the sample respectively; α is the entropy weight coefficient; are the value network and the target network respectively; φ is the strategy network;

[0094] Under the constraints Under this condition, the entropy weight coefficient α is automatically adjusted to balance the policy entropy The ratio between the expected return and the cost function of the hyperparameter α is designed as follows:

[0095]

[0096] Where, is the entropy constraint threshold;

[0097] 33) The iterative objective is designed to minimize the information divergence between the policy function π and the Q-value function, that is, to maximize the expected reward and policy entropy at the same time As shown below:

[0098]

[0099] Where, π φ (s) = f φ (s, ξ) is the sampling value of the policy distribution under the reparameterized skill; is the noise that obeys the standard normal distribution; K(s) is the partition function and does not participate in the strategy optimization;

[0100] 34) Using the state-action dataset, we iteratively optimize by maximizing the expected difference in rewards between good and bad behaviors under the same input, so that the reward network has the ability to accurately evaluate good and bad strategies. The cost function is as follows:

[0101]

[0102] Where r ψ To reward the network; a + , a - are the superior and inferior behaviors corresponding to the same state s; σ is the sigmoid function;

[0103] 35) Information divergence is added to limit the deviation between the new strategy and the original strategy. The cost function is defined as:

[0104]

[0105] Where, π φ is the trained policy model; π ref is the initial model; β is the parameter that controls the degree of deviation between the two strategies; Z = ∑ a πref (a|s)exp(r(s, a) / β) is the partition function, which is only related to the state s and π ref is related to the optimized strategy π φ irrelevant;

[0106] Strategy π φ The optimization goal is equivalent to minimizing the difference between the numerator and the denominator, and the optimal strategy π * Defined as:

[0107]

[0108] Further substitute formula (31) into formula (30), and we get the strategy π φ The optimization goal is to minimize the training strategy π φ and the optimal strategy π * The deviations between the distributions are as follows:

[0109]

[0110] From formula (30), we can get the reward network r ψ and the optimal strategy π * The relationship between them is:

[0111]

[0112] 36) Use the relationship between the reward network and the optimal strategy to eliminate the reward network r in formula (28) ψ , and based on the maximum likelihood optimization idea and the strategy π of formula (32) φ Optimize the objective by replacing the optimal policy with policy π φ , to directly optimize the strategy π using human-labeled data φ The purpose is to maximize the probability difference between the strategy outputting good and bad behaviors under the same state, as shown below:

[0113]

[0114] By assessing the risk of sampled driving behaviors, we can improve the direction of policy iteration and reduce the impact of insufficient reward function design on policy optimization, as shown below:

[0115]

[0116] Where, π φ′ is π φ The objective function of a + , a - They are respectively the good and bad behaviors based on behavioral risk classification, and the target strategy function π φ′Perform two random samplings and classify the risk through the behavioral risk classifier. The lower the driving risk, the better the behavior. - -C + ) / 4 is the behavioral risk category difference coefficient; C∈{0, 1, 2} corresponds to the three risk behavior labels of safe, low-risk and high-risk, respectively.

[0117] The beneficial effects of the present invention are:

[0118] 1) This paper designs a roundabout scenario model, which includes the vehicle's lateral and longitudinal coupled action space, a multi-scale information state space composed of driving task, surrounding environment, and vehicle information, and a multi-objective reward function that considers safety, task, and driving behavior. This model can fully characterize the characteristics of roundabout driving, enabling reinforcement learning to fully understand and handle complex roundabout interaction scenarios.

[0119] 2) Based on the theory of human preference reinforcement learning, this paper designs behavioral risk reinforcement learning that can be applied to the field of driving decision-making. By introducing the behavioral risk classification optimization goal, it compensates for the inadequate reward function design in the reinforcement learning training process.

[0120] 3) This paper derives a behavioral risk optimization objective function based on reward training and strategy training, simplifies the two-stage training process to a single-stage training process, and directly uses behavioral risk data to optimize the strategy iteration direction;

[0121] 4) In order to achieve parallel and accurate evaluation of the relative advantages and disadvantages of sampled behaviors during the strategy optimization process, the present invention adopts a behavior risk classifier based on three behavior risk categories: safe, low-risk, and high-risk;

[0122] 5) In order to implement the behavioral risk classifier and reduce the intensity and difficulty of labeling driving risk data, a self-learning based classifier training strategy was designed, which can complete the classifier training task based on multi-layer perceptron with only a small amount of labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0124] Figure 1 Schematic diagram of the working principle of the present invention

[0125] Figure 2a Schematic diagram of surrounding environment detection when merging into a roundabout;

[0126] Figure 2b This is a schematic diagram of the surrounding environment detection when passing through the roundabout entrance;

[0127] Figure 3 Schematic diagram for training a self-learning behavioral risk classifier;

[0128] Figure 4 This is a schematic diagram of the classifier architecture based on a multi-layer perceptron;

[0129] Figure 5 Schematic diagram of various reinforcement learning training reward curves. DETAILED DESCRIPTION

[0130] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0131] Example 1

[0132] See Figure 1 The present invention provides a method for making decisions about roundabout driving for intelligent vehicles based on behavioral risk reinforcement learning. First, the method analyzes and establishes the vehicle's lateral and longitudinal coupled action space, a multi-scale state space containing driving task-surrounding environment-vehicle information, and a multi-objective reward function that considers safety-task-driving behavior. Secondly, by applying a self-learning behavioral risk classifier based on a multi-layer perceptron, the agent's behavioral risk is classified and assessed, and a behavioral risk-based strategy optimization objective is designed based on the behavioral risk classification results to optimize the reinforcement learning driving decision algorithm, thereby addressing the impact of insufficient reward function design on driving strategy optimization and guiding driving decisions to converge towards a safer direction. Specifically, the method includes the following steps:

[0133] Step 1: Build the island scene model, as follows:

[0134] 11) Design action space;

[0135] To ensure that the intelligent car can complete the driving task in the roundabout scenario, precise control of driving, steering, and braking is required. Considering that the simultaneous output of driving and braking by the intelligent agent will reduce the comfort of the longitudinal motion of the car, the two are combined into the normalized signal p required for the longitudinal acceleration and deceleration control of the simulated vehicle. x ∈[-1, 1], and the front wheel active steering signal δ e ∈[-1, 1], construct the horizontal and vertical coupling action space A of the intelligent car e :

[0136] Ae =[p ex ,δ e ] (1)

[0137] 12) Design state space;

[0138] The state vector designed by the present invention is composed of vectors at three scales: driving task, surrounding environment, and vehicle information. It fully restores the high complexity and strong interactivity problems faced by smart cars when performing dynamic and diverse driving tasks such as merging, driving around the island, and exiting in roundabout scenarios.

[0139] The smart car needs to complete the driving task under the guidance of the planned global path in the roundabout and reach the planned destination smoothly. Therefore, the driving task vector V T The design consists of terminal tasks and driving tasks:

[0140]

[0141] Where ΔP END is the coordinate of the vehicle’s current position relative to the end point; and are the vehicle's relative forward path point P i The distance and orientation deviation, P i Progressive as the vehicle moves forward;

[0142] The surrounding environment state vector V E It is composed of surrounding vehicle information. Figure 2a and Figure 2b The region segmentation coding method shown in the figure divides the surrounding environment of the ego vehicle into seven detection intervals P0 to P6. In order to more effectively identify the information of other vehicles competing for the right of way with the ego vehicle, some high-interaction detection intervals are expanded, such as Figure 2a P0, P1 and Figure 2b When the external perception module detects other vehicles that are traveling in the same direction as the vehicle and whose center point projection is within the detection interval, it is marked as the perceived vehicle, and the corresponding vehicle information V i It can be expressed as:

[0143] V i =[F,ΔL,d -1 ,Δv,δθ,I],i={0,...,6}, (3)

[0144] Where F∈{0,1} is the flag of whether there is a vehicle; ΔL,d,Δv=[Δv x , Δv y ] and δθ are the relative lane offset, distance, speed and relative orientation between the other vehicle and the vehicle; I∈[-1,1] is the possibility of the other vehicle changing lanes;

[0145] When there is no other vehicle in the detection range, the corresponding setting Finally, the surrounding environment vector V is obtained E for:

[0146] V E =[V0,V1,V2,V3,V4,V5,V6]. (4)

[0147] The vehicle's state vector V e Mainly includes vehicle speed information v e =[v ex , v ey ], acceleration information a e =[a ex , a ey ], driving direction information h e =[cosθ e , sinθ e ] and yaw rate information ω e Considering that the target trajectory needs to continuously guide the direction of the vehicle, the future target point preview information is added. To comprehensively establish the state vector V e :

[0148] V e =[v e ,a e ,h e ,ω e ,Δθ e ,A e ], (5)

[0149] Where θ e is the heading angle of the ego vehicle; is the angular difference between the directions of the three future target points and the heading angle of the vehicle;

[0150] In summary, the state space is formed:

[0151] S=[V T ,V e ,V E ] T . (6).

[0152] 13) Design multi-objective reward functions for intelligent vehicles;

[0153] In order to promote the safe, efficient and stable execution of dynamic and diverse driving tasks in roundabout scenarios by smart cars, the present invention designs a multi-objective reward function that considers safety, tasks and driving behavior.

[0154] First, the smart car should avoid collisions with other cars as much as possible when driving in a roundabout. Therefore, the collision reward function r is designed. c for:

[0155]

[0156] Where, min{d i} is the shortest distance between all vehicles and the self-vehicle in the detection interval; D c is the dangerous collision distance threshold.

[0157] Secondly, the vehicle should always maintain a safe distance from surrounding vehicles during driving to deal with emergencies such as emergency braking of the vehicle in front or lane change of the vehicle next to it. f and r l Guide

[0158] r f =-0.5|min(0,Δv fx )|(D f -d f ), (8)

[0159] r l =-2.5|min(0,Δv ky )|(D l -d k ) 2 , (9)

[0160] Where Δv fx and d f ∈(D c , D f ) is the relative longitudinal speed and distance between the preceding vehicle and the vehicle; Δv ky and d k ∈(D c , D l ) is the relative lateral speed and distance between the nearest vehicle next to the vehicle and the vehicle; D f and D l The longitudinal and lateral safe vehicle distances;

[0161] Figure 2b In the high-interaction risk area at the roundabout entrance, the high-risk vehicles competing with the vehicle for the right of way pose a greater threat to the vehicle than other vehicles traveling in the same direction. R To improve the car's ability to deal with high-risk vehicles;

[0162] r RE =-0.25||Δv i ||(D RE -d i ), (10)

[0163] Where, d i ∈(D c , D RE ) and Δv i are the distance and relative speed vector between the ego vehicle and the high-risk vehicle respectively; D RE Safe distance for high-risk interactions;

[0164] In summary, the safety reward function can be obtained as:

[0165] R S =r c +r f +r l +r RE , (11)

[0166] The smart car needs to reach the designated destination according to the driving task requirements. To this end, a reward r is established for reaching the destination. succ :

[0167] r succ =100,||ΔP End ||≤D P , (12)

[0168] Where D P is the distance threshold between the vehicle and the path point P;

[0169] In order to drive the intelligent car to follow the global planning path as much as possible, the path tracking reward r is introduced. tag :

[0170]

[0171] Where, is the driving distance between the two target points; considering the efficient driving requirements of smart cars, the speed reward r is established sp :

[0172]

[0173] On this basis, the heading reward r is designed δθ Guiding the vehicle to move in the specified road direction can be expressed as:

[0174]

[0175] Where V lim is the road speed limit; δθ e is the deviation between the vehicle heading and the road direction; Θ δθ1 ,Θ δθ2 is the vehicle driving direction deviation threshold;

[0176] Finally, the driving task reward function is:

[0177] R T =r succ +r tag +r δθ +r sp (16)

[0178] When a car is driving in a roundabout, it should minimize frequent turning behaviors and stay in the center of the road as much as possible to avoid blocking other vehicles. Therefore, the steering reward r is designed. st :

[0179] r st =-2|δ e |v x (17)

[0180] and lane keeping reward r lk :

[0181]

[0182] Where δd is the lateral offset of the vehicle's current position relative to the road centerline; L is the road width; when the vehicle crosses the road boundary, that is, when |δd| ≥ L, r lk =-20 ends the current round and resets the environment;

[0183] During the driving process of the intelligent car, its action output should maintain a certain continuity to reduce the sudden change of action at adjacent moments. The control command reward r is defined as control for:

[0184] r control =-4||A cur -A per ||2 (19)

[0185] Where A per , A cur are the action vectors at the previous and next moments respectively;

[0186] When the smart car makes necessary turns, it is necessary to establish a yaw comfort reward r ω , guiding the vehicle's yaw rate to change smoothly:

[0187] r ω =-((ω-Ω ω ) / 5) 2 ,Ω ω <ω (20)

[0188] Where, Ω ω is the yaw rate threshold;

[0189] Therefore, the driving behavior reward function RC for:

[0190] R C =r lk +r ω +r st +r control (twenty one)

[0191] Through the above design, we finally get the multi-objective reward function of the smart car:

[0192] R=R S +R T +R C (twenty two).

[0193] Step 2: Build a self-learning behavior risk classifier, as follows:

[0194] In order to achieve the strategy optimization goal of the behavior risk reinforcement learning algorithm and reduce the difficulty of labeling driving decision data, a self-learning behavior risk classifier based on a multi-layer perceptron was constructed. This classifier is used to evaluate the three behavior risk categories of driving decision-making behavior: safe, low-risk, or high-risk. The specific training process is as follows: Figure 3 As shown, it includes the following processes: data collection and annotation, classifier pre-training, unlabeled data prediction and classifier self-learning training;

[0195] The behavior risk classifier is constructed by a multi-layer perceptron. Its input is the state space s and the action space a, and its output is the probability of three behavior risk categories (safe, low risk, high risk). Its architecture is as follows: Figure 4 As shown in Figure 2, the specific training process of the classifier is as follows:

[0196] 1) Data Collection and Labeling: Based on the existing roundabout decision-making strategy, simulated roundabout driving data is collected. State-action pairs {s, a} are randomly sampled from the collected data and labeled with behavioral risk to assess the risk of their driving decisions. Three behavioral risks are determined based on the following criteria: High-risk behaviors are defined as those that, if executed, will further deteriorate driving safety under hazardous conditions such as a potential collision, lane departure, or high-speed steering; Safe behaviors are defined as those executed under safe conditions such as following or changing lanes, or to prevent further deterioration of the situation under hazardous conditions; and Low-risk behaviors are defined as those executed under safe conditions that will deteriorate driving safety.

[0197] 2) Pre-training: Use labeled data to pre-train the classifier based on supervised learning, so that it has the initial ability to distinguish different risk categories;

[0198] 3) Self-learning Training: During the prediction phase, the classifier first makes predictions on unlabeled roundabout data. High-confidence screening is performed, and state-action pairs with a risk category prediction confidence exceeding 90% are labeled with the corresponding predicted category and added to the next round of training. This self-learning process is repeated in a loop, continuously updating the classifier.

[0199] 4) Termination of training: When the classification accuracy of the classifier on the test set no longer improves, the training ends.

[0200] During normal roundabout driving decisions, high-risk behaviors occur much less frequently than safe behaviors, leading to an extreme imbalance in the number of categories, which in turn affects the network's ability to extract and learn features from a small amount of data. Therefore, this paper uses a joint loss consisting of focal loss and label distribution-aware margin loss to address the problems of category imbalance and difficult sample mining during pre-training and self-learning training. The joint loss function is:

[0201]

[0202] In the formula, γ is the hyperparameter for adjusting the weight factor, p y ∈[0,1] is the predicted value, α y =3∑ j n j / n y is the weight parameter for balancing the number of samples, where n j is the number of samples in each category. At the same time, as the self-learning training progresses, newly labeled data will be added to each round of training data, resulting in changes in the proportion of each category. Therefore, the parameter α related to the category proportion in the design loss function is y , n y As the training changes dynamically, the loss function is adaptively adjusted, and the features of a small number of data categories are effectively extracted and the classification accuracy is improved.

[0203] Step 3: Model the behavioral risk reinforcement learning driving decision algorithm as follows:

[0204] 31) This paper uses a soft actor-critic reinforcement learning algorithm with high exploration efficiency and strong training convergence to generate driving decisions for intelligent vehicles, including steering and acceleration and deceleration. The soft actor-critic is a reinforcement learning algorithm based on the actor-critic framework for policy iteration. It introduces policy entropy H to encourage the policy to produce more diverse exploration behaviors:

[0205]

[0206] Where, π φis the policy function;

[0207] 32) The solution process of the reinforcement learning algorithm mainly includes two parts: strategy evaluation and strategy optimization. In the strategy evaluation stage, the reinforcement learning algorithm uses the Bellman regression operator and temporal difference to iteratively solve the Q-value function considering the strategy entropy H, which can be expressed as:

[0208]

[0209] Correspondingly, the cost function of the algorithm value network is defined as:

[0210]

[0211] Where s, a, r, s′ are the current state, action, reward, and next state of the sample respectively; α is the entropy weight coefficient; are the value network and the target network respectively; φ is the strategy network;

[0212] The algorithm needs to be under constraints Under this condition, the entropy weight coefficient α is automatically adjusted to balance the policy entropy The ratio between α and the expected return, therefore, the cost function of the hyperparameter α is designed as:

[0213]

[0214] Where, is the entropy constraint threshold;

[0215] 33) The algorithm strategy optimization stage focuses on using the updated Q value to guide the iterative process of the policy function π; in order to increase the exploration ability of the reinforcement learning algorithm, the iterative goal is designed to minimize the information divergence between the policy function π and the Q value function, that is, to maximize the expected reward and policy entropy at the same time. Expressed as:

[0216]

[0217] Where, π φ (s) = f φ (s, ξ) is the sampling value of the policy distribution under the reparameterized skill; is the noise that obeys the standard normal distribution; K(s) is the partition function and does not participate in the strategy optimization;

[0218] The aforementioned policy optimization is primarily based on iterations to maximize expected reward. However, manually designed reward functions in complex traffic scenarios are not optimal, which can cause the policy optimization process to deviate from the original reward design intent. Therefore, we use a classifier to determine the risk of driving behavior on roundabouts to correct the direction of reward-guided policy iteration. By introducing a behavioral risk strategy optimization objective, the policy outputs safer driving behaviors.

[0219] 34) Introducing the training process of human-feedback reinforcement learning, which includes reward network training and policy function training, derives the optimization goal of the reinforcement learning behavior risk strategy. During the reward network training process of human-feedback reinforcement learning, iterative optimization is performed by maximizing the expected difference in rewards for good and bad behaviors under the same input, so that the reward network has the ability to identify the better behavior strategy. The cost function is:

[0220]

[0221] Where r ψ To reward the network; a + , a - is the superior and inferior behavior under state s; σ is the sigmoid function;

[0222] 35) Policy function training aims to maximize the expected reward, so that the policy can output advantageous behaviors that are more consistent with the expected reward, that is, human subjective expectations. At the same time, information divergence is added to limit the deviation between the new policy and the original policy. Its cost function is defined as:

[0223]

[0224] Where, π φ is the trained policy model; π ref is the initial model; β is the parameter that controls the degree of deviation between the two strategies; Z = ∑ a π ref (a|s)exp(r(s, a) / β) is the partition function, which is only related to the state s and π ref is related to the optimized strategy π φ irrelevant;

[0225] From the above formula, we can see that the strategy π φ The optimization objective is equivalent to minimizing the difference between the numerator and the denominator; in addition, the optimal strategy π * Defined as:

[0226]

[0227] Further substitute formula (31) into formula (30), and we get the strategy π φ The optimization goal is to minimize the training strategy π φ and the optimal strategy π * The deviations between the distributions are as follows:

[0228]

[0229] From formula (31), we can get the reward network r ψ and the optimal strategy π *The relationship between them is:

[0230]

[0231] 36) Use the relationship between the reward network and the optimal strategy to eliminate the reward network r in formula (28) ψ , and based on the maximum likelihood optimization idea and the strategy π of formula (32) φ Optimize the objective by replacing the optimal policy with policy π φ , to directly optimize the strategy π using the dataset φ The purpose is to maximize the probability difference between the strategy outputting good and bad behaviors under the same state, as shown below:

[0232]

[0233] The above optimization objectives are introduced into the reinforcement learning algorithm strategy optimization process. At the same time, combined with actual driving conditions and human safety expectations for driving decisions, riskier driving behaviors are considered disadvantageous behaviors. This allows the behavioral risk strategy optimization goal to be achieved in the reinforcement learning algorithm, reducing the impact of insufficient reward function design on strategy optimization.

[0234]

[0235] Where, π φ′ is π φ The objective function of a + ,a - are respectively the superior and inferior behaviors based on subjective risk classification, and the target strategy function π φ′ Perform two random samplings and classify the risk through the behavioral risk classifier. The lower the driving risk, the better the behavior. - -C + ) / 4 is the behavioral risk category difference coefficient; C∈{0,1,2} are the three behavioral risk categories of safe, low-risk and high-risk respectively;

[0236] For behaviors with the same risk category, we further adopt the value function Q θ To improve the accuracy of behavior reward estimation, this paper introduces a dual time scale update rule for Q θ The dual time scale update rule is optimized by increasing Q θ The update frequency can improve the accuracy of the evaluation.

[0237] After building the above-mentioned driving decision-making strategy based on behavioral risk reinforcement learning, it can be trained, tested and verified in the roundabout simulation scenario.

[0238] Example 2

[0239] This example uses a virtual driving simulator to build a roundabout simulation test platform and conducts testing and verification of an intelligent vehicle's roundabout driving decision-making strategy based on behavioral risk reinforcement learning. The conditions for ending a simulation round are defined as follows: when the vehicle reaches the planned endpoint, collides, crosses the road boundary, or remains stationary for an extended period, the simulation round ends and triggers an environment reset.

[0240] The safety of the island driving decision-making strategy based on behavioral risk reinforcement learning was tested and verified in a simulation scenario. First, the basic reinforcement learning algorithm, the reinforcement learning algorithm with dual time scale update rules, and the behavioral risk reinforcement learning algorithm were trained in the same island driving scenario to perform island driving tasks such as merging, driving around the island, and exiting the island. The rewards during training are as follows: Figure 5 As shown, the curve represents the mean reward per episode, and the light-colored area represents the standard deviation of the reward per episode. All three reinforcement learning algorithms achieved a reward value of 700, demonstrating that the algorithms possess the basic ability to complete the round-island driving task. The rise in the mid-term reward curves reveals that the dual-timescale update rule reinforcement learning algorithm and the behavioral risk reinforcement learning algorithm achieved faster performance improvements than the basic reinforcement learning algorithm, demonstrating that the dual-timescale update rule can accelerate algorithm convergence. Furthermore, the behavioral risk reinforcement learning algorithm achieved a more significant improvement than the dual-timescale update rule reinforcement learning algorithm, further demonstrating that the behavioral risk reinforcement learning algorithm's policy iteration direction has been modified, allowing the policy to iterate more quickly toward reward and subjective expectation. In the later stages of training, the behavioral risk reinforcement learning algorithm achieved higher rewards at convergence than the other algorithms, demonstrating that behavioral risk-based algorithm optimization can help the algorithm avoid prematurely falling into local optima, thereby achieving safer and more efficient round-island driving decision-making performance.

[0241] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A decision-making method for intelligent car driving around an island based on behavioral risk reinforcement learning, characterized by: The following steps are involved: Step 1: Build a roundabout scene model; Step 2: Build a self-learning behavior risk classifier; Step 3: Model the behavioral risk reinforcement learning driving decision-making algorithm to enable intelligent vehicles to make decisions on roundabout driving; The specific method of step 2 is as follows: 21) Construct a multi-layer perceptron with the input of state space s and action space a, and the output of the machine is the probability of three behavioral risk categories: safe, low risk, and high risk. 22) Train the classifier as follows: 1) Data Collection and Labeling: Based on the existing roundabout decision-making strategy, simulated roundabout driving data is collected. State-action pairs {s, a} are randomly sampled from the collected data and labeled with behavioral risk categories to assess the risk of driving decision-making behaviors. Three behavioral risks are determined based on the following criteria: high-risk behaviors are those that, if executed, will further deteriorate driving safety under dangerous conditions such as collision, lane departure, or high-speed steering; safe behaviors are those that follow or change lanes under safe conditions, or those that avoid further deterioration of the driving condition under dangerous conditions; and low-risk behaviors are those that, under safe conditions, will deteriorate driving safety. 2) Pre-training: Use labeled data to pre-train the classifier based on supervised learning, so that it has the initial ability to distinguish different risk categories; 3) Self-learning Training: During the prediction phase, the classifier first makes predictions on unlabeled roundabout data. High-confidence screening is performed, and state-action pairs with a risk category prediction confidence exceeding 90% are labeled as corresponding predicted categories. This data is then added to the next round of training, and the training cycle repeats, with the self-learning process continuously updating the classifier. 4) Termination of training: When the classification accuracy of the classifier on the test set no longer improves, the training ends; A joint loss consisting of focal loss and label distribution-aware margin loss is used to address the problems of class imbalance and difficult sample mining in pre-training and self-learning training. The joint loss function is constructed as follows: Where γ is the hyperparameter for adjusting the weight factor; p y ∈[0,1] is the predicted value; α y =3∑ j n j / n y is the weight parameter for balancing the number of samples, where n j is the number of samples in each category; Design the parameter α related to the category proportion in the loss function y ,n y As the training changes dynamically, the loss function can be adaptively adjusted.

2. The intelligent car roundabout driving decision-making method based on behavioral risk reinforcement learning according to claim 1 is characterized in that: The specific method of step one is as follows: 11) Design action space; 12) Design state space; 13) Design a multi-objective reward function for intelligent vehicles.

3. The intelligent car roundabout driving decision-making method based on behavioral risk reinforcement learning according to claim 2 is characterized in that: The specific method of step 11) is as follows: Combine intelligence and drive into the normalized signal p required for simulating vehicle longitudinal acceleration and deceleration control x ∈[-1,1], and the front wheel active steering signal δ e ∈[-1,1] to construct the horizontal and vertical coupling action space A of the intelligent car e : A e =[p ex ,d e ] (1).

4. The intelligent car roundabout driving decision-making method based on behavioral risk reinforcement learning according to claim 2 is characterized in that: The specific method of step 12) is as follows: Design a driving task vector V consisting of key tasks and driving tasks T ,as follows: Where ΔP END is the coordinate of the vehicle’s current position relative to the end point; and are the vehicle's relative forward path point P i The distance and orientation deviation, P i Progressive as the vehicle moves forward; Design the surrounding environment state vector V composed of surrounding vehicle information E ; The vehicle's surroundings are divided into seven detection intervals from P0 to P6 by using the region cutting coding method; the high interaction detection intervals P0, P1, P4, and P5 are expanded; when the external perception module detects another vehicle that is traveling in the same direction as the vehicle and whose center point projection is within the detection interval, it is marked as the perceived vehicle, and the corresponding vehicle information V i Expressed as: V i =[F,△L,d -1 ,△v,δθ,I],i={0,...,6}, (3) Where F∈{0,1} is the flag of whether there is a vehicle; ΔL, d, Δv=[Δv x ,Δv y ] and δθ are the relative lane offset, distance, speed and relative orientation between the other vehicle and the vehicle; I∈[-1,1] is the possibility of the other vehicle changing lanes; When there is no other vehicle in the detection range, the corresponding setting Finally, the surrounding environment vector V is obtained E for: V E =[V0,V1,V2,V3,V4,V5,V6] (4) Design includes the vehicle speed information v e =[v ex ,v ey ], acceleration information a e =[a ex ,a ey ], driving direction information h e =[cosθ e ,sinθ e ] and yaw rate information ω e The vehicle state vector, with the future target point preview information added To comprehensively establish the state vector V e : V e =[v e ,a e ,h e ,oh e ,Dth e ,A e ], (5) Where θ e is the heading angle of the ego vehicle; is the angular difference between the directions of the three future target points and the heading angle of the vehicle; In summary, the state space is formed: S=[V T ,V e ,V E ] T (6)。 5. The intelligent car roundabout driving decision-making method based on behavioral risk reinforcement learning according to claim 2 is characterized in that: The specific method of step 13) is as follows: Design collision reward function r c for: Where, min{d i } is the shortest distance between all vehicles and the self-vehicle in the detection interval; D c is the dangerous collision distance threshold; Through vertical and horizontal safety rewards f and r l Boot as follows: r f =-0.5|min(0,Δv fx )|(D f -d f ), (8) r l =-2.5|min(0,Δv ky )|(D l -d k ) 2 , (9) Where Δv fx and d f ∈(D c ,D f ) is the relative longitudinal speed and distance between the preceding vehicle and the vehicle; Δv ky and d k ∈(D c ,D l ) is the relative lateral speed and distance between the nearest vehicle next to the vehicle and the vehicle; D f and D l The longitudinal and lateral safe vehicle distances; Design roundabout entrance safety reward RE , as shown below: r RE =-0.25||△v i ||(D RE -d i ), (10) Where, d i ∈(D c ,D RE ) and Δv i are the distance and relative speed vector between the ego vehicle and the high-risk vehicle respectively; D RE Safe distance for high-risk interactions; The security reward function is obtained as follows: R S ＝r c +r f +r l +r RE , (11) Establish the reward r for reaching the end point succ , as shown below: r succ =100,||△P End ||≤D P , (12) Where D P is the distance threshold between the vehicle and the path point P; Introducing path tracking reward r tag , as shown below: Where, is the driving distance of the vehicle between the two target points; Build Speed ​​Bonus sp , as shown below: Design Course Rewards δθ Guide the vehicle to move in the specified road direction as shown below: Where V lim is the road speed limit; δθ e is the deviation between the vehicle heading and the road direction; Θ δθ1 ,Θ δθ2 is the vehicle driving direction deviation threshold; Design the driving task reward function as follows: R T ＝r succ +r tag +r δθ +r sp (16) Design Turn Rewards st , as shown below: r st =-2|δ e |v x (17) Designing lane keeping rewards lk , as shown below: Where δd is the lateral offset of the current position of the main vehicle relative to the center line of the road; L is the road width; when the vehicle crosses the road boundary, that is, when |δd| ≥ L, r lk =-20 ends the current round and resets the environment; Define the control instruction reward r control , as shown below: r control =-4||A cur -A per ||2 (19) Where A per ,A cur are the action vectors at the previous and next moments respectively; Establishing yaw comfort bonus ω , guiding the car's yaw rate to change smoothly, as shown below: r ω =-((ω-Ω ω ) / 5) 2 ,Oh ω <ω (20) Where, Ω ω is the yaw rate threshold; Design driving behavior reward function R C , as shown below: R C ＝r lk +r ω +r st +r control (21) Finally, the multi-objective reward function of the intelligent car is obtained as follows: R=R S +R T +R C (22)。 6. The intelligent car roundabout driving decision-making method based on behavioral risk reinforcement learning according to claim 1 is characterized in that: The specific method of step three is as follows: 31) By introducing policy entropy To encourage the policy to produce more diverse exploration behaviors, as shown below: Where, π φ is the policy function; 32) Reinforcement learning algorithms consider policy entropy through Bellman regression operators and temporal difference pairs The Q-value function is iteratively solved as follows: The cost function of the algorithm value network is defined as Where s, a, r, s′ are the current state, action, reward, and next state of the sample respectively; α is the entropy weight coefficient; θ i , They are value network and target network respectively; φ is the policy network; Under the constraints Under this condition, the entropy weight coefficient α is automatically adjusted to balance the policy entropy The ratio between the expected return and the cost function of the hyperparameter α is designed as follows: Where, is the entropy constraint threshold; 33) The iterative objective is designed to minimize the information divergence between the policy function π and the Q-value function, that is, to maximize the expected reward and policy entropy at the same time As shown below: Where, π φ (s) = f φ (s,ξ) is the sampling value of the policy distribution under the reparameterized skill; is the noise that obeys the standard normal distribution; K(s) is the partition function and does not participate in the strategy optimization; 34) Using the state-action dataset, we iteratively optimize by maximizing the expected difference in rewards between good and bad behaviors under the same input, so that the reward network has the ability to accurately evaluate good and bad strategies. The cost function is as follows: Where r ψ To reward the network; a + ,a - are the superior and inferior behaviors corresponding to the same state s; σ is the sigmoid function; 35) Information divergence is added to limit the deviation between the new strategy and the original strategy. The cost function is defined as: Where, π φ is the trained policy model; π ref is the initial model; β is the parameter that controls the degree of deviation between the two strategies; Z = ∑ a π ref (a|s)exp(r(s,a) / β) is the partition function, which is only related to the state s and π ref is related to the optimized strategy π φ irrelevant; Strategy π φ The optimization goal is equivalent to minimizing the difference between the numerator and the denominator, and the optimal strategy π * Defined as: Further substitute formula (31) into formula (30), and we get the strategy π φ The optimization goal is to minimize the training strategy π φ and the optimal strategy π * The deviations between the distributions are as follows: From formula (30), we can get the reward network r ψ and the optimal strategy π * The relationship between them is: 36) Use the relationship between the reward network and the optimal strategy to eliminate the reward network r in formula (28) ψ , and based on the maximum likelihood optimization idea and the strategy π of formula (32) φ Optimize the objective by replacing the optimal policy with policy π φ , to directly optimize the strategy π using human-labeled data φ The purpose is to maximize the probability difference between the strategy outputting good and bad behaviors under the same state, as shown below: By assessing the risk of sampled driving behaviors, we can improve the direction of policy iteration and reduce the impact of insufficient reward function design on policy optimization, as shown below: Where, π φ′ is π φ The objective function of a + ,a - They are respectively the good and bad behaviors based on behavioral risk classification, and the target strategy function π φ′ Perform two random samplings and classify the risk through the behavioral risk classifier. The lower the driving risk, the better the behavior. - -C + ) / 4 is the behavioral risk category difference coefficient; C∈{0,1,2} corresponds to the three risk behavior labels of safe, low-risk and high-risk, respectively.

Images