Security constraint behavior decision-making system considering social preferences of vehicles behind target lane
Through a layered reinforcement learning system to identify and decide the social preferences of the target lane behind the vehicle, the SVM and PPO-DQN algorithms are used to solve the problem of self-driving vehicles dealing with time-transforming social preferences in a dynamic environment, and improve the efficiency and safety of behavioral decision-making sex.
Patent Information
- Application Number
- CN202411422069.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-22
AI Technical Summary
It is difficult for autonomous vehicles to effectively deal with the time-varying and volatile social preferences of vehicles behind the target lane in a dynamic environment, resulting in an increase in the complexity of behavioral decision-making. The training cycle of existing technologies is long and difficult to converge, which affects traffic safety.
The hierarchical reinforcement learning system is adopted, including the first and second social preference recognition models, and the SVM and PPO-DQN algorithms are used, combined with the CNN and LSTM-CNN architectures to identify and decide to consider the social preferences of the target lane behind the car when changing lanes. By changing lane return as a security constraint, the training complexity is reduced and the algorithm convergence speed is improved.
It improves the efficiency and safety of behavioral decision-making strategies of autonomous vehicles in multi-lane dynamic and complex environments, reduces the difficulty of training cycles and algorithm convergence, and enhances the generalization and interpretability of the model.
Smart Images

Figure CN118991829A_ABST
Abstract
Description
Technical Field
[0001] This case involves autonomous driving control, and in particular, a safety constraint behavior decision system that considers the social preferences of the following vehicle in the target lane. Background Art
[0002] Due to the influence of multiple variable factors such as real-time traffic conditions and one's own emotions, the social preferences of vehicles behind the target lane often have certain time-varying and volatile characteristics. For example, when an altruistic driver interacts with a neighboring vehicle that suddenly changes lanes, he or she may sometimes suddenly show selfish social preferences such as not giving way due to emotional fluctuations, which makes autonomous driving behavior decisions more complicated. Summary of the invention
[0003] The purpose of this case is to propose a safety constraint behavior decision system that considers the social preferences of the vehicles behind the target lane, which can make effective autonomous driving behavior decisions based on the time-varying and volatile social preferences of the vehicles behind the target lane. The specific technical solution is as follows.
[0004] On the first aspect, this case proposes a safety constraint behavior decision system that takes into account the social preference of the vehicle behind the target lane. The system uses hierarchical reinforcement learning to implement autonomous driving behavior decisions, including a first social preference recognition model, a second social preference recognition model, an upper module and a lower module; the first social preference recognition model is configured to identify the social preference type of the nearest vehicle behind the adjacent lane that enters the range of the autonomous vehicle, and obtain a first recognition result; the second social preference recognition model is configured to identify the social preference type of the nearest vehicle behind the target lane, and obtain a second recognition result; the upper module is configured to make autonomous driving behavior decisions based on the driving conditions of the autonomous vehicle, the driving conditions of the adjacent vehicles and the first recognition result; when the first recognition result is selfish, the autonomous driving behavior decision is to change lanes to the lane where it is located, then a penalty is imposed; the lower module is configured to, when the autonomous vehicle starts to change left or right lanes, if the second recognition result is selfish, the autonomous driving behavior decision executed is to change lanes and return.
[0005] In one implementation of the above technical solution, the first social preference recognition model adopts an SVM model, and its training data is clustered using principal component factor analysis PCA combined with kmeans.
[0006] In one implementation of the above technical solution, the second social preference recognition model adopts an SVM model, and its training data uses principal component factor analysis PCA combined with hierarchical clustering to cluster the social preferences of the vehicles behind the target lane during the lane changing process.
[0007] In one implementation of the above technical solution, the upper layer module adopts the PPO algorithm for training and learning, and the lower layer module adopts the DQN algorithm for training and learning.
[0008] In one implementation of the above technical solution, the upper layer module adopts a CNN architecture, and the lower layer module adopts a LSTM-CNN architecture.
[0009] In one implementation of the above technical solution, the second recognition result is obtained based on the mean longitudinal acceleration of the vehicle behind the target lane, the maximum longitudinal acceleration of the vehicle behind the target lane, the standard deviation of the longitudinal acceleration of the vehicle behind the target lane, the mean longitudinal jerk of the vehicle behind the target lane, the mean maximum longitudinal jerk of the vehicle behind the target lane, and the standard deviation of the longitudinal jerk of the vehicle behind the target lane.
[0010] In one implementation of the above technical solution, the first recognition result is obtained based on the mean longitudinal speed of the following vehicle, the maximum longitudinal speed of the following vehicle, the standard deviation of the longitudinal speed of the following vehicle, the mean longitudinal acceleration of the following vehicle, the maximum longitudinal acceleration of the following vehicle, the standard deviation of the longitudinal speed of the following vehicle, the mean following distance of the following vehicle, the minimum following distance of the following vehicle, and the standard deviation of the following distance of the following vehicle, and the following vehicle is the closest vehicle behind the adjacent lane that enters the range of the autonomous vehicle.
[0011] In one implementation of the above technical solution, the path trajectory of the action corresponding to the autonomous driving behavior decision is composed of 3 sampling points.
[0012] On the second aspect, the present case proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the above-mentioned systems.
[0013] On the third aspect, this case proposes a safety constraint behavior decision method that takes into account the social preference of the vehicle behind the target lane, and the method includes the following steps: identifying the social preference type of the nearest vehicle behind the adjacent lane that has entered the range of the autonomous vehicle, obtaining a first recognition result, and making an automatic driving behavior decision based on the driving condition of the autonomous vehicle, the driving condition of the adjacent vehicle and the first recognition result; when the first recognition result is selfish, if the automatic driving behavior decision is to change lanes to the lane where it is located, a penalty is imposed; when the autonomous vehicle starts to change left or right lanes, if the second recognition result is selfish, the automatic driving behavior decision executed is to change lanes and return.
[0014] The beneficial technical effects of this case: Compared with the existing technology, it helps to improve the efficiency of the optimal behavior decision-making strategy of autonomous driving vehicles in dynamic environments and improve the safety of autonomous driving vehicles in multi-lane dynamic and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0016] Figure 1 , one Schematic diagram of the process of constructing an automatic driving lane changing decision model in an embodiment.
[0017] Figure 2 , one Schematic diagram of the controller structure in this embodiment. DETAILED DESCRIPTION
[0018] During the lane-changing process of an autonomous vehicle, the drivers of vehicles behind the target lane have significant differences in psychology, beliefs, and values. Their social preferences show a strong driving style of selfishness, altruism, and reciprocity, making autonomous driving behavior decisions complicated.
[0019] Lane changing strategies that consider social preferences in the prior art all assume that the driving style is fixed and stable, and is not time-varying, which is inconsistent with the actual traffic environment. They do not consider the situation where the social preferences of the vehicle behind the target lane may change after the agent starts changing lanes. For example, if the vehicle behind the target lane is a selfish vehicle, if the agent speeds up to overtake, it is easy to cause the vehicle behind to become more angry and other negative emotions, which will be detrimental to traffic safety.
[0020] The autonomous driving behavior decision system that considers the social preferences of surrounding vehicles with application number CN202410072764.5, although it adds three social preference detection points of the rear vehicle in the target lane, the social preference real-time of the front vehicle in the current lane, and the social preference detection point of the front vehicle in the adjacent lane with the intention to cut into the lane in the state space state, although it reflects the time-varying and volatility of social preferences, but due to the use of the HDQN algorithm, the goal space directly adopts the behavior decision of lane change and return, which will cause a long training cycle, making it difficult for the training effect of reinforcement learning to converge, and at the same time increase the computing power and training time of the industrial computer. Moreover, the decision to change lanes and return that needs to be executed during the lane change process is prone to sparse rewards, and there is a problem of high computational complexity, which is not conducive to real-time decision-making.
[0021] Based on this, this case uses the lane change return measure as a safety constraint within the lane change goal, that is, a safety redundancy measure for the lane change decision. By triggering the lane change return condition, the safety constraint measure is directly executed, which not only reduces the complexity of training, but also improves the training speed and greatly reduces the difficulty of algorithm convergence. The algorithm is trained in special scenarios and ordinary scenarios respectively. After the models of the two scenarios are trained, the algorithm implementation is completed based on multi-task learning to increase the generalization and interpretability of the model call. The special scenario is a scenario where the vehicle behind the target lane is a selfish social preference, and the ordinary scenario is a scenario where the vehicle behind the target lane is a non-selfish social preference.
[0022] The following will be combined with the accompanying drawings to clearly and completely describe how to implement this technical solution. Obviously, the described implementation method is only a part of the implementation method of this case, not all the implementation methods. Based on the implementation method in this case, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0023] 1. Social preference recognition model for the vehicle behind the target lane
[0024] The target lane is the lane that the vehicle is going to enter from the current lane. According to the definition of social psychology, social preference refers to the embodiment of people's innate sociality at the preference level. In mixed traffic scenarios, the social preferences of the vehicles behind the target lane are egoistic, reciprocal, and altruistic, respectively, to distinguish the longitudinal driving modes of drivers with different styles.
[0025] There are two main scenarios for clustering and identifying social preferences of vehicles behind the target lane. Scenario 1: The vehicles behind the target lane are not disturbed by the lane change of the neighboring vehicle, that is, the following driving behavior. Scenario 2: After the autonomous vehicle starts to change lanes and during the lane change, the vehicles behind the target lane are affected by the lane change behavior of the agent vehicle, which has the characteristics of time-varying and fluctuating social preferences.
[0026] The social preference recognition models for the two scenarios are trained using the offline datasets of the corresponding scenarios. For scenario 1, the offline dataset that can be used is the driving data of the stable following condition of the vehicle in the 2-4 lanes of the i80 section of the NGSIM dataset, that is, the driving behavior data related to the small vehicle driving in a straight line for more than 20 seconds and the vehicles in the left and right adjacent lanes have no lane change behavior during this period; the vehicle type is selected as a sedan. For scenario 2, the offline dataset that can be used is the driving data of the 2-4 lanes in the i80 section of the NGSIM dataset, and the vehicle type is selected as a sedan. The data consists of two parts of the scene datasets during the lane change process. The first part is the scenario where the lane-changing vehicle successfully changes lanes. For the condition where there is only one lane change behavior and the lane change is successful in the dataset, the process from the lane-changing starting point to the adjacent lane line of the lane-changing vehicle is intercepted, and the interaction data of the vehicle behind the target lane is extracted; at the same time, the lane-changing vehicle and the vehicle data of the vehicle behind the target lane without the vehicle in front are eliminated; and the vehicles that stop in the middle are eliminated. The second part is the scenario where the lane-changing vehicle fails to change lanes and returns to the original lane. For the unsuccessful lane-changing conditions in the data set, the process of the lane-changing vehicle driving from the lane-changing starting point to the lane-changing end point (return point) is intercepted, and the interaction data of the vehicle behind the target lane is obtained in advance. At the same time, the vehicle dataset of the lane-changing vehicle and the vehicle behind the target lane without the vehicle in front is eliminated; the vehicles that stopped midway are eliminated.
[0027] Preprocess the offline data and use the Savitzky-Golay filter algorithm to localize the coordinate position. _ X. Local _ Y is smoothed, and the window length is selected as 21 and the polynomial order is 3. The characteristic parameters such as velocity and acceleration are determined by Local _ X. Local _ The derivative of Y.
[0028] Relevant feature vectors are extracted from the preprocessed data. For the following car dataset of scenario 1, the extracted feature vectors are shown in Table 1. The following car in Table 1 is the closest vehicle behind the adjacent lane that enters the range of the autonomous vehicle. For the lane change interaction dataset of scenario 2, the extracted feature vectors are shown in Table 2.
[0029] Table 1
[0030]
[0031] Table 2
[0032]
[0033] For scenario 1, the extracted feature vectors are normalized, and the common algorithm principal component factor analysis PCA combined with kmeans is used to cluster the three types of social preferences: selfish, reciprocal and altruistic. The commonly used algorithm SVM is used for classification and the first social preference recognition model SVM_1 is stored.
[0034] For scenario 2, the extracted feature vectors are normalized, and the common algorithm principal component factor analysis PCA combined with hierarchical clustering method is used to cluster the social preferences of the vehicles behind the target lane during the lane changing process. The second social preference recognition model SVM_2 is classified based on the commonly used algorithm SVM and stored.
[0035] The trained first social preference recognition model SVM_1 and the second social preference recognition model SVM_2 can be used for online recognition of the social preference of the vehicle behind the target lane (see Figure 1 ). The autonomous vehicle has sensors, and the sensor acquisition frequency is 0.1s. The Agent extracts the feature vector value from the sensor data (usually within 2s). For the nearest vehicle behind the adjacent lane that enters the Agent range, collect information for a certain period of time (within 2s), extract the feature vector, and use the SVM_1 model to identify the social preference of the vehicle behind the adjacent lane when it is in the following state, and obtain the first recognition result. The first recognition result is recorded as C pre_t_range , its value is {1, 0, -1}, the values represent selfish, reciprocal, and altruistic respectively, and the default value is 0.
[0036] (II) PPO-DQN Hierarchical Reinforcement Learning Model
[0037] (2.1) Configure a multi-lane driving environment, filter out unreasonable traffic scenarios, and complete the construction of autonomous driving decision-making training scenarios on structured roads.
[0038] The multi-lane driving environment includes the number of scene lanes, the width of each lane, the total length of the road, the speed limit of each lane, the number of surrounding vehicles, etc. In one embodiment, the number of scene lanes is 3, the width of each lane is 3.75, and the length is 400m. There are 6 surrounding vehicles, each distributed in front, behind, left and right of the Agent vehicle. The surrounding vehicles are based on the data set selected by NIGSIM, and the speed limit of each lane is set to v_min=6m / s and v_max=19m / s. In order to allow the main vehicle to better train the lane change decision, the vehicle in front of the main vehicle is set as a slow vehicle. The speed of the surrounding vehicles is randomly set within the speed limit, the initial position is randomly generated and TTC>3.
[0039] The surrounding vehicles of the Agent are set. The speed and acceleration attributes of non-lane-changing vehicles are set based on the social preferences clustered from the car-following dataset to set the minimum, average, and maximum speeds and accelerations; the speed and acceleration attributes of lane-changing vehicles are set based on the social preferences clustered from the lane-changing dataset to set the minimum, average, and maximum speeds and accelerations. The social preference types of non-lane-changing vehicles and lane-changing vehicles of the surrounding vehicles are randomly set and remain stable. For non-lane-changing vehicles that are behind the target lane, their social preferences are randomly reset when the Agent changes lanes.
[0040] The Agent vehicle speed sets the optimal vehicle speed based on the average speed V of the road section traffic flow. agent_target_0 Set to {δ0V agent_target_0 , V agent_target_0 ,δ1V agenttarget_0} Choose L for the three speeds of low, medium and high s , w a L s , w b L s δ0 is the low speed coefficient, which is less than 1; δ1 is the high speed coefficient, which is greater than 1.
[0041] Eliminate unreasonable traffic flow scenarios. For example, the agent cancels left lane changes when it is in the leftmost lane and cancels right lane changes when it is in the rightmost lane.
[0042] (2.2) Constructing the PPO-DQN hierarchical reinforcement learning model
[0043] The upper-level decision model PPO (Proximal Policy Optimization) algorithm of the PP0-DQN algorithm provides a more flexible and stable decision-making framework, while allowing the lower-level decision model DQN (Deep Q-Network) algorithm to focus on specific action selection, thereby improving learning efficiency and stability when facing complex tasks. The lane change return measures are used as safety constraints within the lane change goal, that is, safety redundant measures for lane change decisions. By directly executing safety constraint measures when the lane change return conditions are triggered, not only the complexity of training is reduced, but also the speed of training is improved and the difficulty of algorithm convergence is greatly reduced. The algorithm is trained in special scenarios and ordinary scenarios respectively. After the models of the two scenarios are trained, the algorithm implementation is completed based on multi-task learning, which increases the generalization and interpretability of model calls.
[0044] Specifically, the PPO-DQN layered reinforcement learning model is a two-layer deep learning network architecture (see Figure 1The dark gray part) is briefly divided into an upper module (meta-controller) and a lower module (controller). The meta-controller module uses the PPO algorithm, and its deep learning network structure is the CNN architecture; the controller module uses the DQN algorithm, and its deep learning network structure is the LSTM-CNN architecture.
[0045] The state space of the upper module is denoted as s, which is composed of the agent, the operating parameters of its neighboring vehicles, and the type of social preference. i , C pre_trange}, i = 1, 2, 3. AV represents the driving status of the Agent, and MV1-MV3 represent the driving status of the three vehicles closest to the Agent. pre_trange It is the social preference of the vehicle closest to the target lane, identified using the SVM_1 model.
[0046] AV={V lat , V lon ,yaw,laneID},V lat is the lateral velocity, V lon is the longitudinal velocity, yaw is the yaw angle, and lane_ID is the lane ID of the autonomous vehicle.
[0047] MV i ={L 1on_i , L 1at_i , ΔL lat_i , ΔL 1on_i , α MV_lon_i , α MV_lat_i , laneID mv_i}, L lon_i is the relative longitudinal distance, L lat_i is the relative lateral distance, ΔL l_i is the relative lateral velocity, ΔL lon_i is the relative longitudinal velocity, α MV_lon_i is the longitudinal acceleration, α MV_lat_i is the lateral acceleration, laneID mv_i MV for Zhouche i The lane ID of the
[0048] In one implementation, the upper layer module is composed of an Actor network and a Critic network. Actor network training uses tf.GradientTape() to record the calculation process; calculates new action goal probabilities and clipping ratios, and uses temporal difference (TD) errors to calculate actor losses; calculates entropy losses to promote exploration and merges them into actor losses. Critic network training uses tf.GradientTape() to record the calculation process, predict the values of the current and next states; calculates temporal difference (TD) errors, and updates losses based on rewards and future values; obtains the gradient of the Critic network by calculating the gradient of the loss. The gradient of the actor network is clipped to avoid excessive updates. Use an optimizer to update the parameters of the actor and critic, and finally update the old policy to ensure the stability of the policy.
[0049] The upper module uses the PPO algorithm for training and learning, which interacts with the environment and t ) Select a target goal from the subtask goals using the output probability distribution of the policy network t Execute; if and only if the underlying module Controller completes a goal t When the task is completed, Meta-controller generates a new goal t+1 .
[0050] The above goal space is three kinds of autonomous driving behavior decisions: going straight, changing lanes left, and changing lanes right. In one implementation, 0, 1, and 2 are used for going straight, changing lanes left, and changing lanes right, respectively, which means goal = {0, 1, 2}.
[0051] The lower-level module uses the DQN algorithm for training and learning, based on the subtask goal of the upper-level module t And state history information, use Q function to select an action to execute, that is, based on the current goal t and state history information to select the action that maximizes the reward.
[0052] In one implementation, the state history information is recorded as h t ,h t =(s t-2 ,s t-1 ,s t ).
[0053] In one implementation, the Q function is:
[0054]
[0055] Where: g is the target in state s, πag=P(a|s,g) represents the action strategy, γ is the discount factor, E is the expected return, r is the reward, When action g is executed in state s at time t, the state changes to s t+1 Next, perform action a t+1 The maximum expected return to achieve goal g is Q1, and Q2 is the maximum expected return of performing action a in state s to achieve goal g.
[0056] In one implementation, action is a target path point and a target vehicle speed for the autonomous vehicle behavior decision.
[0057] Combination Figure 2 , a specific implementation of the lower-level module is: h t As the input of LSTM, goal t As the input of the fully connected layer, the output of the LSTM and fully connected layers is used as the input of the CNN model, and an action is output. Among them, the LSTM input is three time points, each time point is 26 dimensions, that is, (3, 26), and the output last dimension is 512*1. The goal is empty and input to the fully connected layer. The output last dimension is 512*1, and the activation function is relu. The first hidden layer of the CNN model is a fully connected layer. The data after the goal is processed by the spatial fully connected layer and the state history space is processed by the LSTM layer are used as the input of CNN. The last dimension of the output is 256*1, and the activation function is relu; the second hidden layer is a fully connected layer, and the last dimension of the output is 64*1, and the activation function is relu; the output layer is a fully connected layer, outputting the action space, and the last dimension of the output is 3*1, and the activation function is linear. In training, the lower module uses the mean square error (MSE, Mean-SquaredLoss) to calculate the loss function. RMSprop is used as the optimizer, with decay exponent rho=0.95, learning rate lr=0.00025, and epsilon=0.01.
[0058] The action space is the target trajectory sampling point of the agent on the center line of the current lane or the adjacent lanes on both sides, action = {ptij}. i is the path sampling point in the lateral direction, represented by i = {1, 2, 3}, corresponding to the point on the center line of the left lane, the current lane, and the right lane of the vehicle. 1 is the left lane change distance, 3.75m, and 3 is the right lane change distance, -3.75m. j is the path sampling point in the longitudinal direction, located on the center line of the current lane or the center line of the adjacent lane along the vehicle's forward direction, represented by j = {1, 2, 3}, corresponding to the point in front of the vehicle L s , w a L s , wb L s The sampling point is at meters, and the longitudinal sampling spacing is L. s m, w a 、w b is the sampling distance coefficient, w a 、w b are greater than 0 and w a <w b .
[0059] The actions satisfy the following constraints: (1) When goal = 0, the autonomous vehicle follows and moves straight, and the action space is = {pt 2j}, the space size is 3. (2) When goal = 1, the autonomous vehicle changes lanes to the left, and the action space is = {pt 1j}, the space size is 3. (3) When goal = 2, the autonomous vehicle changes lanes to the right, and the action space is = {pt 3j}, the space size is 3.
[0060] The reward value is set according to the social preference of the vehicle behind in the target lane.
[0061] The straight driving behavior (i.e., following behavior) of the vehicle behind the target lane that is not disturbed by the lane change of the neighboring vehicle is the social preference stability characteristic maintained by the vehicle behind the target lane before the autonomous vehicle starts to change lanes. Assuming that the effective distance for the autonomous vehicle to detect the social attributes of the rear vehicle is 60m, when there are vehicles behind the lanes on both sides entering the range of 60m, the autonomous vehicle uses the first social preference recognition model to detect the social attributes of the vehicles entering the range. If the social preference of the nearest rear neighboring vehicle is identified as selfish, and the autonomous vehicle changes lanes to the lane where the autonomous vehicle is located when changing lanes, the autonomous vehicle will be punished, thereby encouraging the autonomous vehicle to choose to go straight or choose another lane to change lanes when making decisions. It can be expressed as:
[0062]
[0063] When the Agent collides with surrounding vehicles, a penalty is imposed. For example, the penalty value is set to 500. The expression is R collision = -500, if the Agent collides with surrounding vehicles.
[0064] In addition, the bonus value also includes the operating condition bonus value.
[0065] When the agent's behavior decision goal is to go straight, if the agent's TTC in the following condition is less than 3, a penalty is imposed. While avoiding collisions, the agent can learn to make a behavior decision to change lanes when necessary. It can be expressed as: R follow_s_1=-300, if TTC<3, TTC is the collision time between the Agent and the nearest preceding vehicle in the lane.
[0066] When the Agent is more than 40m away from the front car and goal == 0 (i.e., going straight), the reward is 50. It can be expressed as: R follow_s_2 =50,if goal==0。
[0067] When the Agent is more than 40m away from the vehicle in front, action==pt iLs (i.e. the speed is the lowest and the shortest following path is chosen), the penalty is -50. It can be expressed as R follow_s _3=50,if action==pt 2Ls .
[0068] If the distance to the neighboring car is less than the set safety distance, when goal == 0 (i.e., go straight), the reward is 50, which can be expressed as R follow_s_4 =50,if goal==0。
[0069] If the distance to the neighboring vehicle is greater than the set safety distance, when goal == 1or 2 (i.e., left lane change or right lane change), the reward is 50, which can be expressed as R change_s_1 =50,if goal==1or 2.
[0070] When the agent reaches the destination, a reward is given, for example, the value is set to 1000. It can be expressed as R complete =1000, if ego_dis<5 meters, ego_dis is the distance between the agent and the endpoint.
[0071] In summary, the first reward function of the upper module is as follows:
[0072] R=R pre +R collision +R follow_s_1 +R follow_s_2 +R follow_s_4 +R change_s_1 +R change_s_1 +R complete
[0073] The second reward function of the lower module is as follows:
[0074] R=R pre +R collision +R follow_s_1 +R follow_s_2 ++R follow_s_3 +R follow_s_4 +R change_s_1 +R change_ s _1 +Rcomplete .
[0075] (III) Training
[0076] The PPO-DQN hierarchical reinforcement learning algorithm is trained in special scenarios and common scenarios respectively to obtain their respective training models.
[0077] During training, the scenario with lane-changing return in the goal lane-changing selection (i.e., the social preference of the vehicle behind the target lane after the agent changes lanes shows selfish characteristics) is defined as a special scenario, and the scenario without lane-changing return in the goal lane-changing selection (i.e., the social preference of the vehicle behind the target lane after the agent changes lanes does not show selfish characteristics) is defined as a normal scenario. At the same time, the social preference attributes of the vehicles behind the agent in the lanes on both sides of the agent before and after entering 60m are random and kept stable.
[0078] The local planning path of the agent from the current position to the target trajectory point is formed into a series of path points through cubic spline interpolation and curve smoothing. The acceleration and steering wheel angle output in the subsequent trajectory tracking process are completed by the PID algorithm, forming the agent's complete autonomous driving function. After the autonomous driving obtains the complete autonomous driving function, it then begins to use the maximum reward function in training to obtain the optimal behavior decision and complete the algorithm training.
[0079] When applied, when the Agent starts to execute the initial lane change, it collects vehicle information for a certain period of time (less than 2 seconds) for the vehicle closest to the target lane, extracts the feature vector, and uses the SVM_2 model to identify social preferences to obtain the second identification result. If the goal is performing a left or right lane change, and the second identification result of the vehicle closest to the target lane identified by the SVM_2 model is egoistic, the safety measure of lane change return is executed.
[0080] In one implementation, when changing lanes and returning, three points are selected: the vehicle's own position, the position a meters ahead of the center line of the lane (i.e., the position of the first point of the center line at a meter), and the position b meters ahead of the center line of the adjacent lane or the lane being changed (i.e., continuing to travel b meters after returning to the center line of the lane). Cubic spline interpolation is performed, and path smoothing is performed to form smooth tracking path points. The PID tracking algorithm is used to control the agent to complete trajectory tracking.
[0081] In summary, this case designs PPO_DQN hierarchical reinforcement learning based on the HDQN algorithm structure. The HDQN structure using discrete PPO as the upper module and DQN as the lower module can provide a more flexible and stable decision-making framework at a high level, while allowing the lower modules to focus on specific action selection, thereby improving learning efficiency and stability when facing complex tasks. In contrast, using DQN at both levels may face higher complexity and training instability when learning strategies. This case is based on the social preference attributes of the nearest car behind the target lane, PPO _ The behavior decision algorithm of DQN's hierarchical reinforcement learning adds a social preference reward setting for the stability of the vehicle behind the target lane, and a safety constraint for changing lanes based on the selfish social preference of the vehicle behind the target lane according to the goal. It uses the principle of multi-task learning for training and model loading testing, which can better adapt to the uncertainty in mixed traffic scenarios, improve the efficiency of the behavior decision algorithm, and improve the convergence effect and speed during training.
[0082] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0083] Through the above description of the implementation mode, the technicians in the relevant field can clearly understand that the system of the present case can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation mode in most cases.
[0084] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, which are merely illustrative and instructive, rather than restrictive. A person skilled in the art can make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which are protected by the present invention.
Claims
1. A safety constraint behavior decision system considering the social preference of the vehicle behind the target lane, characterized in that: The system uses hierarchical reinforcement learning to achieve autonomous driving behavior decision-making, including a first social preference recognition model, a second social preference recognition model, an upper module and a lower module; The first social preference recognition model is configured to recognize the social preference type of the nearest vehicle behind the adjacent lane that enters the range of the autonomous vehicle, and obtain a first recognition result; The second social preference recognition model is configured to recognize the social preference type of the vehicle closest to the rear of the target lane to obtain a second recognition result; The upper layer module is configured to make an autonomous driving behavior decision based on the driving condition of the autonomous vehicle, the driving condition of the neighboring vehicle and the first recognition result; when the first recognition result is egoistic, if the autonomous driving behavior decision is to change lanes to the lane where the autonomous vehicle is located, a penalty is imposed; The lower layer module is configured to, when the autonomous vehicle starts to execute a left lane change or a right lane change, if the second recognition result is a self-interested type, then the executed autonomous driving behavior decision is to change lanes and return.
2. The system according to claim 1, characterized in that The first social preference identification model adopts the SVM model, and its training data is clustered using the principal component factor analysis PCA combined with the kmeans method.
3. The system according to claim 1, characterized in that The second social preference identification model adopts an SVM model, and its training data uses principal component factor analysis (PCA) combined with a hierarchical clustering method to cluster the social preferences of the vehicles behind the target lane during the lane changing process.
4. The system according to claim 1, characterized in that The upper layer module adopts the PPO algorithm for training and learning, and the lower layer module adopts the DQN algorithm for training and learning.
5. The system according to claim 1, characterized in that The upper layer module adopts the CNN architecture, and the lower layer module adopts the LSTM-CNN architecture.
6. The system according to claim 1, characterized in that The second recognition result is obtained based on the mean longitudinal acceleration of the vehicle behind the target lane, the maximum longitudinal acceleration of the vehicle behind the target lane, the standard deviation of the longitudinal acceleration of the vehicle behind the target lane, the mean longitudinal jerk of the vehicle behind the target lane, the mean maximum longitudinal jerk of the vehicle behind the target lane, and the standard deviation of the longitudinal jerk of the vehicle behind the target lane.
7. The system according to claim 1, characterized in that The first recognition result is obtained based on the mean longitudinal speed of the following vehicle, the maximum longitudinal speed of the following vehicle, the standard deviation of the longitudinal speed of the following vehicle, the mean longitudinal acceleration of the following vehicle, the maximum longitudinal acceleration of the following vehicle, the standard deviation of the longitudinal speed of the following vehicle, the mean following distance of the following vehicle, the minimum following distance of the following vehicle, and the standard deviation of the following distance of the following vehicle. The following vehicle is the closest vehicle behind the adjacent lane that enters the range of the autonomous vehicle.
8. The system according to claim 1, characterized in that The path trajectory of the action corresponding to the autonomous driving behavior decision is composed of 3 sampling points.
9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the system as claimed in any one of claims 1 to 8.
10. A safety constraint behavior decision method considering the social preference of the vehicle behind the target lane, characterized in that: The method comprises the following steps: Identify the social preference type of the nearest vehicle behind the adjacent lane that has entered the autonomous vehicle, obtain a first identification result, and make an autonomous driving behavior decision based on the driving condition of the autonomous vehicle, the driving condition of the adjacent vehicle, and the first identification result; when the first identification result is a selfish type, if the autonomous driving behavior decision is to change lanes to the lane where the autonomous vehicle is located, a penalty is imposed; When the autonomous vehicle starts to execute a left lane change or a right lane change, if the second recognition result is a self-interested type, the executed autonomous driving behavior decision is to change lanes and return.
Citation Information
Cited By
Decision behavior multi-dimensional evaluation method and system under multiple driving modes, and medium
CN120440070A
