Automated driving behavior decision system considering social preferences of surrounding vehicles

Abstract

The application relates to an automatic driving behavior decision system considering social preferences of surrounding vehicles, which is used for solving the problem that automatic driving decision is difficult due to the uncertainty of the social preferences of drivers in the existing traffic environment. The scheme adopts hierarchical reinforcement learning to realize automatic driving behavior decision, and a meta-controller utilizes the historical state of an autonomous vehicle to make decisions on lane keeping and continuous driving, lane changing or returning to the original lane; wherein the returning to the original lane is realized by interrupting the lane changing of the autonomous vehicle, and the interrupting condition is that the social preference type of a vehicle behind a target lane recognized according to the current state is egoistic when the autonomous vehicle just starts to change lanes or the edge of the vehicle body just contacts the adjacent lane line of the target lane during the lane changing; a controller is based on state historical information and is used for sampling point planning of a target trajectory of the autonomous vehicle on the center line of the lane or the adjacent lane; the meta-controller and the controller adopt the same reward calculation including target decision based on the social preferences of surrounding vehicles.

Description

Technical Field

[0001] This invention relates to behavioral decision-making for multi-lane autonomous vehicles, and more particularly to an autonomous driving behavioral decision-making system that takes into account the social preferences of surrounding vehicles. Background Technology

[0002] Compared to the relative stability of driving style, interactive driving scenarios are insufficient to reflect the temporary changes in a driver's social value orientation that may occur due to variable factors such as mood and mindset. Therefore, the concept of "social preference" from psychology is introduced. According to the definition in social psychology, social preference refers to the manifestation of an individual's innate social nature at the preference level. Based on the differences in drivers' social value orientations, the different social value orientations of drivers in surrounding vehicles can be divided into three types: self-interest, reciprocity, and altruism.

[0003] In dynamic scenarios where surrounding vehicles exhibit uncertain social preferences, rule-based autonomous driving behavior decision-making design is challenging. Solutions combining social preferences with game theory algorithms, or lane-changing based on driving style with lane-changing rules, suffer from limitations in predefined rules that cannot cover all possible scenarios. While reinforcement learning (RL)-based behavior decision-making techniques generate rewards through continuous interaction between the autonomous vehicle (Agent) and the environment (Env), using these rewards to improve the Agent's strategy can enhance autonomous driving performance in interactive driving scenarios to some extent. However, due to the uncertainties inherent in these scenarios, classic RL behavior decision-making algorithms face challenges such as sparse rewards, high training intensity, and large spatial dimensionality. This is particularly problematic when autonomous vehicles need to return to their original lane during lane changes, leading to low convergence efficiency during training. The goal-based hierarchical reinforcement learning (HRL) autonomous driving behavior decision-making algorithm, H-DQN, has the advantages of simple structure and can greatly improve the dimensionality of the state space and convergence speed. However, the H-DQN algorithm faces the situation that the goal must be fully executed. That is, the goal will be maintained until the goal is achieved or the training sequence is completed, which is when the termination state is reached. It is not suitable for the behavior decision of lane change and return during lane change. It is more suitable for regular traffic scenarios. It is difficult to produce good results in time-varying and uncertain scenarios, especially for the decision of lane change and return during lane change, where the convergence effect is poor or even difficult to converge. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, the present invention aims to propose an autonomous driving behavior decision-making system that considers the social preferences of surrounding vehicles, applicable to mixed traffic environments where human-driven vehicles are present. To achieve the above technical objective, the specific technical solution is as follows.

[0005] Firstly, this case proposes an autonomous driving behavior decision-making system that considers the social preferences of surrounding vehicles. The system employs hierarchical reinforcement learning to achieve autonomous driving behavior decisions, including a meta-controller and a controller. The meta-controller is configured to make decisions based on the autonomous vehicle's state history information h. t =(s t-2 ,s t-1 ,s t The decision to maintain lane, change lanes, or return to the original lane is made; where: state s t This includes the autonomous vehicle's driving status at time t, the driving status of the nearest K vehicles at time t within the autonomous vehicle's perception range, and the social preference types of surrounding vehicles, where K is a set value. Social preference types include self-interested, reciprocal, and altruistic. Returning to the original lane is achieved by the autonomous vehicle interrupting a lane change. The interruption condition is when the vehicle has just begun changing lanes or when the edge of the vehicle body just touches the adjacent lane line of the target lane during the lane change process. Based on the current state, the vehicle identifies the social preference type of the vehicles behind it in the target lane as self-interested. The controller is configured to base its operation on the state history information h. t =(s t-2 ,s t-1 ,s t The system plans target trajectory sampling points for autonomous vehicles on the center line of their own lane or adjacent lane; the system's meta-controller and controller adopt the same reward strategy, which includes reward calculation based on the social preferences of surrounding vehicles for target decision-making.

[0006] In one embodiment of the above technical solution, the social preference type identification mechanism for surrounding vehicles is as follows: if it is a vehicle behind the target lane, social preference identification is triggered by key points, namely: just entering the autonomous vehicle's perception range, when the autonomous vehicle just begins to change lanes, and when the edge of the autonomous vehicle just touches the adjacent lane line of the target lane; if it is a vehicle in front of the autonomous vehicle's lane, social preference identification is performed in real time; if it is a vehicle in front of the adjacent lane that intends to cut into the autonomous vehicle's lane within the autonomous vehicle's perception range, social preference identification is triggered after the intention is determined.

[0007] In one embodiment of the above technical solution, the reward calculation for target decision-making based on the social preferences of surrounding vehicles is as follows: R pre = R pre_t + R pre_cutin + R pre_front Wherein: if a vehicle in the adjacent lane enters the autonomous vehicle's perception range, the social preference C identified for the vehicle in the adjacent lane is determined based on the current state. pre_t_rangeFor a self-interested model, if the objective decision of the meta-controller is to maintain lane movement, then the reward R for this objective decision is... pre_t If r1 is used, otherwise 0; if at the moment the autonomous vehicle begins to change lanes, the social preference C identified by the vehicle behind it in the target lane within its perception range is based on the current state at that moment. pre_t_change For a self-interested system, if the target decision of the meta-controller is to return to the original lane, then the reward R for this target decision is... pre_t If r2 is used, otherwise 0; if the autonomous vehicle's edge just touches the adjacent lane line of the target lane, the social preference C identified by the vehicle behind the target lane within its perception range is determined based on the current state. pre_t _cross For a self-interested system, if the target decision of the meta-controller is to return to the original lane, then the reward R for this target decision is... pre_t If the social preference C is 3, otherwise 0; pre_cutin For a self-interested model, the objective decision of the meta-controller is to maintain lane movement, and the path sampling step size in the longitudinal direction is the shortest sampling step size in the longitudinal direction. Therefore, the reward R for this objective decision is... pre_cutin If the social preference C is r4, otherwise it is 0; pre_front For a self-interested scenario, the objective decision of the meta-controller is to maintain lane keeping, and the controller instructs the autonomous vehicle to choose the fastest speed and the longest longitudinal sampling step size. The reward R for this objective decision is... pre_front If the value is -r5, otherwise it is 0; r1, r2, r3, r4, and r5 are set reward values ​​greater than 0.

[0008] In one embodiment of the above technical solution, a first social preference recognition model is used to identify the social preferences of vehicles following in adjacent lanes and vehicles preceding in the same lane that have just entered the perception range of the autonomous vehicle.

[0009] The first social preference recognition model is constructed using SVM (Support Vector Machine). The training data consists of driving data from the NGSIM dataset that maintains lane driving and where vehicles in adjacent lanes do not change lanes, after social preference classification using a clustering algorithm. The extracted features used for classification include: mean longitudinal speed of vehicles, maximum longitudinal speed of vehicles, standard deviation of longitudinal speed of vehicles, mean longitudinal acceleration of vehicles, maximum longitudinal acceleration of vehicles, standard deviation of longitudinal speed of vehicles, mean distance between vehicles and the vehicle in front, minimum distance between vehicles and the vehicle in front, and standard deviation of distance between vehicles and the vehicle in front.

[0010] In one embodiment of the above technical solution, a second social preference recognition model is used to identify the social preferences of vehicles behind the target lane within the autonomous vehicle's perception range when the autonomous vehicle has just begun changing lanes, and the social preferences of vehicles behind the target lane within the autonomous vehicle's perception range when the edge of the autonomous vehicle's body has just touched the adjacent lane line of the target lane. The second social preference recognition model is constructed using SVM, and the training data consists of data extracted from the NGSIM dataset that shows successful lane changes and unsuccessful lane changes returning to the original lane, after social preference classification. The former extracts the interaction data with vehicles behind during the process from the start of the lane change to the adjacent lane line, and the latter extracts the data from the process from the start of the lane change to the return point after the lane change. Interaction data with vehicles behind; features used for classification by the second social preference recognition model include: mean longitudinal speed of vehicles behind in the target lane, maximum longitudinal speed of vehicles behind in the target lane, standard deviation of longitudinal speed of vehicles behind in the target lane, mean longitudinal acceleration of vehicles behind in the target lane, maximum longitudinal acceleration of vehicles behind in the target lane, standard deviation of longitudinal acceleration of vehicles behind in the target lane, mean longitudinal abruptness of vehicles behind in the target lane, mean maximum longitudinal abruptness of vehicles behind in the target lane, standard deviation of longitudinal abruptness of vehicles behind in the target lane, difference between the longitudinal average speed of vehicles changing lanes and vehicles behind in the target lane, standard deviation of the difference between the longitudinal speed of vehicles changing lanes and vehicles behind in the target lane, difference between the longitudinal average acceleration of vehicles changing lanes and vehicles behind in the target lane, and standard deviation of the difference between the longitudinal acceleration of vehicles changing lanes and vehicles behind in the target lane.

[0011] In one embodiment of the above technical solution, a third social preference model is used to identify the social preferences of vehicles in the adjacent lane that intend to merge into the lane where the autonomous vehicle is located within the perception range of the autonomous vehicle. The third social preference model is constructed using SVM, and the training data is the data after social preference classification of the interaction data from the lane-changing point to the adjacent lane line in the NGSIM clustering dataset. The features extracted by the third social preference model include: mean lateral speed of lane-changing vehicles, maximum lateral speed of lane-changing vehicles, standard deviation of lateral speed of lane-changing vehicles, mean longitudinal speed of lane-changing vehicles, and maximum longitudinal speed of lane-changing vehicles. Speed, standard deviation of longitudinal speed of lane-changing vehicles, mean of lateral acceleration of lane-changing vehicles, maximum lateral acceleration of lane-changing vehicles, standard deviation of lateral acceleration of lane-changing vehicles, mean of lateral jerk of lane-changing vehicles, mean of maximum lateral jerk of lane-changing vehicles, standard deviation of lateral jerk of lane-changing vehicles, average longitudinal distance between lane-changing vehicles and vehicles following in the target lane, minimum longitudinal distance between lane-changing vehicles and vehicles following in the target lane, standard deviation of longitudinal distance between lane-changing vehicles and vehicles following in the target lane, average longitudinal distance between lane-changing vehicles and vehicles in front of the target lane, minimum longitudinal distance between lane-changing vehicles and vehicles in front of the target lane, standard deviation of longitudinal distance between lane-changing vehicles and vehicles in front of the target lane.

[0012] In one embodiment of the above technical solution, a lane change intention recognition model is used to determine whether there is a vehicle in front of the adjacent lane that intends to cut into the lane where the autonomous vehicle is located. The lane change intention recognition model is constructed using LSTM, and the training data is offline driving data of lanes 2-4 in the U-101 section of the NGSIM dataset. Data of vehicles that change lanes once and vehicles that do not change lanes but only go straight are selected, and the data of vehicles that change lanes once are marked as left / right lane change. The features extracted by the lane change intention recognition model include: (1) longitudinal speed of the lane-changing vehicle; (2) longitudinal acceleration of the lane-changing vehicle; (3) lateral speed of the lane-changing vehicle; (4) lateral acceleration of the lane-changing vehicle; (5) TTC between the lane-changing vehicle and the vehicle in front in the same lane; (6) TTC between the lane-changing vehicle and the vehicle behind in the target lane; (7) judgment mark of left / right lane change and keeping straight.

[0013] In one embodiment of the above technical solution, the current driving status of the autonomous vehicle includes: longitudinal speed, lateral speed, yaw angle, and lane markings of the autonomous vehicle.

[0014] In one embodiment of the above technical solution, the current driving status of the K nearest vehicles within the perception range of the autonomous vehicle includes: relative longitudinal distance, relative lateral distance, relative lateral speed, relative longitudinal speed, longitudinal acceleration, lateral acceleration, and the lane markings of each of the K vehicles.

[0015] Secondly, this case proposes a computer-readable storage medium, characterized in that it stores a computer program capable of being loaded by a processor and executing any of the methods described above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) For the social preference recognition of surrounding vehicles, targeted datasets for different working conditions are used for learning, which improves the effect and accuracy of data clustering. Different recognition algorithms are used, which can reduce the computational load of the behavior decision algorithm while ensuring recognition efficiency. (2) In behavior decision-making, rewards and punishments based on social preferences are designed to be applicable to different scenario decisions, thereby improving the safety of autonomous driving. (3) In the process of changing lanes, a midway return to the original lane is designed to better conform to the actual traffic operation conditions, further improving the practicality and safety of autonomous driving. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the social preference clustering process in the embodiment.

[0019] Figure 2 This is a schematic diagram illustrating the distance to the vehicle behind the target lane in the embodiment.

[0020] Figure 3 This is a schematic diagram illustrating the process of constructing an autonomous driving lane-changing decision model in this embodiment.

[0021] Figure 4 This is a schematic diagram of the meta controller structure in the embodiment.

[0022] Figure 5 This is a schematic diagram of the controller structure in the embodiment. Detailed Implementation

[0023] The terminology is explained below.

[0024] Social preferences: Based on the cluster analysis of longitudinal driving behavior of overtaken vehicles in a large amount of real overtaking data, different social preferences are identified by filtering and clustering the large amount of real overtaking data. The types of social preferences can be set; this application sets three types: altruistic, reciprocal, and self-interested.

[0025] Behavioral Decision-Making: Autonomous vehicle behavioral decision-making technology refers to the process of using sensor-collected external information, combined with map data and vehicle status within the autonomous driving system, and processed by algorithms to make decisions and plans regarding the vehicle's speed, steering, lane changing, obstacle avoidance, and other behaviors. In other words, the autonomous driving behavioral decision-making system rationally decides the vehicle's current behavior based on information output from the perception layer, choosing when to change lanes, when to maintain the lane, or when to brake, etc., and determines the constraints for trajectory planning based on different behaviors. This guides the trajectory planning module to plan appropriate paths, speeds, and other information, which is then sent to the control layer.

[0026] The meanings of the parameters are explained in Table 1.

[0027] Table 1

[0028]

[0029]

[0030]

[0031] Research on safe and effective autonomous behavior decision-making for single-vehicle intelligent vehicles is of great significance in mixed traffic environments involving both autonomous and human-driven vehicles. However, human drivers in interactive driving scenarios are influenced by multiple factors, including the driving environment and emotions, and their social value orientations often exhibit different social preferences (such as self-interest, altruism, and reciprocity), which are also time-varying. This means that the state of surrounding vehicle drivers is uncertain, manifested in the sudden time-varying nature of vehicle speed, acceleration, and front wheel steering angle in interactive driving scenarios, making autonomous driving behavior decision-making difficult.

[0032] While driving style is related to a driver's internal psychological state and personality, accurately assessing the social preferences of surrounding vehicles can help design practical autonomous driving behavior decisions. However, to identify the social preferences of surrounding vehicles, the first step is to select appropriate dynamic scene interaction data for training and classifying clustering models; the second step is to formulate effective autonomous driving behavior decisions based on the social preferences of surrounding vehicles, which is particularly important for improving the safety and efficiency of behavior decision-making algorithms.

[0033] Based on this, the present invention selects different operating condition data for different operating scenarios. It not only includes the feature parameters of the vehicle behind the target lane during the successful lane change, but also the motion parameters of the vehicle during the straight-moving process and the motion parameters of the autonomous vehicle during the lane change process but failing and returning. This is to obtain data that can fully describe the social preference characteristics of the surrounding vehicles. Different recognition mechanisms are adopted for different operating conditions, thereby constructing an autonomous driving behavior decision-making method suitable for multi-lane dynamic mixed traffic environments. This method will help autonomous vehicles discover the optimal behavior decision-making strategy in dynamic environments and has profound research significance and application prospects for improving the decision-making and planning capabilities of intelligent vehicles in multi-lane dynamic complex environments such as urban expressways.

[0034] The following description, in conjunction with the accompanying drawings, clearly and completely describes how the technical solution of this case is implemented. Obviously, the described embodiments are only a part of the embodiments of this case, and not all of them. Based on the embodiments in this case, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0035] S1. Establish a social preference recognition model for surrounding vehicles.

[0036] According to the definition in social psychology, social preference refers to the manifestation of an individual's innate social nature at the preference level. In mixed traffic scenarios, this solution uses social preference to distinguish different driving styles of drivers in the longitudinal driving mode: self-interested, reciprocal, and altruistic. Specifically, surrounding vehicles are categorized into three types: vehicles behind the target lane, vehicles in front of the current lane, and vehicles in front of the adjacent lane. The recognition mechanisms for their respective social preferences are set up (see [link to relevant documentation]). Figure 1 ).

[0037] 1.1 Establishment of a Social Preference Recognition Model for Vehicles Behind the Target Lane

[0038] 1.1.1 Selection of Training Dataset for Social Preference Clustering of Vehicles Behind the Target Lane

[0039] Existing technologies for identifying social preferences of surrounding vehicles use datasets that focus on vehicles that have successfully changed lanes, specifically the vehicle parameters of those vehicles in front, behind, to the left, and to the right during the lane change process. However, they lack data on the vehicle parameters of surrounding vehicles during the lane change return process, which does not reflect actual vehicle driving conditions. Therefore, this solution clusters the social preferences of vehicles behind the target lane within the lane, dividing the process into two scenarios.

[0040] Scenario 1: The straight-going behavior of the vehicle behind the target lane is not affected by the lane-changing behavior of the vehicle in the adjacent lane. In the subsequent behavior decision algorithm, when the vehicle behind the target lane enters the perception range of the autonomous vehicle, the social preference of the vehicle behind the target lane maintaining its lane and the vehicle in the adjacent lane not changing lanes is identified.

[0041] Scenario 2: During an autonomous vehicle's lane change, when the vehicle behind it is affected by the autonomous vehicle's lane change behavior, the short-term dynamic changes in its social preferences correspond to the social preferences of the autonomous vehicle at two key points in the subsequent behavior decision algorithm: when it initiates the lane change and when it crosses the adjacent lane line.

[0042] In one implementation: For the offline dataset used in Scenario 1, which is the NGSIM dataset U-101 lanes 2-4, driving behavior data of small vehicles driving straight in the lane for more than 60 seconds without changing lanes, and during this period, vehicles in adjacent lanes also not changing lanes, are used as driving data for straight-ahead driving conditions unaffected by lane changes in adjacent lanes. The vehicle type is selected as a sedan. For the offline dataset used in Scenario 2, which is the NGSIM dataset U-101 lanes 2-4, the vehicle type is selected as a sedan. The data consists of two parts of the scenario dataset during the lane-changing process. The first part is the scenario where the lane-changing vehicle successfully changes lanes. For the scenario where there is only one lane-changing behavior and the lane change is successful, the process from the lane-changing vehicle to the adjacent lane line is captured, and the interaction data of vehicles behind the target lane is extracted; at the same time, the data of the lane-changing vehicle, the vehicle behind the target lane without a preceding vehicle are removed; and vehicles that stop midway are also removed. Part 2 covers scenarios where a lane-changing vehicle fails to change lanes and returns to its original lane. For these unsuccessful lane-changing scenarios in the dataset, the process from the lane-changing starting point to the lane-changing termination point (return point) is captured, and the interaction data of vehicles behind in the target lane is extracted. Simultaneously, data sets containing lane-changing vehicles, vehicles behind in the target lane without preceding vehicles, and vehicles that stopped midway are removed.

[0043] Data preprocessing was performed on the data in the two scenarios described above. The Savitzky-Golay filtering algorithm was used to smooth the Local_X and Local_Y coordinates of each data coordinate, with a window length of 21 and a polynomial order of 3. The characteristic parameters such as velocity and acceleration were obtained by differentiating Local_X and Local_Y.

[0044] After data preprocessing, relevant feature vectors are extracted from the dataset. For the dataset of vehicles traveling straight within the lane without lane-changing interference in Scenario 1, the extracted feature vectors include the mean longitudinal speed of the vehicle, the maximum longitudinal speed of the vehicle, the standard deviation of the longitudinal speed of the vehicle, the mean longitudinal acceleration of the vehicle, the maximum longitudinal acceleration of the vehicle, the standard deviation of the longitudinal speed of the vehicle, the mean distance between the vehicle and the vehicle in front, the minimum distance between the vehicle and the vehicle in front, and the standard deviation of the distance between the vehicle and the vehicle in front. For the lane-changing interaction dataset in Scenario 2, the extracted feature vectors are: mean longitudinal velocity of the vehicle behind in the target lane, maximum longitudinal velocity of the vehicle behind in the target lane, standard deviation of longitudinal velocity of the vehicle behind in the target lane, mean longitudinal acceleration of the vehicle behind in the target lane, maximum longitudinal acceleration of the vehicle behind in the target lane, standard deviation of longitudinal acceleration of the vehicle behind in the target lane, mean longitudinal abruptness of the vehicle behind in the target lane, mean longitudinal maximum abruptness of the vehicle behind in the target lane, standard deviation of longitudinal abruptness of the vehicle behind in the target lane, difference between the average longitudinal velocity of the lane-changing vehicle and the vehicle behind in the target lane, standard deviation of the difference between the longitudinal velocity of the lane-changing vehicle and the vehicle behind in the target lane, difference between the average longitudinal acceleration of the lane-changing vehicle and the vehicle behind in the target lane, and standard deviation of the difference between the longitudinal acceleration of the lane-changing vehicle and the vehicle behind in the target lane.

[0045] Next, the social preference recognition training for vehicles behind the target lane is performed. After normalizing the data features extracted from the dataset, this scheme first uses a clustering algorithm to classify the dataset into three social preference types: self-interested, reciprocal, and altruistic. Then, the dataset and its clustering results are used to train the classification model built using SVM (Support Vector Machine).

[0046] In one implementation, the clustering algorithm combines Principal Component Factor Analysis (PCA) with k-means. The classification data obtained from Scenario 1 and Scenario 2 are used to train a first social preference recognition model and a second social preference recognition model, respectively. Both models are built using SVM, and are stored after training.

[0047] During the training process, the autonomous vehicle's sensor data acquisition frequency is 0.1s. Feature vector values ​​are extracted from the 2s of data collected by the sensor. Then, based on the preset first social preference recognition model and second social preference recognition model, online recognition of the social preferences of vehicles behind the target lane is performed.

[0048] 1.2 Establishment of a social preference recognition model for vehicles ahead in this lane

[0049] 1.2.1 Training dataset for social preference clustering of vehicles ahead in this lane

[0050] The social preference clustering of vehicles ahead in this lane is applied to the straight-ahead driving condition where vehicles ahead in this lane are not disturbed by vehicles changing lanes in adjacent lanes. Correspondingly, in the subsequent behavior decision algorithm, when vehicles ahead in this lane enter the perception range of the autonomous vehicle, the social preferences of vehicles ahead in this lane are considered in real time.

[0051] The dataset, extracted feature vectors, and data preprocessing used for the straight-ahead driving scenario (where the vehicle ahead in this lane is not affected by lane-changing vehicles in adjacent lanes) are consistent with the dataset and extracted features used in the analysis of the straight-ahead driving scenario (where the vehicle behind in the target lane is not affected by lane-changing vehicles in adjacent lanes). The social preference clustering method for the vehicle ahead in this lane is consistent with the identification model used for the straight-ahead driving scenario (where the vehicle behind in the target lane is not affected by lane-changing vehicles in adjacent lanes), i.e., the first social preference identification model is adopted.

[0052] During training and operation, the autonomous vehicle employs a sliding window method for online identification of the social preferences of vehicles ahead in its lane. The sensor collects data at a frequency of 0.1 seconds, gathering longitudinal velocity, longitudinal acceleration, and following distance data of the vehicle ahead in the lane. The window length is set to 20, meaning each window consists of 20 data points (2 seconds). Each sliding window is 1 second long (0.1 seconds). Based on the feature vector values ​​selected from the offline dataset, the mean, maximum, and standard deviation of the feature parameters within each window length are calculated. Then, a first social preference identification model, trained from the training set, is used to identify the social preferences of vehicles ahead in the lane online.

[0053] 1.3 Establishment of a Social Preference Recognition Model for Vehicles in the Adjacent Lane

[0054] The establishment of the social preference recognition model for vehicles ahead in adjacent lanes mainly includes two types: one is to identify the lane-changing intention of vehicles ahead in adjacent lanes, and the other is to establish a social preference recognition model for vehicles that intend to cut into the autonomous lane based on the lane-changing intention recognition results from adjacent lanes.

[0055] 1.3.1 Lane-changing intention recognition model for vehicles ahead in adjacent lanes

[0056] Offline driving data for lanes 2-4 of road segment U-101 in the NGSIM dataset were selected, with passenger cars as the vehicle type. Data showing one lane change was chosen. Vehicles changing lanes and those traveling straight without changing lanes were identified by checking if their lane IDs changed. For vehicles with changing lane IDs, the lane change point was located in the data. The direction of the lane change (left or right) was determined based on the positive or negative values ​​of the longitudinal velocity and acceleration when passing the lane change point, and the data was labeled as either left or right lane change. The number of data points with left / right lane change and straight-ahead labels was set to be the same.

[0057] Lane change intention data features were extracted from the above dataset and standardized. The lane change intention data features include (1) longitudinal speed of the lane-changing vehicle; (2) longitudinal acceleration of the lane-changing vehicle; (3) lateral speed of the lane-changing vehicle; (4) lateral acceleration of the lane-changing vehicle; (5) TTC between the lane-changing vehicle and the vehicle in front in the same lane; (6) TTC between the lane-changing vehicle and the vehicle behind in the target lane; and (7) left / right lane change and straight-ahead driving judgment indicators.

[0058] In one implementation, an LSTM algorithm is used to establish a driving intention recognition model with a prediction time of 3 seconds. The model is evaluated using a 5-fold cross-validation method to obtain the evaluation result (validation set accuracy) of the model, and the average value is taken as the final result of the model.

[0059] The model parameters are as follows: the LSTM has 128 hidden layer neurons, using a stacked 4-layer LSTM model with a learning rate of 0.0001. The output of the first LSTM unit is used as the input of the second LSTM unit, and the output of the fourth LSTM unit at the last time step is used as the output of the stacked 4-layer LSTM model. The dimension is (128,1), and the following will be used. Next, connect the fully connected layer, and... The dimension is reduced to (3,1), and then the softmax function is used to convert the model's output into the probability of left and right lane change intentions.

[0060] The loss function for the driving intention recognition model is the cross-entropy loss function, and the optimizer is the Adam optimizer. The model corresponding to the point with the lowest loss on the validation set is taken as the final model.

[0061] During operation, the autonomous vehicle's sensors collect feature vector values ​​for 3 seconds (30 frames), and the driving intention recognition model is used to identify the lane-changing intention of the vehicle in front in the adjacent lane online.

[0062] 1.3.2 Establish a social preference recognition model for vehicles that intend to enter the autonomous vehicle lane.

[0063] For the social preference clustering of vehicles in adjacent lanes that intend to cut into the lane of autonomous vehicles, the offline dataset used is: driving data of lanes 2-4 in the U-101 segment of the NGSIM dataset, with the vehicle type selected as cars. The interaction data of the lane-changing vehicle from the starting point of the lane change to the adjacent lane line is extracted from the successful lane-changing situation.

[0064] Next, the system trains to identify vehicles that intend to enter the autonomous vehicle lane. This approach normalizes the data features extracted from the dataset, then uses a clustering algorithm to classify the dataset into three social preference types: self-interested, reciprocal, and altruistic. Finally, it uses the dataset and its clustering results to train a classification model built using SVM (Support Vector Machine).

[0065] In one implementation, the feature vectors extracted from the dataset are normalized. Principal Component Factor Analysis (PCA) is first used, followed by k-means clustering of vehicles intending to enter autonomous lanes. A third social preference model is then established using SVM. The clustered classification data is used to train the third social preference model. Features used for classification include: mean lateral speed of lane-changing vehicles, maximum lateral speed of lane-changing vehicles, standard deviation of lateral speed of lane-changing vehicles, mean longitudinal speed of lane-changing vehicles, maximum longitudinal speed of lane-changing vehicles, standard deviation of longitudinal speed of lane-changing vehicles, mean lateral acceleration of lane-changing vehicles, maximum lateral acceleration of lane-changing vehicles, standard deviation of lateral acceleration of lane-changing vehicles, mean lateral abruptness of lane-changing vehicles, mean maximum lateral abruptness of lane-changing vehicles, standard deviation of lateral abruptness of lane-changing vehicles, average longitudinal distance between lane-changing vehicles and vehicles following in the target lane, minimum longitudinal distance between lane-changing vehicles and vehicles following in the target lane, standard deviation of longitudinal distance between lane-changing vehicles and vehicles following in the target lane, average longitudinal distance between lane-changing vehicles and vehicles ahead in the target lane, minimum longitudinal distance between lane-changing vehicles and vehicles ahead in the target lane, and standard deviation of longitudinal distance between lane-changing vehicles and vehicles ahead in the target lane.

[0066] During operation, autonomous vehicles extract feature vector values ​​from 2 seconds of data collected by sensors and use a third social preference model to identify the social preferences of vehicles in the adjacent lane ahead online.

[0067] The terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0068] Step 2: Configure the multi-lane driving environment for simulation

[0069] Simulated multi-lane driving environment (see) Figure 2 This includes the number of lanes in the scene, the width of each lane, the total length of the road, the speed limit for each lane, and the number of surrounding vehicles. It sets the perception range for autonomous vehicles, filters out unreasonable traffic flow scenarios, and completes the construction of a structured road autonomous driving decision-making training scenario.

[0070] 2.1 Traffic Flow Settings

[0071] (1) Low-density traffic scenarios

[0072] Six vehicles were set up, positioned in front of, behind, to the left and right of the autonomous vehicle. Based on the dataset selected by NIGSIM, the speed limit for each lane was set to v_min = 6 m / s and v_max = 19 m / s. The speeds of surrounding vehicles were randomly set within the speed limit range, and their initial positions were randomly generated with TTC > 3.

[0073] (2) Medium-density traffic scenarios

[0074] Nine vehicles were set up, positioned in front of, behind, to the left and right of the autonomous vehicle. Based on the dataset selected by NIGSIM, the speed limit for each lane was set to v_min = 6 m / s and v_max = 19 m / s. The speeds of surrounding vehicles were randomly set within the speed limit range, and their initial positions were randomly generated with TTC > 3.

[0075] (3) High-density traffic scenarios

[0076] Twelve vehicles were positioned in front of, behind, to the left and right of the Agent vehicles. Based on the dataset selected by NIGSIM, the speed limit for each lane was set to v_min = 6 m / s and v_max = 19 m / s. The speeds of surrounding vehicles were randomly set within the speed limit range, and their initial positions were randomly generated with TTC > 3.

[0077] 2.2 Social Preference Settings for Surrounding Vehicles

[0078] In traffic flow scenarios with different densities, the surrounding vehicles of the Agent are configured as follows: the speed and acceleration attributes of non-lane-changing vehicles are set to minimum, average, and maximum speed and acceleration based on the social preference settings derived from the clustering of straight-going data sets that are not disturbed by lane-changing vehicles in adjacent lanes; the speed and acceleration attributes of lane-changing vehicles are set to minimum, average, and maximum speed and acceleration based on the social preference settings derived from the clustering of lane-changing data sets.

[0079] The social preference types of non-lane-changing and lane-changing vehicles in the surrounding area are randomly set and remain stable; if a non-lane-changing vehicle is behind the target lane, its social preference is randomly reset when the autonomous vehicle performs a lane-changing behavior.

[0080] 2.3 Filter out unreasonable traffic flow scenarios

[0081] The following traffic flow scenarios are filtered out: when the autonomous vehicle is in the leftmost lane, the left lane change is cancelled; when the autonomous vehicle is in the rightmost lane, the right lane change is cancelled; when the autonomous vehicle reaches the maximum speed limit, the acceleration action is cancelled; when the autonomous vehicle reaches the minimum speed limit, the deceleration action is cancelled; when the distance between the autonomous vehicle and the vehicle in front is less than the reference following distance, the acceleration command is cancelled.

[0082] Step 3: Implement an autonomous driving behavior decision model using hierarchical reinforcement learning.

[0083] See Figure 3 A hierarchical reinforcement learning model is used to implement the autonomous driving behavior decision-making model. The model is a two-layer deep learning network structure, which can be simply divided into an upper-layer module (meta-controller) and a lower-layer module (controller). The deep learning network structure of both the upper-layer and lower-layer modules is LSTM-CNN architecture.

[0084] 3.1 Upper-level module

[0085] The upper-layer module uses the state history information h t =(s t-2 ,s t-1 ,s t Make behavioral decisions, including keeping in the lane, changing lanes, or returning to the original lane.

[0086] The state space s of the upper-level module consists of the autonomous vehicle, the operating parameters of its neighboring vehicles, and the type of social preference. If AV represents the autonomous vehicle, MV1-MV3 represent the three nearest vehicles to the Agent, and C... pre Given the social preferences of surrounding vehicles, the state space is s={AV,{MV}. i} ,C pre}, i=1,2,3.

[0087] Driving status of autonomous vehicles AV={V lat V lon ,yaw,laneID}.

[0088] Traffic conditions of surrounding vehicles (MV) i ={L lon_i ,L lat_i ,ΔL lat_i ,ΔL lon_i , α MV_lon_i , α MV_lat_i laneID mv_i}

[0089] Social preference type C of surrounding vehicles pre ={C pre_target C pre_cut_in C pre_front}

[0090] While real-time monitoring of vehicles during lane changes can improve safety, the large amount of data generated can slow down the efficiency of decision-making algorithms. Therefore, this application addresses the social preference identification of vehicles behind the target lane by setting key point recognition methods, which significantly improves the training time and convergence speed of the H-DQN algorithm while ensuring lane change safety. In one embodiment, in the state space s, there are three key points for real-time social preference identification of vehicles behind the autonomous vehicle: C pre_target ={C pre_t_range C pre_t_change C pre_t_cross This improves the efficiency and security of behavioral decision-making algorithms.

[0091] Specifically, C pre_t_rangeInformation is collected for 2 seconds when a vehicle in the adjacent lane enters the autonomous vehicle's perception range (60m in the example). Feature vectors are extracted from this information, and the social preferences of the vehicle in the adjacent lane are identified using the aforementioned pre-set first social preference model. When identified as self-interested, C... pre_t_range The value is 1. When identified as reciprocal, C... pre_t_range The value is 0, and this type is set as the default type in this scheme. When identified as altruistic, C pre_t_range The value of C is -1. pre_t_change When an autonomous vehicle begins to change lanes, it collects information on vehicles behind it in the target lane within its perception range for 2 seconds. Feature vectors are extracted from the collected information, and social preferences are identified using the aforementioned pre-set second social preference model. pre_t _cross The value of is the same as C pre_t_range C pre_t _cross When the edge of the Agent's vehicle body just touches the adjacent lane line of the target lane, it collects information on vehicles behind the target lane within its perception range for 2 seconds. Feature vectors are extracted from the collected information, and the aforementioned pre-set second social preference model is used to identify social preferences. pre_t _cross The value of is the same as C pre_t_range .

[0092] Based on the state space, construct state history information h t =(s t-2 ,s t-1 ,s t Using an ε1 greedy strategy, select one goal from the set of subtask goal decisions. t Execution. The target decision set is goal={0,1,2,3}, where each value represents, in turn, maintaining the lane, changing lanes to the left, changing lanes to the right, and returning to the original lane. Returning to the original lane occurs when the autonomous vehicle has just begun changing lanes or when the edge of the vehicle body has just touched the adjacent lane line of the target lane. If the social preference type of the vehicles behind the target lane is identified as self-serving, the lane change is interrupted to conform to the actual traffic environment.

[0093] The meta-controller module uses an LSTM-CNN network architecture (see...). Figure 4 This will enable the transfer of state history information h at three points in time. t =(s t-2 ,s t-1 ,s tThe LSTM is used as input to output the target decision. In one implementation, each time point has 30 dimensions, the input dimension of the LSTM is set to (3, 30), and the output is set to 512*1. The output of the LSTM layer is used as the input of the CNN. The first hidden layer of the CNN is a fully connected layer, the last dimension of the output is 256*1, and the activation function is ReLU. The second hidden layer is a fully connected layer, the last dimension of the output is 64*1, and the activation function is ReLU; the last output layer is a fully connected layer (FC), which outputs the target decision (goal option), the last dimension of the output is 4*1, and the activation function is ReLU. The entire module uses Mean-Squared Loss (MSE) to calculate the loss function, uses RMSprop as the optimizer, the decay exponent rho = 0.95, the learning rate lr = 0.00025, and epsilon = 0.01.

[0094] In the fully connected layer, the reward maximization function is utilized, and the update objective of the Q function is as follows.

[0095]

[0096] In this hierarchical reinforcement learning scheme, the meta-controller is configured to construct the autonomous vehicle's state s at time t using the autonomous vehicle's current driving status, the current driving status of the K nearest vehicles within the autonomous vehicle's perception range, and the social preferences of surrounding vehicles. t K is a set value, and it is determined based on the state s. t The decision is to maintain the lane, change lanes, or return to the original lane. Returning to the original lane occurs when the autonomous vehicle has just begun changing lanes or when the edge of the vehicle body has just touched the adjacent lane line of the target lane during the lane change process, and when the social preference type of the vehicles behind in the target lane is identified as self-interested, by interrupting the lane change. That is, during training, when C... pre_t_change =1∪C pre_t _cross When =1, it is equivalent to the training sequence being interrupted, goal t From this point on, we must start selecting goals again. t+1 Combined with the subsequent social preference reward function, when C pre_t_change =1∪C pre_t _cross When C = 1, it prompts autonomous vehicles to learn the goal decision of returning to their destination during a lane change. In application, when C... pre_t_change =1∪C pre_t _cross When =1, the current goal t The current time and position are immediately terminated, and the process restarts by selecting the next goal. t+1 This refers to a temporary interruption in the execution of a goal decision while traveling from the current lane to the center line of the adjacent lane, and the execution of the next goal from that current position. t+1The choice, combined with the subsequent social preference reward function, when C... pre_t_change =1∪C pre_t _cross When the value is 1, it prompts autonomous vehicles to learn the goal decision of returning to their destination while changing lanes.

[0097] (3.2) Lower-level control module (controller)

[0098] The lower-level control module receives subtask goals from the upper-level module. t and status history information (h t =(s t-2 ,s t-1 ,s t Then, select an action to execute, that is, based on the current goal. t Based on state history information, select the action that maximizes the reward.

[0099] In one implementation, the specific network architecture of the lower-level control module is an LSTM-CNN structure (see [link]). Figure 5 ).

[0100] The LSTM layer takes the states at three time points as input, i.e., h t Each time point has 30 dimensions, i.e., (3, 30), and the last dimension of the output is 512*1. The goal space is not processed by LSTM, but only by a fully connected layer. The processing method is: a fully connected layer, with the goal space as the input and the last dimension of the output being 512*1, and the activation function is ReLU.

[0101] The first hidden layer of the CNN is a fully connected layer. It takes the data from the goal space (after processing by the spatial fully connected layer) and the data from the state history space (after processing by the LSTM layer) as input to the CNN. The last dimension of the output is 256*1, and the activation function is ReLU. The second hidden layer is also a fully connected layer, with the last dimension of the output being 64*1 and the activation function being ReLU. The output layer is a fully connected layer that outputs the action space, with the last dimension being 12*1 and the activation function being Linear.

[0102] The lower-level control module uses Mean-Squared Loss (MSE) to calculate the loss function, and RMSprop is used as the optimizer with a decay exponent rho = 0.95, a learning rate lr = 0.00025, and epsilon = 0.01.

[0103] The lower-level control module uses the DQN algorithm to maximize the reward function, and the update objective of the Q function is as follows.

[0104]

[0105] The action in the lower-level control module is set to the target trajectory point of the agent in the current lane or adjacent lane, including the target path point and target speed:

[0106] The action space is defined by sampling points of the target trajectory on the center line of the agent in the current lane or adjacent lane, mainly including the target path point space C. trajectory And target vehicle speed space V agent_target Two categories.

[0107] action={ C trajectory V agent_target}

[0108] C trajectory This includes the set of path points in both the horizontal and vertical directions of the lane centerlines of this lane and the two adjacent lanes, with a set space size of 9. trajectory ={ C agent_lat C agent_lon}, Path sampling point C in the horizontal direction agent_lat ={ C d_L ,0, C d_R The sampling points are located at the center lines of the adjacent lanes on the left and right, and the center line of the current lane, respectively, with a combined space of 3. The lane width is 3.75m. The longitudinal path sampling point C... agent_lon ={ L s , w a L s , w b L s The location is situated along the centerline of the current lane or the centerline of the adjacent lane in the direction of vehicle travel, with a set space of 3. The trajectory points in this scheme are path points that include speed; that is, the trajectory point information includes both path points and speed.

[0109] V agent_target For the vehicle's speed space, the aggregate space is 4. V agent_target ={ 0, δ0V agent_target_0 V agent_target_0 ,δ1V agent_target_0}

[0110] Constraints on actions in the lower-level control module: When the autonomous vehicle aims to remain within the lane (goal = 0), the action space is {{ 0}{ L}} s , w a L s , w b L s}, V agent_target The space size is 12; when the autonomous vehicle changes lanes to the left, i.e., goal=1, the action space is ={{ C d_L}{ Ls , w a L s , w b L s}, V agent_target The space size is 12; when the autonomous vehicle changes lanes to the right, i.e., goal=2, the action space is ={{ C d_R}{ L s , w a L s , w b L s}, V agent_target The space size is 12; when the autonomous vehicle chooses to return to its original lane, i.e., goal=3, the action space is ={{C agent_lat_i}{ L s , w a L s , w b L s}, V agent_target The space size is 12.

[0111] (3.3) Setting the reward function

[0112] In the hierarchical behavior decision-making algorithm, the upper-level module is responsible for the overall decision, while the lower-level module is only responsible for the detailed sub-decision. Therefore, the round reward function (such as the reward for reaching the destination) is only effective when the entire loop task and the goal of the upper-level module are completed. That is, the reward functions set by the upper and lower modules are consistent, and both require the completion of the loop task and the execution of the goal decision (including temporary interruptions). At the same time, combining the process reward function (such as the rapid lane-changing reward function, etc.) can better improve the training effect of goal and action.

[0113] Autonomous vehicles consider social preferences, safety, traffic efficiency, and ride comfort during behavioral decision-making. The reward functions of both modules should include the following four parts.

[0114] (3.3.1) Setting the social preference reward function for surrounding vehicles:

[0115] R pre = R pre_t + R pre_cutin + R pre_front

[0116] The social preference reward function R of the vehicle following in the target lane pre_t The autonomous vehicle identifies the social preferences of vehicles behind it in both lanes within a 60m radius, based on the current state. pre_t_rangeAfter `goal=1`, if the autonomous vehicle chooses to remain within its lane (i.e., when `goal=0`), a reward of `r1` is given. If the autonomous vehicle has already executed a lane-changing strategy, the social preference reward function of surrounding vehicles only considers the social preference reward function of the vehicle behind the target lane. The influence of the vehicle in front in the adjacent lane and the vehicle in front in the current lane on the lane change is learned autonomously by the autonomous vehicle based on other parameters in the state space. That is, the lane-changing return strategy is only affected by the social preference reward function of the vehicle behind the target lane. When the autonomous vehicle is about to start a lane change, if it identifies the social preference C of the vehicles behind the target lane within its perception range based on the current state... pre_t_change =1, at this point, if the autonomous vehicle executes the strategy of returning to its original lane, i.e., goal=3, a reward of r2 is given. When the edge of the autonomous vehicle's body just touches the adjacent lane line of the target lane, if C is identified based on the social preference of the vehicles behind the target lane at the current moment... pre_t_cross =1. If the autonomous vehicle executes the strategy of returning to the original lane, i.e., goal=3, a reward of r3 is given.

[0117]

[0118] The reward or penalty values ​​here and in the examples below can be changed.

[0119] The social preference reward function R of the vehicle in front in the adjacent lane pre_cutin In the decision-making process of an autonomous vehicle to continue driving within its lane, the social preference reward function of surrounding vehicles only considers the social preference reward functions of the vehicle in front in its own lane and the vehicle in an adjacent lane that intends to merge into its lane. Under the condition of maintaining lane keeping, for a vehicle in an adjacent lane within a 60m range that intends to merge into its lane, its social preference reward function C... pre_cutin When =1, based on the current state, goal=0 and C MV_lon =L s If so, a reward of r4 will be given.

[0120]

[0121] The social preference reward function R for the vehicle ahead in this lane pre_front In the scenario where an autonomous vehicle remains within its lane, the social preference reward function of surrounding vehicles only considers the social preference reward functions of the vehicle in front in its own lane and vehicles in adjacent lanes that intend to merge into its lane. In this lane-keeping scenario, if the autonomous vehicle has a self-serving social preference towards the vehicle in front in its lane, it will be penalized if it chooses the fastest speed and the longest path to remain within its lane; that is, the social preference reward function C... pre_front When =1, goal=0 and A penalty of -r5 will be imposed. That is: .

[0122] (3.3.1) Collision penalty function Rcollision:

[0123] When an autonomous vehicle collides with surrounding vehicles, a penalty is imposed, R collision =-500.

[0124] (3.3.2) Operating Condition Reward Function

[0125] R cond = R follow_V +R follow_s +R change_t +R complete

[0126] Following vehicle speed penalty function R follow_v When the autonomous vehicle's behavioral decision is to remain within the lane (goal=0), it maintains the cruising speed V. AV_target_0 Driving will be rewarded. That is: R follow_v =10, if goal=0, and action ={{i}{i},V MV_target_0}

[0127] Dangerous distance penalty function R follow_s When an autonomous vehicle makes a decision to continue driving within its lane, if the total time to traffic (TTC) during the lane-keeping operation is less than 3, a penalty is imposed, i.e., R. follow_s =-500, ifTTC < 3 While avoiding collisions, autonomous vehicles can learn to make lane-changing decisions when necessary.

[0128] Quick lane change reward R change_t When an autonomous vehicle makes a lane-changing decision (goal=1 or goal=2), and the target lane is a longer journey point C... MV_lon = w b L s At that time, if the vehicle speed is selected as high speed δ1V MV_target_0 A reward will be given.

[0129] Right now:

[0130] Reward function R for reaching the destination complete A reward is given when the agent reaches its destination. That is: R complete =1000, if ego_dis < 5 meters.

[0131] The reward / penalty function R for comfort comfort A penalty will be imposed when the comfort of autonomous vehicles decreases.

[0132]

[0133] in, and These are lateral jerk and longitudinal jerk; α and β are the corresponding weights for lateral and longitudinal comfort.

[0134] Therefore, the overall reward function R = R pre + R collision + R cond + R comfort .

[0135] The action represents the target path point and target speed for the autonomous vehicle's behavior decision-making. The local path planned by the agent from its current position and speed to the target trajectory point is fitted and connected by a fifth-order polynomial. The acceleration and steering wheel angle outputs during subsequent trajectory tracking are handled by a PID algorithm, forming a complete autonomous driving function for the agent and completing the algorithm training. The lane-changing time t required by the fifth-order polynomial is defined relative to δ0V. MV_target_0 V MV_target_0 ,δ1V MV_target_0 Three lane-changing speeds are used, with a unique time interval t1-t3. After the local dynamic programming path is determined, trajectory tracking is completed using a common PID algorithm, achieving full autonomous driving functionality. Once the autonomous driving system has acquired full autonomous driving capabilities, it then begins training to obtain optimal behavioral decisions by maximizing the reward function.

[0136] The pseudocode implementation of the above-mentioned hierarchical reinforcement learning for autonomous driving behavior decision-making is shown in Table 2 below.

[0137] Table 2

[0138]

[0139]

[0140] As can be seen from the above implementation methods, the autonomous driving behavior decision-making system that considers the social preferences of surrounding vehicles proposed in this scheme firstly improves the effect and accuracy of data clustering by selectively choosing datasets based on different operating conditions for the social preferences of surrounding vehicles. Secondly, it employs different recognition algorithms for the social preferences of surrounding vehicles under different operating conditions, ensuring recognition efficiency while reducing the computational load of the behavior decision-making algorithm. Finally, in the hierarchical reinforcement learning, based on the social preference attributes of surrounding vehicles, the autonomous vehicle can make behavioral decisions such as lane changing, returning to the original lane midway through a lane change, stopping, and maintaining the lane, thereby improving the safety of autonomous vehicle operation. Furthermore, by incorporating reward settings for different operating conditions based on the social preferences of surrounding vehicles, the autonomous vehicle can cope with the uncertainties of mixed scenarios, improving the effectiveness and accuracy of behavioral decisions and better meeting the needs of actual operating conditions.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the system disclosed herein can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the purposes of this disclosure, software implementation is often a better implementation method.

[0142] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. An autonomous driving behavior decision-making system that considers the social preferences of surrounding vehicles, characterized in that: The system employs hierarchical reinforcement learning to achieve autonomous driving behavior decisions, including a meta-controller and a controller; The meta-controller is configured to, based on the autonomous vehicle state history information h t =(s t-2 ,s t-1 ,s t The system makes a decision to maintain lane control, change lanes, or return to the original lane; where: state s t This includes the autonomous vehicle's driving status at time t, the driving status of the K nearest vehicles within the autonomous vehicle's perception range at time t, and the social preference type of surrounding vehicles, where K is a set value. Social preference types include self-interested, reciprocal, and altruistic. Returning to the original lane is achieved by the autonomous vehicle interrupting a lane change. The interruption condition is when the vehicle has just started changing lanes or when the edge of the vehicle body just touches the adjacent lane line of the target lane during the lane change. Based on the current state, the autonomous vehicle identifies the social preference type of the vehicle behind it in the target lane as self-interested. The identification mechanism for the social preference type of surrounding vehicles varies depending on the operating conditions: if it is a vehicle behind the target lane, social preference identification is triggered by key points, namely: just entering the autonomous vehicle's perception range, when the autonomous vehicle has just started changing lanes, and when the edge of the autonomous vehicle body just touches the adjacent lane line of the target lane; if it is a vehicle in front of the autonomous vehicle in its lane, social preference identification is performed in real time; if it is a vehicle in front of the autonomous vehicle in its adjacent lane within the autonomous vehicle's perception range intending to cut into the autonomous vehicle's lane, social preference identification is triggered after determining the intention. The controller is configured to base its operation on state history information h. t =(s t-2 ,s t-1 ,s t ), to plan the target trajectory sampling points for autonomous vehicles on the center line of their own lane or adjacent lane; The system's meta-controller and controller employ the same reward strategy, which includes reward calculation for target decision-making based on the social preferences of surrounding vehicles.

2. The system according to claim 1, characterized in that: The reward calculation for goal-oriented decision-making based on the social preferences of surrounding vehicles is as follows: R pre = R pre_t + R pre_cutin + R pre_front in: If a vehicle in the adjacent lane enters the autonomous vehicle's perception range, the social preference C identified for the vehicle in the adjacent lane is determined based on the current state. pre_t_range For a self-interested model, if the objective decision of the meta-controller is to maintain lane movement, then the reward R for this objective decision is... pre_t r1; If, at the moment an autonomous vehicle begins to change lanes, the social preference C identified by the vehicle behind it in the target lane within its perception range is based on the current state at that instant... pre_t_change For a self-interested system, if the target decision of the meta-controller is to return to the original lane, then the reward R for this target decision is... pre_t r2; If, at the moment the edge of the autonomous vehicle's body just touches the adjacent lane line of the target lane, the social preference C identified by the vehicle behind the target lane within its perception range is based on the current state, then... pre_t _cross For a self-interested system, if the target decision of the meta-controller is to return to the original lane, then the reward R for this target decision is... pre_t It is r3; If the social preference C is identified for the vehicle in the adjacent lane that intends to cut into this lane... pre_cutin For a self-interested model, the objective decision of the meta-controller is to maintain lane movement, and the controller's path sampling step size in the longitudinal direction is the shortest sampling step size in the longitudinal direction. Therefore, the reward R for this objective decision is... pre_cutin It is r4; If the social preferences C of the vehicles ahead in this lane are identified in real time pre_front For a self-interested scenario, the objective decision of the meta-controller is to maintain lane keeping, and the controller instructs the autonomous vehicle to choose the fastest speed and the longest longitudinal sampling step size. The reward R for this objective decision is... pre_front -r5; r1, r2, r3, r4, and r5 are set reward values ​​greater than 0.

3. The system according to claim 1, characterized in that: The first social preference recognition model is used to identify the social preferences of vehicles behind in adjacent lanes and vehicles in front in the same lane when they just enter the perception range of the autonomous vehicle. The first social preference recognition model is constructed using SVM. The training data consists of driving data from the NGSIM dataset under the condition of maintaining lane and vehicles in adjacent lanes not changing lanes, which are classified according to social preferences using a clustering algorithm. The features used for classification include: mean longitudinal speed of vehicles, maximum longitudinal speed of vehicles, standard deviation of longitudinal speed of vehicles, mean longitudinal acceleration of vehicles, maximum longitudinal acceleration of vehicles, standard deviation of longitudinal speed of vehicles, mean distance between vehicles and the vehicle in front, minimum distance between vehicles and the vehicle in front, and standard deviation of distance between vehicles and the vehicle in front.

4. The system according to claim 1, characterized in that: The second social preference recognition model is used to identify the social preferences of vehicles behind the target lane within the perception range of the autonomous vehicle when it just begins to change lanes, and the social preferences of vehicles behind the target lane within the perception range when the edge of the autonomous vehicle just touches the adjacent lane line of the target lane. The second social preference recognition model is constructed using SVM. The training data consists of data extracted from the NGSIM dataset that shows successful lane changes and unsuccessful lane changes returning to the original lane, and then classified according to social preferences using a clustering algorithm. The former extracts the interaction data with vehicles behind during the process from the start of the lane change to the adjacent lane line, while the latter extracts the interaction data with vehicles behind during the process from the start of the lane change to the lane change termination point and return point. The features used for classification in the second social preference identification model include: mean longitudinal speed of the vehicle following in the target lane, maximum longitudinal speed of the vehicle following in the target lane, standard deviation of longitudinal speed of the vehicle following in the target lane, mean longitudinal acceleration of the vehicle following in the target lane, maximum longitudinal acceleration of the vehicle following in the target lane, standard deviation of longitudinal acceleration of the vehicle following in the target lane, mean longitudinal abruptness of the vehicle following in the target lane, mean maximum longitudinal abruptness of the vehicle following in the target lane, standard deviation of longitudinal abruptness of the vehicle following in the target lane, difference between the average longitudinal speed of the vehicle changing lane and the vehicle following in the target lane, standard deviation of the difference between the longitudinal speed of the vehicle changing lane and the vehicle following in the target lane, difference between the average longitudinal acceleration of the vehicle changing lane and the vehicle following in the target lane, and standard deviation of the difference between the longitudinal acceleration of the vehicle changing lane and the vehicle following in the target lane.

5. The system according to claim 1, characterized in that: The third social preference model is used to identify the social preferences of vehicles in the adjacent lane that intend to cut into the lane where the autonomous vehicle is located within the perception range of the autonomous vehicle. The third social preference model is constructed using SVM. The training data consists of social preference classification of the interaction data from the lane change starting point to the adjacent lane line in the NGSIM dataset, which is obtained by using the clustering algorithm. The features used for classification in the third social preference model include: mean lateral speed of lane-changing vehicles, maximum lateral speed of lane-changing vehicles, standard deviation of lateral speed of lane-changing vehicles, mean longitudinal speed of lane-changing vehicles, maximum longitudinal speed of lane-changing vehicles, standard deviation of longitudinal speed of lane-changing vehicles, mean lateral acceleration of lane-changing vehicles, maximum lateral acceleration of lane-changing vehicles, standard deviation of lateral acceleration of lane-changing vehicles, mean lateral abruptness of lane-changing vehicles, mean maximum lateral abruptness of lane-changing vehicles, standard deviation of lateral abruptness of lane-changing vehicles, average longitudinal distance between lane-changing vehicles and vehicles following in the target lane, minimum longitudinal distance between lane-changing vehicles and vehicles following in the target lane, standard deviation of longitudinal distance between lane-changing vehicles and vehicles following in the target lane, average longitudinal distance between lane-changing vehicles and vehicles ahead in the target lane, minimum longitudinal distance between lane-changing vehicles and vehicles ahead in the target lane, and standard deviation of longitudinal distance between lane-changing vehicles and vehicles ahead in the target lane.

6. The system according to claim 1, characterized in that: A lane-change intention recognition model is used to determine whether there are vehicles in the adjacent lane that intend to cut into the lane where the autonomous vehicle is located. The lane change intention recognition model is constructed using LSTM. The training data is offline driving data of lanes 2-4 in the U-101 section of the NGSIM dataset. Data of vehicles that change lanes once and vehicles that do not change lanes and only go straight are selected, and the data of vehicles that change lanes once are marked as left / right lane change. The features extracted by the lane change intention recognition model include: (1) longitudinal speed of the lane-changing vehicle; (2) longitudinal acceleration of the lane-changing vehicle; (3) lateral speed of the lane-changing vehicle; (4) lateral acceleration of the lane-changing vehicle; (5) TTC between the lane-changing vehicle and the vehicle in front in the same lane; (6) TTC between the lane-changing vehicle and the vehicle behind in the target lane; (7) left / right lane change and straight-ahead judgment indicators.

7. The system according to claim 1, characterized in that, The current driving status of autonomous vehicles includes: longitudinal speed, lateral speed, yaw angle, and lane markings.

8. The system according to claim 1, characterized in that, The current driving status of the K nearest vehicles within the autonomous vehicle's perception range includes: relative longitudinal distance, relative lateral distance, relative lateral speed, relative longitudinal speed, longitudinal acceleration, lateral acceleration, and the lane markings of each of the K vehicles.

9. A computer-readable storage medium, characterized in that: The system contains a computer program that can be loaded by a processor and executed by the system as described in any one of claims 1 to 8.

Images