Method for optimizing behavior decision model of unmanned vehicle

Abstract

The invention discloses a method for optimizing a behavior decision model of an unmanned vehicle, and belongs to the technical field of intelligent decision of unmanned vehicles. Comprising the steps of obtaining historical data of traffic participants; establishing a traffic scene interaction relation graph including a node feature matrix and an adjacent matrix; inputting the node feature matrix and the adjacent matrix into a graph attention neural network to obtain topological features of the graph; a trajectory model is established, decision features are selected to generate a reward function, and maximum entropy inverse reinforcement is adopted for learning; and combining the learned reward function with the DQN, and making a vehicle behavior decision according to the topological characteristics of the graph. The multi-dimensional reward function designed by the invention comprehensively considers the driving efficiency, the comfort, the risk avoidance and the interaction, so that the decision of the unmanned vehicle is more comprehensive and meticulous. And a maximum entropy inverse reinforcement learning technology is adopted to directly learn a reward function from human driving data, so that the decision personification degree of the unmanned vehicle is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent decision-making of unmanned vehicles, and in particular to a method for optimizing a behavior decision-making model of an unmanned vehicle. Background Art

[0002] The behavioral decision model of driverless vehicles is the core link in realizing autonomous driving. Its goal is to simulate the decision-making process of human drivers under the premise of ensuring safety in order to adapt to complex traffic environments. Existing driverless vehicle decision models are mainly based on traditional rule engines or simple reinforcement learning methods. These methods perform well under specific conditions, but their generalization ability and adaptability are limited in changing traffic scenarios, especially in scenarios with multi-agent interactions.

[0003] In a multi-agent interactive environment, the decision-making of an unmanned vehicle is not only affected by its own state, but also significantly affected by the behavior of other traffic participants. The complexity of these interactions requires the decision-making model to be able to understand and predict the behavior of other traffic participants in order to make more reasonable and humane decisions. However, existing models often ignore this interactivity or fail to accurately capture and simulate these interactions, resulting in significant differences between the decision-making behavior of unmanned vehicles in actual traffic and that of human drivers.

[0004] In addition, existing reinforcement learning models may encounter problems such as difficulty in reward function design, low training efficiency, and weak model generalization during training. These problems limit the application potential of driverless vehicles in complex traffic scenarios. In order to solve these problems, researchers began to explore inverse reinforcement learning (IRL) technology, which learns reward functions from the demonstration behavior of human drivers in order to obtain better decision-making models.

[0005] Although IRL technology has potential in theory, it still faces many challenges in practical applications, such as high computational complexity, easy overfitting, and limited model generalization ability. In addition, existing IRL methods often have difficulty in accurately capturing and simulating the impact of interactions on decision-making when dealing with complex scenarios with multiple interacting agents. Summary of the invention

[0006] In view of the defects of the prior art, the present invention improves the performance of the unmanned vehicle behavior decision model by establishing a trajectory evaluation model and optimizing the reward function. This method not only takes into account the interaction between the unmanned vehicle and other traffic participants in the environment, but also learns the reward function from real human driving data through inverse reinforcement learning technology, thereby improving the anthropomorphism and generalization ability of the decision model.

[0007] In order to achieve the above object, the present invention proposes a method for optimizing an unmanned vehicle behavior decision model, comprising the following steps:

[0008] (1) Obtain historical data of traffic participants, including their positions, speeds, accelerations, and deflection angles over the past t time period;

[0009] (2) Establish a traffic scene interaction relationship diagram, including the node feature matrix and adjacency matrix;

[0010] The historical data of the traffic participants are encoded using a gated recurrent unit (GRU) to obtain dynamic features, and the dynamic features are input into a node feature vector to construct a node feature matrix;

[0011] Gaussian process regression (GPR) is used to represent the Gaussian velocity field to construct the adjacency matrix;

[0012] (3) Inputting the node feature matrix and the adjacency matrix into the graph attention neural network GAT to obtain the topological features of the graph;

[0013] (4) Establish a trajectory model, select decision features to generate a reward function, and use maximum entropy inverse reinforcement for learning;

[0014] (5) Combining the learned reward function with DQN to make vehicle behavior decisions based on the topological features of the graph.

[0015] Furthermore, the node feature matrix is ​​specifically:

[0016] GRU is used to encode the historical data of the traffic participants and extract the dynamic features of the traffic participants:

[0017]

[0018] Among them: GRU h stands for Gated Recurrent Unit GRU, represents the dynamic characteristics of traffic participant i at time t, represents the historical data of traffic participant i at time t;

[0019] Input this dynamic feature into the node feature vector to construct the node feature matrix;

[0020]

[0021] in: is the node feature vector of node i at time t, which contains the current speed information, location information, vehicle driving direction, pedestrian orientation and dynamic characteristics of traffic participants.

[0022] Furthermore, the adjacency matrix is ​​specifically:

[0023]

[0024] Where: E f ={ei j Each item in ,i,j∈[1,n]} represents the edge feature vector of the relationship between the ith traffic participant and the jth traffic participant;

[0025] The edge feature vector includes a Gaussian velocity field and relative velocities, relative positions and angles between the orientations of different traffic participants;

[0026] The relative velocity can be expressed as v ij =(v jx -v ix ,v jy -v iy ), the relative position is represented by the Euclidean distance The orientation angle can be expressed as θ ij =atan2(y j -y i ,x j -x i ); x i ,y i is the position of the ith traffic participant, v ix 、v iy is the speed of the ith traffic participant in the x and y directions;

[0027] Use GPR to establish the probability distribution of the velocity field of traffic participants and obtain a quantitative description of the uncertainty of the movement of traffic participants:

[0028]

[0029] Where: A is the magnification factor, x i is the position of the ith traffic participant, x j is the position of the jth surrounding traffic participant, σ is the scale constant;

[0030] Take the i-th traffic participant and its surrounding j-th traffic participant at x i The relative velocity of the position is used to describe the Gaussian velocity field between vehicles:

[0031] B t (i, j) = K(x i , x j )K(x i , x j ) -1 ΔV(x i , x j )

[0032] ΔV(x i , x j )=V i -V j

[0033] Where: ΔV(x i ,x j ) is the distance between the ith traffic participant and the jth traffic participant around him at x i Relative velocity of position, V i 、V j is the speed of the i-th and j-th traffic participants.

[0034] Furthermore, the trajectory model is specifically:

[0035] The trajectory model of the participating vehicles is generated using a polynomial trajectory sampler, which is represented by two polynomial functions with respect to the x and y coordinates, respectively:

[0036]

[0037] Where: τ represents time, {a0,...,a4} and {b0,...,b5} are the coefficients of each term of the polynomial; x represents the position function of the traffic participant in the longitudinal direction of the road; y represents the position function of the traffic participant in the transverse direction of the road;

[0038] The initial state, target state and time T required to reach the target of the participating vehicles, and the boundary conditions of the polynomial functions on the vertical and horizontal axes can be expressed as:

[0039]

[0040] Where: (x s ,v xs ,a xs ,y s ,v ys ,a ys ) is the initial state, i.e. the state of τ = 0, where each sub-item is the position and the velocity and acceleration in the horizontal and vertical directions; (v xe ,a xe ,y e ,v ye ,a ye ) is the target state, i.e., the state of τ = T, in which there is no longitudinal position; is the first-order derivative of x and y, representing the velocity function; is the second-order derivative of x and y, representing the acceleration function;

[0041] The coefficients of the polynomial function are determined by solving the boundary equations to obtain the trajectory model.

[0042] Furthermore, the selected decision features include: driving efficiency, comfort, risk aversion and interaction; according to the trajectory model, the decision features on each trajectory are obtained;

[0043] The driving efficiency is characterized by the driving speed of the vehicle:

[0044]

[0045] The comfort is determined by the longitudinal acceleration a x , lateral acceleration a y and longitudinal jerk j x Characterization:

[0046]

[0047] The risk avoidance is characterized by the positional relationship between the vehicle and the preceding and following vehicles:

[0048]

[0049] Where: x f (t) is the longitudinal position of the nearest preceding vehicle, x ego (t) is the longitudinal position of the vehicle, v ego (t) is the current longitudinal speed of the vehicle; x r (t) is the longitudinal position of the nearest rear vehicle, v r (t) is the longitudinal speed of the following vehicle;

[0050] The interaction is characterized by the sum of the absolute values ​​of the predicted deceleration of the vehicle affected by the ego vehicle behavior:

[0051]

[0052] Among them: a i (t) is the acceleration of the vehicle affected by the ego vehicle; i represents the surrounding vehicles affected by the ego vehicle’s behavior;

[0053] The above decision features are calculated at each time step to obtain all the features of a trajectory and normalize them between 0 and 1 to offset the effects of different units and scales.

[0054] Furthermore, the reward function is specifically:

[0055] For s t , define the reward function r(s t )as follows:

[0056] r(s t )=θ T f(s t )

[0057] Where: θ=[θ1,θ2,...,θ K ] is the K-dimensional weight vector, f(s t )=[f1(s t ),f2(s t ),...,f K (s t )] is the feature vector of the decision feature;

[0058] For trajectory ζ, the corresponding reward function R(ζ) is expressed as:

[0059]

[0060] Where: f ζ are all the features of trajectory ζ;

[0061] The Boltzmann model is used to assign corresponding probabilities to all trajectories. The probability of a trajectory is proportional to the exponential of the reward of the trajectory, and we get:

[0062]

[0063] Where: P(ζ|θ) is the probability of trajectory ζ given a reward parameter θ; Z(θ) is the distribution function. Using a finite number of trajectories, the distribution function is approximated to obtain the following formula:

[0064]

[0065] in: To generate a trajectory, its initial state is consistent with ζ, is the feature vector of the trajectory, and M is the number of generated trajectories.

[0066] Furthermore, the learning using maximum entropy inverse reinforcement is specifically as follows:

[0067] Use maximum entropy to learn weights θ;

[0068]

[0069] Based on the maximum entropy, the reward function is updated using the gradient ascent method until the loss converges and the weight θ is obtained;

[0070]

[0071] in: is a set of real human driving trajectories; f ζ is the feature vector of the human driver’s driving trajectory, is a generated trajectory whose initial state is consistent with ζ. is the eigenvector corresponding to the trajectory.

[0072] Furthermore, the step (5) is specifically as follows:

[0073] (5.1) replacing the reward function in the original DQN with the learned reward function to obtain an enhanced decision algorithm;

[0074] (5.2) The topological features of the graph are input into the above decision algorithm to select the action with the greatest value in the action space as the optimal action.

[0075] Furthermore, the action space is specifically:

[0076] The information in the action space is a combination of driving behaviors, as shown below:

[0077] a=[CL,GS,CR]

[0078] Among them: CL, GS, and CR represent lane change to the left lane, straight driving, and lane change to the right lane, respectively. These actions are based on semantic-level commands and are used to guide the movement of the vehicle.

[0079] Beneficial effects of the present invention:

[0080] 1. The present invention can predict and take into account the reactions of other traffic participants by establishing a trajectory evaluation model, thereby enhancing the interactivity and safety of decision-making.

[0081] 2. The multi-dimensional reward function designed by the present invention comprehensively considers driving efficiency, comfort, risk aversion and interaction, making the decision of the driverless vehicle more comprehensive and detailed. By using the learning model, commonalities are extracted from multiple trajectories, which improves the generalization ability and adaptability of the model. While ensuring safety, it can also balance driving efficiency and achieve a more reasonable driving strategy.

[0082] 3. The present invention adopts maximum entropy inverse reinforcement learning technology to directly learn the reward function from human driving data, effectively improving the degree of anthropomorphism of decision-making of unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A flowchart of a method for optimizing an unmanned vehicle behavior decision model according to an embodiment of the present invention.

[0084] Figure 2 A schematic diagram of the reward value (training) obtained by the driverless vehicle according to an embodiment of the present invention.

[0085] Figure 3 Schematic diagram of loss value (training) of the driving behavior decision algorithm according to an embodiment of the present invention.

[0086] Figure 4Schematic diagram of the autonomous lane-changing driving trajectory (left lane-changing scenario) according to an embodiment of the present invention.

[0087] Figure 5 Schematic diagram of trajectory change during lane changing (left lane changing scenario) according to an embodiment of the present invention.

[0088] Figure 6 It is a schematic diagram of speed change during the lane changing process (left lane changing scenario) according to an embodiment of the present invention.

[0089] Figure 7 The embodiment of the present invention is a schematic diagram of an autonomous lane-changing driving trajectory (left lane-changing withdrawal scenario).

[0090] Figure 8 Schematic diagram of trajectory change during lane changing (left lane change withdrawal scenario) according to an embodiment of the present invention.

[0091] Fig. 9 Schematic diagram of trajectory change during lane changing (left lane change withdrawal scenario) according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0093] like Figure 1 As shown, an embodiment of the present invention provides a method for optimizing the behavior decision model of an unmanned vehicle, especially for the driving behavior decision problem in a multi-agent interactive environment. The method learns the reward function from real human driving data through trajectory generation and evaluation, combined with maximum entropy inverse reinforcement learning technology, to improve the anthropomorphic degree of decision-making of unmanned vehicles. It includes the following steps:

[0094] S101. Obtain historical data of all traffic participants, including positions, speeds, accelerations, and deflection angles of the traffic participants.

[0095] S102: Establish a traffic scene interaction relationship diagram, including a node feature matrix and an adjacency matrix.

[0096] (1) Node feature matrix

[0097] GRU is used to encode the historical data of the traffic participants and extract the dynamic features of the traffic participants.

[0098]

[0099] Among them: GRU h stands for Gated Recurrent Unit GRU, represents the dynamic features extracted from traffic participant i at time t, Represents the historical data of traffic participant i at time t.

[0100] This dynamic feature is input into the node feature vector to construct the node feature matrix.

[0101]

[0102] in: is the node feature vector of node i at time t, which contains the current speed information, location information, vehicle driving direction, pedestrian orientation and dynamic characteristics of traffic participants.

[0103] (2) Adjacency Matrix

[0104] The relationship between traffic participants is represented by the adjacency matrix, which can be expressed as:

[0105]

[0106] Where: E f ={e ij Each item in ,i,j∈[1,n]} represents the edge feature vector of the relationship between the ith traffic participant and the jth traffic participant.

[0107] The edge feature vector includes the Gaussian velocity field and the relative speed, relative position and angle between different traffic participants. The relative speed can be expressed as v ij =(v jx -v ix ,v jy -v iy ), the relative position is represented by the Euclidean distance The orientation angle can be expressed as θ ij =atan2(y j -y i ,x j -x i ); x i ,y i is the position of the ith traffic participant, v ix 、v iy is the speed of the ith traffic participant in the x and y directions.

[0108] The Gaussian velocity field is calculated as follows:

[0109] Use GPR to establish the probability distribution of the velocity field of traffic participants and obtain a quantitative description of the uncertainty of the movement of traffic participants:

[0110]

[0111] Where: A is the magnification factor, x i is the position of the ith traffic participant, x jis the position of the jth surrounding traffic participant, σ is the scale constant;

[0112] Take the i-th traffic participant and its surrounding j-th traffic participant at x i The relative velocity of the position is used to describe the Gaussian velocity field between vehicles:

[0113] B t (i, j) = K(x i , x j )K(x i , x j )ΔV(x i , x j )

[0114] ΔV(x i , x j )=V i -V j

[0115] Where: ΔV(x i ,x j ) is the distance between the ith traffic participant and the jth traffic participant around him at x i Relative velocity of position, V i 、V j is the speed of the i-th and j-th traffic participants.

[0116] S103, inputting the node feature matrix and the adjacency matrix into the graph attention neural network GAT to obtain the topological features of the graph.

[0117] The node feature matrix and adjacency matrix are input into the graph attention neural network GAT. Different attention is assigned to each node through the attention mechanism. The node features from the neighborhood are aggregated to update the node and transfer the node and edge features to obtain the topological features of the graph.

[0118] S104: Establish a trajectory model, select decision features to generate a reward function, and use maximum entropy inverse reinforcement for learning.

[0119] (1) Establishing trajectory model

[0120] The trajectory models of the participating vehicles are generated using a polynomial trajectory sampler. In order to make the generated trajectory smooth and dynamically feasible, the acceleration and acceleration along the trajectory should be time continuous. For the lateral y coordinate, the target position, velocity, and acceleration need to be specified, forming a total of six boundary conditions and the initial state, which requires a fifth-order polynomial function. For the longitudinal x-axis, only the target velocity and acceleration are required, so a fourth-order polynomial function can meet the smoothness requirement. In summary, the trajectory is represented by two polynomial functions (time-continuous functions) with respect to the x and y coordinates, respectively:

[0121]

[0122] Among them: τ represents time, {a0,...,a4} and {b0,...,b5} are the coefficients of each term of the polynomial; x represents the longitudinal position of the traffic participant in the road; y represents the transverse position of the traffic participant in the road.

[0123] Given the initial state of the vehicle, the target state, and the time T required to reach the target, the boundary conditions of the polynomial function on the vertical and horizontal axes can be expressed as:

[0124]

[0125] Where: (x s ,v xs ,a xs ,y s ,v ys ,a ys ) is the initial state, i.e. the state of τ = 0, where each sub-item is the position and the velocity and acceleration in the horizontal and vertical directions; (v xe ,a xe ,y e ,v ye ,a ye ) is the target state, i.e., the state of τ = T, in which there is no longitudinal position; is the first derivative of x and y, indicating the velocity; is the second derivative of x and y, representing acceleration.

[0126] By solving the boundary equations, the coefficients of the polynomial function can be determined. After the trajectory is generated, the position of the vehicle at any given time τ (τ≤T), as well as the velocity and acceleration, can be obtained.

[0127] (2) Reward Function

[0128] The feature selection in the reward function determines the bias of the decision model training. Features are mappings from states to values, which can capture important characteristics of states. Four features that can characterize the decision-making process of human driver behavior include: driving efficiency, comfort, risk aversion, and interaction. According to the trajectory model, the decision features on each trajectory can be obtained.

[0129] (a) Driving efficiency: Driving efficiency is selected as one of the features to reflect the desire of human drivers to reach the destination as quickly as possible, which is defined as the speed of the vehicle:

[0130]

[0131] (b) Comfort: Ride comfort is a factor that human drivers consider during driving. The comfort index is composed of the longitudinal acceleration ax , lateral acceleration a y and longitudinal jerk j x Characterization:

[0132]

[0133] (c) Risk avoidance: Human drivers tend to maintain a safe distance from surrounding vehicles. The safety distances between different drivers are different, reflecting the different perceptions of risk among different drivers. The embodiment of the present invention uses the positional relationship between the vehicle and the preceding vehicle, and the positional relationship between the following vehicle and the vehicle to characterize the risk level between the vehicle and the preceding vehicle, and the risk level between the vehicle and the following vehicle:

[0134]

[0135]

[0136] Where: x f (t) is the longitudinal position of the nearest preceding vehicle, x ego (t) is the longitudinal position of the vehicle, v ego (t) is the current longitudinal speed of the vehicle; x r (t) is the longitudinal position of the nearest rear vehicle, v r (t) is the longitudinal speed of the rear vehicle.

[0137] (d) Interaction: Humans are aware of the impact of their own behavior on surrounding traffic participants and can evaluate whether their own vehicle's driving behavior will cause inconvenience to other traffic participants (for example, sudden deceleration to give way). The embodiment of the present invention quantifies the impact of the own vehicle on other vehicles by using the sum of the absolute values ​​of the predicted decelerations of the vehicles affected by the own vehicle's behavior:

[0138]

[0139] Among them: a i (t) is the acceleration of the vehicle affected by the ego vehicle; i represents the surrounding vehicles affected by the ego vehicle's behavior.

[0140] The above features are calculated at each time step to obtain all the features of a trajectory, and the trajectory features are normalized between 0 and 1 to offset the effects of different units and scales.

[0141] Define the reward function r(s t )as follows:

[0142] r(s t )=θ T f(s t )

[0143] Where: θ=[θ1,θ2,...,θ K ] is the K-dimensional weight vector, f(s t )=[f1(s t ),f2(s t ),...,f K (s t )] is the feature vector of decision features; here K = 4.

[0144] For trajectory ζ, the corresponding reward function R(ζ) is expressed as:

[0145]

[0146] Where: f ζ are all the features of trajectory ζ;

[0147] The Boltzmann model is used to assign corresponding probabilities to all trajectories. The probability of a trajectory is proportional to the exponential of the reward of the trajectory, and we get:

[0148]

[0149] Where: P(ζ|θ) is the probability of trajectory ζ given a reward parameter θ; Z(θ) is the distribution function. Using a finite number of trajectories, the distribution function is approximated to obtain the following formula:

[0150]

[0151] in: To generate a trajectory, its initial state is consistent with ζ, is the feature vector of the trajectory, and M is the number of generated trajectories.

[0152] (3) Using maximum entropy inverse reinforcement learning weights θ

[0153] Maximum entropy IRL uses the gradient ascent method to optimize the parameters of the reward function. The weights are adjusted by maximizing an objective function. The goal of maximum entropy IRL is to learn the weights θ to maximize the likelihood of expert demonstrations in the trajectory distribution, as shown in the formula:

[0154]

[0155] The objective function can be derived as:

[0156]

[0157] in: It is a set of real human driving trajectories.

[0158] Since this objective function is difficult to obtain an analytical solution, it can be optimized using a gradient-based method. Its gradient representation is:

[0159]

[0160] Where: f ζ is the feature vector of the human driver’s driving trajectory (expert data), is a generated trajectory whose initial state is consistent with ζ. is the eigenvector corresponding to the trajectory.

[0161] This gradient can be seen as the difference in feature expectations between the human driver's trajectory and the generated trajectory, which can be expressed as:

[0162]

[0163] The reward function is updated using the gradient ascent method until the loss converges. To prevent overfitting, L2 regularization is added as follows:

[0164]

[0165] Where: λ>0 is the regularization parameter, so the gradient can be expressed as the difference in feature expectations plus the gradient of the regularization term:

[0166]

[0167] Use the gradient ascent method to update the reward function until the loss converges and obtain the weight θ.

[0168] The embodiment of the present invention provides expert data based on the NGSIM (US-101) dataset. The NGSIM (Next Generation Simulation) dataset is the US highway driving data collected by the US FHWA. NGSIM records the global position of traffic participants and the local position information of vehicles relative to the vehicle coordinate system.

[0169] S104: Combining the learned reward function with DQN, and making vehicle behavior decisions according to the topological features of the graph.

[0170] The learned reward function replaces the reward function in the original DQN to obtain an enhanced decision-making algorithm.

[0171] Input the topological features of the graph into the above decision algorithm and select the action with the greatest value in the action space as the optimal action. The action space is specifically:

[0172] The information in the action space is a combination of driving behaviors, as shown below:

[0173] a=[CL,GS,CR]

[0174] Among them: CL, GS, and CR represent lane change to the left lane, straight driving, and lane change to the right lane, respectively. These actions are based on semantic-level commands and are used to guide the movement of the vehicle.

[0175] The method of the embodiment of the present invention is verified below. First, decision training is performed, and the training parameters are set as shown in Table 1:

[0176] Table 1

[0177]

[0178] Figure 2 The reward value for each round of training in the simulation environment is given. Figure 2 It can be seen that the reward value obtained by the driverless vehicle increases synchronously with the increase of training rounds. The reward value increases rapidly during the period from 200 to 350 rounds of training, which may be related to the fact that the behavior decision algorithm learns the key driving strategy during this period. After about 830 rounds of training, the learned driving strategy gradually converges and the reward value tends to be stable.

[0179] Depend on Figure 3 It can be seen that the loss value is high at the beginning of training and has violent fluctuations, which is due to the random exploration behavior at the beginning of training. As the training progresses, the loss value decreases significantly and remains at a low level, which proves the convergence and training stability of the behavior decision method of the embodiment of the present invention.

[0180] In order to test the rationality of lane-changing behavior, the lane-changing time, the change in lateral and longitudinal displacement, and the change in speed are evaluated. The lane-changing time reflects the rapidity of the lane-changing action. If the lane-changing time is too long, it may cause the rear vehicle to decelerate or change lanes unnecessarily, increasing the risk of traffic congestion. At the same time, too fast lane-changing may prevent other traffic participants from making appropriate responses in time, thereby increasing the risk of accidents. The change in lateral and longitudinal displacement shows the movement path of the vehicle during the lane-changing process. The ideal lane-changing should be a smooth curve without sudden changes in angle. The control of lateral and longitudinal displacement within a reasonable range means that the driverless vehicle can effectively and smoothly transition from one lane to another without affecting the normal driving of other vehicles. The change in speed is the key to evaluating whether the road speed limit conditions are met during the lane-changing process. Maintaining an appropriate speed can not only ensure the safety of the lane-changing process, but also avoid interference with other traffic participants due to excessive or slow speed. These three indicators can comprehensively evaluate the safety and efficiency of the lane-changing process and ensure that the driverless vehicle can complete the lane-changing action safely and smoothly.

[0181] After the driving behavior decision module issues a command to change lanes to the left, the driverless vehicle starts to change lanes. The center-of-mass trajectory of the vehicle changing lanes from the right lane to the left lane is as follows: Figure 4 shown. Figure 4 The trajectory of the unmanned vehicle is relatively smooth, without sudden changes in curvature, which improves the comfort of the passengers while ensuring safety.

[0182] from Figure 5 and Figure 6 It can be seen that the behavior of the unmanned vehicle during accelerated lane change meets the requirements of the accelerated lane change scenario, which proves that the driving behavior decision algorithm proposed in this paper can make highly anthropomorphic lane change decisions and effectively complete the driving task when facing similar scenarios.

[0183] In order to verify the decision-making ability of the driving behavior decision algorithm in the face of special situations, a scenario requiring lane change withdrawal was designed on the simulation experimental platform. In this scenario, the trajectory of the center of mass of the unmanned vehicle is as follows: Figure 7 shown.

[0184] In this scenario, the driverless vehicle starts changing lanes after receiving the left lane change command for the first time. During the lane change, the vehicle senses that the vehicle in front of the target lane suddenly slows down and cannot complete the lane change normally. Therefore, under the action of the decision model, the current lane change process is terminated and the vehicle starts to return to the original lane.

[0185] The changes in lateral and longitudinal displacement and speed during lane changing are shown in Figure 8 and Fig. 9 The whole process is completed within 5 seconds, meeting the lane changing time requirement. The lateral and longitudinal displacements are within a reasonable range, and the speed change also complies with the road speed limit conditions.

[0186] After the simulation training and special scenario testing were completed, the decision-making method was deployed to the real vehicle platform and 10 tests were conducted on the two routes. The completion status and average status of the unmanned vehicle test process are recorded in Table 2.

[0187] Table 2

[0188]

[0189] As shown in Table 2, the driverless vehicle behavior decision method proposed in this paper can complete the given driving task under high-density mixed traffic conditions. There was no collision during the test, no frequent lane changes, and the average driving speed was close to the road speed limit. The actual vehicle test verified that the driving behavior decision method proposed in this paper can improve driving efficiency while ensuring safety, and the driving behavior decisions made will not affect other traffic participants.

[0190] The specific embodiments of the present invention are described above in conjunction with the accompanying drawings. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for optimizing an unmanned vehicle behavior decision model, characterized in that: The steps include: (1) Obtain historical data of traffic participants, including their positions, speeds, accelerations, and deflection angles over the past t time period; (2) Establishing a traffic scene interaction relationship diagram, including a node feature matrix and an adjacency matrix; using a gated recurrent unit (GRU) to encode the historical data of the traffic participants to obtain dynamic features, and inputting the dynamic features into a node feature vector to construct a node feature matrix; Gaussian process regression (GPR) is used to represent the Gaussian velocity field to construct the adjacency matrix; (3) Inputting the node feature matrix and the adjacency matrix into the graph attention neural network GAT to obtain the topological features of the graph; (4) Establish a trajectory model, select decision features to generate a reward function, and use maximum entropy inverse reinforcement for learning; (5) Combining the learned reward function with DQN to make vehicle behavior decisions based on the topological features of the graph.

2. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 1, characterized in that: The node feature matrix is ​​specifically: GRU is used to encode the historical data of the traffic participants and extract the dynamic features of the traffic participants: Among them: GRU h stands for Gated Recurrent Unit GRU, represents the dynamic characteristics of the ith traffic participant at time t, represents the historical data of the i-th traffic participant at time t; Input this dynamic feature into the node feature vector to construct the node feature matrix; in: i∈[1,n] is the node feature vector of node i at time t, which contains the current speed information, position information, deflection angle and dynamic characteristics of the i-th traffic participant.

3. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 1, characterized in that: The adjacency matrix is ​​specifically: Where: E f ={e ij Each item in ,i,j∈[1,n]} represents the edge feature vector of the relationship between the ith traffic participant and the jth traffic participant; The edge feature vector includes a Gaussian velocity field and relative velocities, relative positions and angles between the orientations of different traffic participants; The relative speed is represented by v ij =(v jx -v ix ,v jy -v iy ), the relative position is represented by the Euclidean distance The angle of orientation is denoted as θ ij =atan2(y j -y i ,x j -x i ); x i ,y i is the position of the ith traffic participant, v ix 、v iy is the speed of the ith traffic participant in the x and y directions; Use GPR to establish the probability distribution of the velocity field of traffic participants and obtain a quantitative description of the uncertainty of the movement of traffic participants: Where: A is the magnification factor, x i is the position of the ith traffic participant, x j is the position of the jth surrounding traffic participant, σ is the scale constant; Take the i-th traffic participant and its surrounding j-th traffic participant at x i The relative speed of the positions is used to describe the Gaussian velocity field between traffic participants: B t (i,j)=K(x i ,x j )K(x i ,x j ) -1 ΔV(x i ,x j ) Δv(x i ,x j )=V i -V j Where: ΔV(x i ,x j ) is the distance between the ith traffic participant and the jth traffic participant around him at x i Relative velocity of position, V i 、V j is the speed of the i-th and j-th traffic participants.

4. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 1, characterized in that: The trajectory model is specifically: The trajectory model of the traffic participant is generated using a polynomial trajectory sampler, which is represented by two polynomial functions with respect to the x and y coordinates respectively: Where: τ represents time, {a0,...,a4} and {b0,...,b5} are the coefficients of each term of the polynomial; x represents the position function of the traffic participant in the longitudinal direction of the road; y represents the position function of the traffic participant in the transverse direction of the road; The initial state, target state and time T required to reach the target of the traffic participant, and the boundary conditions of the polynomial function on the vertical and horizontal axes can be expressed as: Where: (x s ,v xs ,a xs ,y s ,v ys ,a ys ) is the initial state, i.e. the state of τ=0, where each sub-item is the position and the velocity and acceleration in the horizontal and vertical directions; (v xe ,a xe ,y e ,v ye ,a ye ) is the target state, i.e., the state of τ = T, in which there is no longitudinal position; is the first-order derivative of x and y, representing the velocity function; is the second-order derivative of x and y, representing the acceleration function; The coefficients of the polynomial function are determined by solving the boundary equations to obtain the trajectory model.

5. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 4, characterized in that: The selected decision features include: driving efficiency, comfort, risk aversion and interaction; according to the trajectory model, the decision features on each trajectory are obtained; The driving efficiency is characterized by the driving speed of the traffic participants: The comfort is determined by the longitudinal acceleration a x , lateral acceleration a y and longitudinal jerk j x Characterization: The risk aversion is characterized by the positional relationship between the vehicle and the preceding and following vehicles: Where: x f (t) is the longitudinal position of the nearest preceding vehicle, x ego (t) is the longitudinal position of the vehicle, v ego (t) is the current longitudinal speed of the vehicle; x r (t) is the longitudinal position of the nearest rear vehicle, v r (t) is the longitudinal speed of the following vehicle; The interaction is characterized by the sum of the absolute values ​​of the predicted deceleration of the traffic participant affected by the behavior of the vehicle: Among them: a i (t) is the acceleration of the traffic participant affected by the ego vehicle; i represents the surrounding traffic participants affected by the ego vehicle behavior; The above decision features are calculated at each time step to obtain all the features of a trajectory and normalize them between 0 and 1 to offset the effects of different units and scales.

6. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 5, characterized in that: The reward function is specifically: Define the reward function r(s t )as follows: r(s t )=θ T f(s t ) Where: θ=[θ1,θ2,...,θ K ] is the K-dimensional weight vector, f(s t )=[f1(s t ),f2(s t ),...,f K (s t )] is the feature vector of the decision feature; For trajectory ζ, the corresponding reward function R(ζ) is expressed as: Where: f ζ are all the features of trajectory ζ; The Boltzmann model is used to assign corresponding probabilities to all trajectories. The probability of a trajectory is proportional to the exponential of the reward of the trajectory, and we get: Where: P(ζ|θ) is the probability of trajectory ζ given a reward parameter θ; Z(θ) is the distribution function. Using a finite number of trajectories, the distribution function is approximated to obtain the following formula: in: To generate a trajectory, its initial state is consistent with ζ, is the feature vector of the trajectory, and M is the number of generated trajectories.

7. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 6, characterized in that: The specific learning method using maximum entropy inverse reinforcement is as follows: Use maximum entropy to learn weights θ; Based on the maximum entropy, the reward function is updated using the gradient ascent method until the loss converges and the weight θ is obtained; in: is a set of real human driving trajectories; f ζ is the feature vector of the human driver’s driving trajectory, is a generated trajectory whose initial state is consistent with ζ. is the eigenvector corresponding to the trajectory.

8. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 1, characterized in that: The step (5) is specifically as follows: (5.1) replacing the reward function in the original DQN with the learned reward function to obtain an enhanced decision algorithm; (5.2) The topological features of the graph are input into the enhanced decision algorithm to select the action with the greatest value in the action space as the optimal action.

9. The method for optimizing the behavior decision model of an unmanned vehicle according to claim 8, characterized in that: The action space is specifically: The information in the action space is a combination of driving behaviors, as shown below: a=[CL,GS,CR] Among them: CL, GS, and CR represent lane change to the left lane, straight driving, and lane change to the right lane, respectively. These actions are based on semantic-level commands and are used to guide the movement of the vehicle.

Images