Method for evaluating and generating human-like driving track

Abstract

The invention relates to the technical field of automatic driving human-like driving, and particularly discloses a human-like driving track evaluation and generation method, which comprises the following steps: S1, constructing a driving track data set THAD; s2, constructing deep feature mapping between the vehicle state-track and the corresponding human-like level based on a comparison network, and training a level evaluator for accurately evaluating the state-track human-like degree; and S3, combining the grade evaluator with reinforcement learning, and training the driving reward signal provided by the grade evaluator for reinforcement learning to obtain a track generator for generating a safe and human-like driving track. According to the invention, through combination with the comparison network, when the reinforcement learning is trained to generate the trajectory, the reward supervision signal of human-like level conversion of the trajectory can be obtained, fine adjustment of numerous parameters of a vehicle in a traditional reward function is avoided, and the trajectory generated by reinforcement learning can be optimized towards a human-like direction.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving humanoid driving, and in particular to a method for evaluating and generating a humanoid driving trajectory. Background Art

[0002] Humanoid driving technology is a hot topic in the field of autonomous driving. Its purpose is to achieve autonomous driving of vehicles by simulating human decision-making and driving safely and reasonably in complex traffic environments. Supervised learning has been widely used in autonomous driving due to its advantages in pattern recognition and prediction. For example, imitation learning is to train autonomous driving systems by imitating the behavior of experts. However, in the continuous decision-making process, the model may experience distribution shift, and due to over-reliance on the results of expert behavior rather than the reasons for these behaviors, causal confusion occurs, resulting in errors in the final decision and affecting driving safety.

[0003] Unlike supervised learning, reinforcement learning is a method of learning the optimal strategy through trial and error. In autonomous driving systems, reinforcement learning allows the model to continuously try and adjust the driving strategy in a simulated or real environment to maximize a certain reward function (such as safety, efficiency, etc.), thereby learning the optimal driving behavior. Reinforcement learning mainly consists of five basic elements: agent, environment, state, action, and reward. The agent changes its state by performing actions in the environment, and adjusts its strategy based on the feedback from the environment (i.e., reward) to maximize the long-term accumulated reward, which can effectively alleviate the distribution shift and solve the causal confusion problem. Although reinforcement learning can achieve good performance in complex tasks, it still has two defects: First, it is impossible to judge the similarity between the generated trajectory and the natural driving trajectory of humans, which may cause the autonomous driving behavior to be out of touch with human habits and increase the risk of accidents. Second, the design of the reward mechanism is highly complex, involving the fine adjustment of many vehicle parameters, such as speed, acceleration, etc., making the construction of the reward function cumbersome and complicated. This complexity not only increases the difficulty of algorithm design, but also may have an adverse effect on the learning efficiency and the quality of the final driving strategy. Summary of the invention

[0004] Based on the above defects, the present invention proposes a novel method for generating human-like driving trajectories. By combining a contrast network, when training reinforcement learning to generate trajectories, a reward supervision signal for the human-like level conversion of the trajectory can be obtained. This not only avoids the fine adjustment of numerous vehicle parameters in the traditional reward function, but also optimizes the trajectory generated by reinforcement learning in the direction of human-like.

[0005] In order to achieve the above object, the present invention provides a method for evaluating and generating a human-like driving trajectory, comprising the following steps:

[0006] S1. Construct driving trajectory dataset THAD;

[0007] S2. Based on the contrast network, a deep feature mapping between the vehicle state-trajectory and the corresponding human-like level is constructed, and a level evaluator that accurately evaluates the human-likeness of the state-trajectory is trained;

[0008] S3. Combining the grade evaluator with reinforcement learning, and using the driving reward signal provided by the grade evaluator for reinforcement learning to train a trajectory generator that generates safe, human-like driving trajectories.

[0009] Preferably, step S1 specifically includes the following steps:

[0010] S11, extracting vehicle expert state-trajectory information based on the original highway highD dataset;

[0011] S12, using the lattice trajectory planning method to generalize the expert state-trajectory information of the vehicle in different states so that the vehicle corresponds to 6 trajectory lines in different states, including 1 expert trajectory and 5 lattice generalized trajectories;

[0012] S13. A human expert judges the human-like level of the trajectory according to the six trajectory lines corresponding to the vehicle in different states, records the state trajectory-level information pairs, and constructs the driving trajectory dataset THAD.

[0013] Preferably, the generalization process in step S12 is as follows:

[0014] S1201. Obtain the initial state coordinates of the vehicle in the Frenet coordinate system at time t1: represents the derivative of S with respect to time t, which is the longitudinal velocity, is the longitudinal acceleration, d′ represents the derivative of d with respect to s, is the lateral velocity, and d″ is the lateral acceleration;

[0015] S1202, select the expert trajectory in the original database as the reference line, and obtain the trajectory in the Cartesian coordinate system by fitting the horizontal and vertical trajectories through polynomials, and then use the cost function:

[0016] c ost =(a×Lat+b×Col+c×Laa+d×Ve+e×Lon),

[0017] The trajectories that meet the vehicle kinematic constraints are sorted, where Lat represents the lateral offset cost. The farther the lateral distance from the reference line is, the higher the cost of the trajectory; Col represents the collision risk cost. If there is a collision risk, the collision cost will be relatively high; Laa represents the lateral acceleration cost; Ve represents the speed cost. The greater the speed, the greater the corresponding trajectory cost; Lon represents the longitudinal acceleration cost; a, b, c, d, e, f represent their respective weight coefficients;

[0018] S1203, on the trajectory through c ost After scoring and sorting, 5 generalization trajectories are taken out at intervals.

[0019] Preferably, the state trajectory-level information pair includes vehicle final state information and final trajectory information, and the final vehicle state information includes:

[0020] State=(dis,w,h,v,a,near_x,near_y,near_w,near_h,near_v,near_a),

[0021] Where dis represents the lateral distances of the three lanes on both sides of the vehicle relative to the current direction of the vehicle, w and h represent length and width, v represents speed, a represents acceleration, near_x, near_y, near_w, near_h, near_v, and near_a represent the relative coordinates, length and width, speed, and acceleration of the surrounding vehicles in the same direction; the final trajectory information includes:

[0022] Traj=(x*, y*, w, h, v*, a*), where x*, y*, v*, a* represent the relative coordinates, relative speed, and acceleration of the vehicle in each frame within the next three seconds under the current state.

[0023] Preferably, step S2 specifically includes the following steps:

[0024] S21, constructing a state-trajectory feature encoder and a human-level feature encoder, which are used to extract high-dimensional feature information of vehicle state-trajectory information and trajectory human-level in the new data set respectively;

[0025] S22. Using the feature comparison mechanism, a contrast loss function is introduced to train the evaluator to maximize the similarity between the correct state-trajectory features and the human-like level.

[0026] Preferably, the state-trajectory feature encoder adopts an attention mechanism to capture high-dimensional feature information of the state and trajectory. The calculation process is as follows: first determine the attention score between the vehicle state and trajectory: Where Q is the vehicle state vector, K is the trajectory information vector, and d kis a scaling factor used to scale the dot product to avoid gradient vanishing or exploding problems. Then, the attention scores of all queries are normalized by applying softmax to obtain a set of normalized attention weights W = softmax(Score(Q, K)). Finally, Output = WV is used to weight the sum of the obtained attention weights and the value vector V, where V is the trajectory information vector. The final Output represents the comprehensive feature information of the vehicle state and trajectory.

[0027] The human-level feature encoder converts the 6 levels into one-hot encoding to be used as input to a multi-layer perceptron (MLP), which learns the mapping from input to output through its hidden layers.

[0028] Preferably, the contrast loss function is:

[0029]

[0030] where x i Represents the feature representation of the i-th vehicle state-trajectory, y i Represents the feature representation of the human-like level corresponding to the i-th vehicle state-trajectory, M is the number of levels, sim represents the similarity function, τ is a scaling factor that performs temperature scaling on the similarity to control the sensitivity of the model, and N is the batch size.

[0031] Preferably, step S3 specifically includes the following steps:

[0032] S31. Build an environment for a humanoid trajectory generator, including state space, action space, and reward function;

[0033] S32. Using the PPO algorithm, the human-like level of the generated trajectory is evaluated by the level evaluator and converted into a reward supervision signal of the PPO to train a trajectory generator that can generate safe and human-like driving trajectories.

[0034] Preferably, the state space is expressed as: S = (d', s, w, h, v, hea, col, oline, Rad), including the lateral speed d' and longitudinal offset s, w, h, heading angle hea, speed v of the vehicle in the Frenet coordinate system, collision signal col, and signal oline of exceeding the same-direction edge lane, and Rad includes the distance of surrounding vehicles detected by rays, the distance on both sides of the current lane, and the distance of the same-direction edge lane, and the speed information of surrounding vehicles;

[0035] The action space constructs the trajectory planning action in the Frenet coordinate system framework, and the action vector a is defined as in represents the target lateral offset from the current time T to the future time step T, represents the longitudinal velocity of the target. Generate the future target position based on a;

[0036] The reward function designs the following driving behavior rewards, including Swrew: the faster the vehicle speed, the greater the reward; Ttcrew reward: by calculating the distance between the vehicle and the vehicle in front and their relative speed, the greater the distance, the smaller the relative speed, and the lower the penalty; Colrew: when the vehicle collides with surrounding vehicles, it will be punished; Outflanerew: when the vehicle exceeds the edge lane of the same direction, it will be punished; Keeprew: in the current lane, when the vehicle can continue to stay in the center of the lane, the smaller the penalty, using the trajectory human-like evaluator, by judging the human-like level of the generated trajectory at each step of reinforcement learning, the evaluation result is converted into a human-like degree reward Rankrew, the higher the human-like degree of the generated trajectory, the smaller the penalty, in the training and updating process of the trajectory generator network, the reward value is defined as: reward = Rankrew + r, where r = Colrew + Outflanerew.

[0037] Preferably, the PPO algorithm maximizes the cumulative reward by optimizing the strategy. It updates and maintains the policy network Actor and the value network Critic during the training process. The policy network Actor loss function L is defined as:

[0038]

[0039] where π θ (a|s) is the probability that the new Actor chooses action a given state s, is the probability of the old Actor under the same conditions, ε is the clip function ratio, and A is the advantage function, that is: A t =δ t +(γλ)δ t+1 +…+…+(γλ) T-t-1 δ T-1 ,

[0040] where δ t =r t +γV φ (s t+1 )-V φ (s t ), discount factor 0≤γ<1, parameter 0≤λ≤1, the value network Critic uses mean square error (MSE) as the loss function:

[0041]

[0042] Where V φ (st ) is the current Critic network (with parameter φ) in state s t The predicted value, r t is in state s t The immediate reward after performing an action, V φ (s t+1 ) is the current Critic network in state s t+1 The predicted value.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. At the reward function level, the present invention solves the problem that the intelligent agent in reinforcement learning involves too many vehicle parameters, which leads to complex reward function design and takes too long to adjust each reward weight.

[0045] 2. At the level of optimizing generated trajectories, the present invention solves the problem that during reinforcement learning training, the optimization direction cannot determine the similarity between the generated trajectory and the natural driving trajectory of humans, resulting in the final optimized trajectory being possibly not human-like, increasing the risk of accidents, and optimizes the trajectory generated by reinforcement learning in a human-like direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0047] Figure 2 It is the accuracy of the rank evaluator of the present invention in the training set and the test set in different rounds. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The present invention is a new method combining contrastive network and reinforcement learning. Firstly, a new driving trajectory dataset THAD with human labels is constructed based on the highD dataset, and this dataset is used for training and testing of the large model contrastive network to obtain an evaluator that can accurately judge the human-like level of vehicle trajectories. Then, the human-like level is converted into a reward signal to train a trajectory generator to generate safe and human-like driving trajectories.

[0050] In order to achieve the above objectives, the present invention adopts the following technical solution: a method for evaluating and generating a human-like driving trajectory, comprising the following steps:

[0051] Step S1: construct a driving trajectory dataset THAD.

[0052] Furthermore, step S1 specifically includes the following steps:

[0053] S11, extracting vehicle expert state-trajectory information based on the original highway highD dataset;

[0054] S12, using the lattice trajectory planning method to generalize the expert state-trajectory information of the vehicle in different states so that the vehicle corresponds to 6 trajectory lines in different states, including 1 expert trajectory and 5 lattice generalized trajectories;

[0055] S13. A human expert judges the human-like level of the trajectory according to the six trajectory lines corresponding to the vehicle in different states, records the state trajectory-level information pairs, and constructs the driving trajectory dataset THAD.

[0056] Specifically, the original data set in step S11 selects the trajectory data of location 1 of the highD data set. The road scene is a two-way 4-lane highway with a total length of 410 meters, which contains the trajectory information of each frame (1 second is 25 frames) of 1047 vehicles, including the coordinates, size, speed, acceleration, driving direction and other information of the vehicle. For each vehicle, the vehicle's state at multiple different times and the expert trajectory information of the next three seconds under the current state are processed, and the details of the vehicle's lane change are paid attention to. The state and trajectory information are also recorded before and after the vehicle changes lanes. In order to increase the diversity of the vehicle's trajectory under different states, the lattice algorithm is introduced, and the generalized trajectory processing is performed based on the existing trajectory of each state, so that the vehicle corresponds to 6 possible trajectory lines in different states, including 1 expert trajectory recorded in the original data set and 5 lattice generalized trajectories, and the 6 trajectory lines in each state are graded by human experts. After the trajectories of all vehicles in different states are labeled with human-like levels, the final vehicle state information includes:

[0057] State=(dis,w,h,v,a,near_x,near_y,near_w,near_h,near_v,near_a),

[0058] Where dis represents the distance between the three lanes on both sides of the vehicle relative to the current direction of the vehicle, w and h represent length and width, v represents speed, and a represents acceleration. near_x, near_y, near_w, near_h, near_v, and near_a represent the relative coordinates, length and width, speed, and acceleration of the surrounding vehicles in the same direction. The trajectory information includes:

[0059] Traj=(x*,y*,w,h,v*,a*),

[0060] Among them, x*, y*, v*, a* represent the relative coordinates, relative speed, and acceleration of each frame in the next three seconds under the current state of the vehicle. There are 67,656 state trajectory-level information pairs in total, forming a new driving trajectory dataset THAD with human labels, which will be used for the subsequent training and testing of the trajectory evaluator and trajectory generator.

[0061] Among them, the lattice generalization process mainly consists of state sampling, trajectory generation, collision detection and evaluation function.

[0062] 1) The lattice algorithm first obtains the initial state coordinates of the vehicle in the Frenet coordinate system at time t1: Compared with the traditional Cartesian coordinate system, the Frenet coordinate system simplifies path planning. It splits the problem into independent longitudinal and lateral dimensions. The vehicle position is determined by the longitudinal offset s and the lateral offset d, thus providing a more intuitive path planning method. Since the lateral motion of the vehicle is induced by the longitudinal motion, d is set as a function of s. represents the derivative of S with respect to time t, which is the longitudinal velocity, that is, the velocity along the reference line, is the longitudinal acceleration, d′ represents the derivative of d with respect to s, is the lateral velocity, and d″ is the lateral acceleration. Here, the present invention chooses not to use the traditional lane centerline as the reference line, but selects the expert trajectory in the original database as the reference line. The lane centerline provides the most basic driving path, while the expert trajectory can reflect a more human-like driving mode. In order to generate a trajectory, sample a coordinate of the end state of t2:

[0063] 2) Perform polynomial fitting on the initial state t1 and the final state t2. The longitudinal fitting polynomial function s(t) and the transverse fitting quintic polynomial function d(s) are:

[0064] s(t)=c1t 5 +c2t 4 +c3t 3 +c4t 2 +c5t+c6,

[0065] d(s)=k1s 5 +k2s 4 +k3s 3 +k4s 2 +k5s+k6,

[0066] Where c and k are the coefficients of the polynomial.

[0067] 3) Synthesize the fitted horizontal and vertical trajectories and calculate t * The longitudinal offset at time s* =s(t * ) and lateral offset d * =d(s * ), and then restore the trajectory points in a two-dimensional plane through the global reference line From * A series of trajectory points P1, P2, P3, ... are obtained, and finally a series of complete trajectories are formed. Finally, the trajectory points are converted from the Frenet coordinate system to the Cartesian coordinate system to obtain the physical world sampling trajectory.

[0068] 4) Since both the lateral and longitudinal directions are obtained through high-order polynomial interpolation, the trajectory in the Cartesian coordinate system is also smooth, but it also needs to meet mandatory constraints such as maximum acceleration and deceleration, vehicle kinematic curvature constraints, and cost constraints such as staying away from obstacles and staying close to the reference line. The trajectory that meets the constraints is calculated according to the cost function c ost Sort the trajectories:

[0069] c ost =(a×Lat+b×Col+c×Laa+d×Ve+e×Lon),

[0070] The trajectories that meet the vehicle kinematic constraints are sorted, where Lat represents the lateral offset cost. The farther the lateral distance from the reference line is, the higher the cost of the trajectory; Col represents the collision risk cost. If there is a collision risk, the collision cost will be relatively high; Laa represents the lateral acceleration cost; Ve represents the speed cost. The higher the speed, the higher the corresponding trajectory cost; Lon represents the longitudinal acceleration cost; a, b, c, d, e, and f represent their respective weight coefficients. To ensure the difference of the generalized trajectory, when the trajectory passes through c ost After scoring and sorting, 5 generalized trajectories are taken out at intervals, providing diverse human-like trajectory information for the dataset.

[0071] Step S2: construct a deep feature mapping between the vehicle state-trajectory and the corresponding human-like level based on the contrast network, and train a level evaluator that accurately evaluates the human-likeness of the state-trajectory.

[0072] Furthermore, step S1 specifically includes the following steps:

[0073] S21, constructing a state-trajectory feature encoder and a human-level feature encoder, which are used to extract high-dimensional feature information of vehicle state-trajectory information and trajectory human-level in the new data set respectively;

[0074] S22. Using the feature comparison mechanism, a contrast loss function is introduced to train the evaluator to maximize the similarity between the correct state-trajectory features and the human-like level.

[0075] Specifically, the state-trajectory feature encoder in step S21 uses an attention mechanism to capture high-dimensional feature information of the state and trajectory. The calculation process first determines the attention score between the vehicle state and trajectory: Where Q is the vehicle state vector, K is the trajectory information vector, and d k is a scaling factor used to scale the dot product to avoid the problem of gradient vanishing or exploding. Subsequently, the attention scores of all queries are normalized by applying softmax, and then a set of normalized attention weights W = softmax(Score(Q,K)) is obtained. Finally, Output = WV is used to weight the sum of the obtained attention weights to the value vector V, where V is the trajectory information vector, and the final Output represents the comprehensive feature information of the vehicle state and trajectory. The specific input state dimension is 70 and the trajectory dimension is 302. Through 3 linear layers, the state dimension and trajectory dimension are mapped to 512 dimensions respectively, and then a fully connected layer is used to calculate the similarity score between the state and the trajectory. When calculating the attention weight, the Softmax activation function is used to ensure that the sum of the weights is 1. In addition, there is a linear layer used to further integrate the output of the attention mechanism to keep the dimension at 512.

[0076] The human-level encoder converts 6 levels into one-hot encoding to be used as the input of the multi-layer perceptron MLP, which learns the mapping from input to output through its hidden layer. In this process, the MLP can learn the feature representation of the input data and output the feature information of the human-level. The specific structure input is a one-hot encoding of 6 levels with a dimension of 6, which is mapped to the hidden layer through a fully connected layer with a dimension of 128. The ReLU activation function is then applied to increase the network learning of more complex feature representations, and finally through the fully connected layer, the output dimension is 512.

[0077] Furthermore, the feature comparison mechanism in step S22 is to introduce a loss function after obtaining the state-trajectory feature representation and the human-level feature representation:

[0078]

[0079] where x i Represents the feature representation of the i-th vehicle state-trajectory. iRepresents the feature representation of the human-like level corresponding to the i-th vehicle state-trajectory. M is the number of levels, sim represents the similarity function, and the dot product is calculated here to evaluate the similarity score between each state-trajectory feature and the human-like level feature. τ is a scaling factor that performs temperature scaling on the similarity to control the sensitivity of the model, and N is the batch size, that is, the number of state-trajectories in each training iteration. In this way, the model is able to judge the correlation between features and learn to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. That is, learning to maximize the similarity between the correct state-trajectory features and the human-like level, and minimize the similarity between the wrong state-trajectory features and the human-like level. The model is trained for 1500 rounds, and the batch_size of the training set data in each round is 600. After every 25 rounds, a comprehensive test is performed on the test data set, and the test batch_size remains unchanged. Through this comparative learning strategy, the model can more accurately learn the correlation between state-trajectory features and human-like levels, and improve the accuracy of its prediction of human-like levels. In the present invention, positive samples refer to states-trajectories corresponding to correct human-like levels, while negative samples correspond to incorrect human-like levels. Through this comparative learning strategy, the model can more accurately learn the correlation between state-trajectory features and human-like levels, and improve the accuracy of its prediction of human-like levels.

[0080] Step S3: combining the grade evaluator with reinforcement learning, and using the driving reward signal provided by the grade evaluator for reinforcement learning to train a trajectory generator that generates a safe, human-like driving trajectory.

[0081] Furthermore, step S3 specifically includes the following steps:

[0082] S31. Build an environment for a humanoid trajectory generator, including state space, action space, and reward function;

[0083] S32. Using the PPO algorithm, the human-like level of the generated trajectory is evaluated by the level evaluator and converted into a reward supervision signal of the PPO to train a trajectory generator that can generate safe and human-like driving trajectories.

[0084] Specifically, in the PPO algorithm of the humanoid trajectory generator, the state space is S = (d′, s, w, h, v, hea, col, oline, Rad), which consists of two parts. The first part includes the lateral speed d′ and longitudinal offset s, length w, width h, heading angle hea, speed v, collision signal col, and signal oline of exceeding the same direction edge lane in the Frenet coordinate system. The second part Rad can detect obstacles around the current vehicle by simulating a radar sensor. Taking the current vehicle as the center point, 100 radar rays cover 360 degrees on average, and the length of the radar ray is 100m, including the distance of the surrounding vehicles detected by the ray, the distance on both sides of the current lane, and the distance of the same direction edge lane, and the speed information of the surrounding vehicles.

[0085] For the action space, the trajectory planning action is constructed in the Frenet coordinate system framework, and the action vector is defined in is the lateral displacement of the target in the future time step T, is the longitudinal velocity, and the position of the target satisfies the following polynomial relationship:

[0086]

[0087] Where m, n are the polynomial degrees, α k , β k are polynomial coefficients, x, y, θ, k, v, acc are the horizontal and vertical coordinates in the Cartesian coordinate system, heading angle, curvature, speed, acceleration, through F coor , converted to s in Frenet coordinates, d,d ’ d”, respectively, longitudinal offset, longitudinal velocity, longitudinal acceleration, lateral offset, lateral velocity, and lateral acceleration. Then the target information in the future time step T is:

[0088]

[0089] The reward function in step S31 is designed with the following driving behavior rewards, including Srew: the faster the vehicle speed, the greater the reward; Ttcrew reward: by calculating the distance between the vehicle and the vehicle in front and their relative speed, the greater the distance, the smaller the relative speed, and the lower the penalty; Colrew: when the vehicle collides with surrounding vehicles, it will be punished; Outflanerew: when the vehicle exceeds the edge lane of the same direction, it will be punished; Keeprew: in the current lane, when the vehicle can continue to stay in the center of the lane, the penalty is smaller. Using the trajectory human-like evaluator, by judging the human-like level of the trajectory generated at each step of reinforcement learning, the evaluation result is converted into a human-like degree reward Rankrew. The higher the human-like degree of the generated trajectory, the smaller the penalty.

[0090] It is worth noting that during the training and updating process of the generator, the reward value is only defined as: reward = Rankrew + r, where r = Colrew + Outflanerew, which guides the intelligent agent to simply and efficiently learn how to generate an autonomous driving trajectory that is both safe and highly close to human driving habits. Other reward items are used as auxiliary tools to observe and verify the effectiveness of the method in depth, ensuring that the generated trajectory meets the optimization standards in many aspects. Among them, the specific structure of the policy network consists of an input layer, a hidden layer, and an output layer. The input layer accepts the state of the environment as input, with a state dimension of 409. A fully connected layer is used to map the input to the hidden layer. The number of neurons in the hidden layer is 300, and the ReLU activation function is used. The output layer contains two sublayers, which are used to predict the mean and standard deviation of the action. The mean is scaled and activated using the hyperbolic tangent function, and the standard deviation uses the Softplus activation function. The value network also consists of an input layer, a hidden layer, and an output layer. The state dimension is 409, the hidden layer dimension is 300, and the output layer outputs a value to represent the estimated value of the current state. The agent uses the first 5 frames of the planned frame number for each step, and the termination conditions of the vehicle step number are designed as follows: (1) The vehicle collides. During the training process, when the vehicle collides with the surrounding vehicles, the current step ends and the training starts at a new step. (2) The vehicle leaves the road. During the training process, when the intelligent vehicle leaves the road, the current step ends and the training starts at a new step. (3) The vehicle exceeds the edge lane. During the training process, when the vehicle exceeds the edge lane, the current step ends and the training starts at a new step.

[0091] Furthermore, the PPO in step S32 maximizes the cumulative reward by optimizing the strategy, which updates and maintains the policy network Actor and the value network Critic during the training process. The present invention applies PPO-clip, and the Actor updates the strategy:

[0092]

[0093] To maximize the Actor loss function L, the Actor loss function L is defined as:

[0094]

[0095] where π θ (a|s) is the probability that the new Actor chooses action a given state s, is the probability of the old Actor under the same conditions, ε is the clip function ratio, and A is the advantage function, that is:

[0096] A t =δ t +(γλ)δ t+1 +…+…+(γλ) T-t-1 δ T-1 ,

[0097] where δ t =r t +γV φ (s t+1 )-V φ (s t ), discount factor 0≤γ<1, parameter 0≤λ≤1. t represents the current time step, T represents the last time step of the entire sequence, so that the strategy tends to choose actions with higher advantage values. Critic is updated by minimizing the loss function, using mean square error (MSE) as the loss function:

[0098]

[0099] Where V φ (s t ) is the current Critic network (with parameter φ) in state s t The predicted value, r t is in state s t The immediate reward after performing an action, V φ (s t+1 ) is the current Critic network in state s t+1 The predicted value.

[0100] The present invention first uses a contrast network to provide a reward signal for reinforcement learning, and solves the problem of complex parameter adjustment of the reward function from the level of trajectory human-likeness rather than fine-tuning the numerous parameters of the vehicle. Secondly, the algorithm enables the trajectory generated by the intelligent agent to be optimized in the direction of safety and human-likeness, so that it can understand the reason why the trajectory is highly human-like, and the agent will perform actions more similar to expert behavior, so that it can get better rewards. This solves the problem that reinforcement learning cannot judge the similarity between the generated trajectory and the natural driving trajectory of humans, and may not be similar to the human driving trajectory, increasing the risk of accidents.

[0101] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A method for evaluating and generating human-like driving trajectories, characterized in that: The following steps are involved: S1. Construct driving trajectory dataset THAD; S2. Based on the contrast network, a deep feature mapping between the vehicle state-trajectory and the corresponding human-like level is constructed, and a level evaluator that accurately evaluates the human-likeness of the state-trajectory is trained; S3. Combining the grade evaluator with reinforcement learning, and using the driving reward signal provided by the grade evaluator for reinforcement learning to train a trajectory generator that generates safe, human-like driving trajectories.

2. The method for evaluating and generating a human-like driving trajectory according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11, extracting vehicle expert state-trajectory information based on the original highway highD dataset; S12, using the lattice trajectory planning method to generalize the expert state-trajectory information of the vehicle in different states so that the vehicle corresponds to 6 trajectory lines in different states, including 1 expert trajectory and 5 lattice generalized trajectories; S13. A human expert judges the human-like level of the trajectory according to the six trajectory lines corresponding to the vehicle in different states, records the state trajectory-level information pairs, and constructs the driving trajectory dataset THAD.

3. The method for evaluating and generating a human-like driving trajectory according to claim 2, characterized in that: The generalization process in step S12 is as follows: S1201. Obtain the initial state coordinates of the vehicle in the Frenet coordinate system at time t1: represents the derivative of S with respect to time t, which is the longitudinal velocity, is the longitudinal acceleration, d ′ represents the derivative of d with respect to s, which is the lateral velocity, and d″ is the lateral acceleration; S1202, select the expert trajectory in the original database as the reference line, and obtain the trajectory in the Cartesian coordinate system by fitting the horizontal and vertical trajectories through polynomials, and then use the cost function: c ost =(a×Lat+b×Col+c×Laa+d×Ve+e×Lon), The trajectories that meet the vehicle kinematic constraints are sorted, where Lat represents the lateral offset cost. The farther the lateral distance from the reference line is, the higher the cost of the trajectory; Col represents the collision risk cost. If there is a collision risk, the collision cost will be relatively high; Laa represents the lateral acceleration cost; Ve represents the speed cost. The greater the speed, the greater the corresponding trajectory cost; Lon represents the longitudinal acceleration cost; a, b, c, d, e, f represent their respective weight coefficients; S1203, on the trajectory through c ost After scoring and sorting, 5 generalization trajectories are taken out at intervals.

4. The method for evaluating and generating a human-like driving trajectory according to claim 2, characterized in that: The state trajectory-level information pair includes vehicle final state information and final trajectory information, and the final vehicle state information includes: State=(dis,w,h,v,a,near_x,near_y,near_w,near_h,near_v,near_a), Where dis represents the lateral distances of the three lanes on both sides of the vehicle relative to the current direction of the vehicle, w and h represent length and width, v represents speed, a represents acceleration, near_x, near_y, near_w, near_h, near_v, and near_a represent the relative coordinates, length and width, speed, and acceleration of the surrounding vehicles in the same direction; the final trajectory information includes: Traj=(x*, y*, w, h, v*, a*), where x*, y*, v*, a* represent the relative coordinates, relative speed, and acceleration of the vehicle in each frame within the next three seconds under the current state.

5. The method for evaluating and generating a human-like driving trajectory according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21, constructing a state-trajectory feature encoder and a human-level feature encoder, which are used to extract high-dimensional feature information of vehicle state-trajectory information and trajectory human-level in the new data set respectively; S22. Using the feature comparison mechanism, a contrast loss function is introduced to train the evaluator to maximize the similarity between the correct state-trajectory features and the human-like level.

6. The method for evaluating and generating a human-like driving trajectory according to claim 5, characterized in that: The state-trajectory feature encoder adopts an attention mechanism to capture the high-dimensional feature information of the state and trajectory. The calculation process is as follows: first determine the attention score between the vehicle state and trajectory: Where Q is the vehicle state vector, K is the trajectory information vector, and d k is a scaling factor used to scale the dot product to avoid gradient vanishing or exploding problems. Then, the attention scores of all queries are normalized by applying softmax to obtain a set of normalized attention weights W = softmax(Score(Q, K)). Finally, Output = WV is used to weight the sum of the obtained attention weights and the value vector V, where V is the trajectory information vector. The final Output represents the comprehensive feature information of the vehicle state and trajectory. The human-level feature encoder converts the 6 levels into one-hot encoding to be used as input to a multi-layer perceptron (MLP), which learns the mapping from input to output through its hidden layers.

7. The method for evaluating and generating a human-like driving trajectory according to claim 6, characterized in that: The contrast loss function is: where x i Represents the feature representation of the i-th vehicle state-trajectory, y i Represents the feature representation of the human-like level corresponding to the i-th vehicle state-trajectory, M is the number of levels, sim represents the similarity function, τ is a scaling factor that performs temperature scaling on the similarity to control the sensitivity of the model, and N is the batch size.

8. The method for evaluating and generating a human-like driving trajectory according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31. Build an environment for a humanoid trajectory generator, including state space, action space, and reward function; S32. Using the PPO algorithm, the human-like level of the generated trajectory is evaluated by the level evaluator and converted into a reward supervision signal of the PPO to train a trajectory generator that can generate safe and human-like driving trajectories.

9. The method for evaluating and generating a human-like driving trajectory according to claim 8, characterized in that: The state space is expressed as: S = (d ′ ,s,w,h,v,hea,col,oline,Rad), including the lateral velocity d of the vehicle in the Frenet coordinate system ′ and longitudinal offset s, w, h, heading angle hea, speed v, collision signal col, signal of exceeding the same direction edge lane oline, Rad includes the distance of surrounding vehicles detected by the ray, the distance on both sides of the current lane, the distance of the same direction edge lane, and the speed information of surrounding vehicles; The action space constructs the trajectory planning action in the Frenet coordinate system framework, and the action vector a is defined as in represents the target lateral offset from the current time T to the future time step T, Represents the target longitudinal velocity, generating the future target position based on a; The reward function designs the following driving behavior rewards, including Swrew: the faster the vehicle speed, the greater the reward; Ttcrew reward: by calculating the distance between the vehicle and the vehicle in front and their relative speed, the greater the distance, the smaller the relative speed, and the lower the penalty; Colrew: when the vehicle collides with surrounding vehicles, it will be punished; Outflanerew: when the vehicle exceeds the edge lane of the same direction, it will be punished; Keeprew: in the current lane, when the vehicle can continue to stay in the center of the lane, the smaller the penalty, using the trajectory human-like evaluator, by judging the human-like level of the generated trajectory at each step of reinforcement learning, the evaluation result is converted into a human-like degree reward Rankrew, the higher the human-like degree of the generated trajectory, the smaller the penalty, in the training and updating process of the trajectory generator network, the reward value is defined as: reward = Rankrew + r, where r = Colrew + Outflanerew.

10. The method for evaluating and generating a human-like driving trajectory according to claim 9, characterized in that: The PPO algorithm maximizes the cumulative reward by optimizing the strategy. It updates and maintains the policy network Actor and the value network Critic during the training process. The policy network Actor loss function L is defined as: where π θ (a|s) is the probability that the new Actor chooses action a given state s, is the probability of the old Actor under the same conditions, ε is the clip function ratio, and A is the advantage function, that is: A t =d t +(gl)d t+1 +…+…+(cl) T-t-1 d T-1 , where δ t =r t +γV φ (s t+1 )-V φ (s t ), discount factor 0≤γ<1, parameter 0≤λ≤1, the value network Critic uses mean square error (MSE) as the loss function: Where V φ (s t ) is the current Critic network (with parameter φ) in state s t The predicted value, r t is in state s t The immediate reward after performing an action, V φ (s t+1 ) is the current Critic network in state s t+1 The predicted value.

Images