A driving behavior simulation method and device based on inverse reinforcement learning

By constructing a driver behavior model through maximum entropy deep inverse reinforcement learning, quantifying the intensity of vehicle interaction and performing soft bucket clustering, the problem of traffic simulation systems being unable to accurately predict human driver intentions is solved, and safety verification and risk prediction are realized in complex environments.

CN123116871APending Publication Date: 2026-11-17SOUTHWEST JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611601999.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing traffic simulation technologies struggle to accurately predict and respond to the complex intentions of human drivers, leading to conservative decision-making by autonomous vehicles in mixed human-machine environments. This makes them ill-suited to complex traffic conditions and increases the risk of accidents.

Method used

We employ a deep inverse reinforcement learning approach based on maximum entropy. By constructing a deep neural network reward function and Hessian matrix, we quantify the intensity of vehicle interactions, perform soft-bucket clustering analysis, extract heterogeneous driving features of individual vehicles, and fine-tune network parameters to achieve lightweight simulation of driver behavior.

Benefits of technology

It accurately identifies high-conflict areas in the road network, overcomes the limitations of traditional methods in simplifying nonlinear interactions, and enables safety verification and risk prediction in complex human-machine mixed environments. It avoids cumbersome online training and improves the accuracy and real-time response capability of the simulation system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116871A_ABST
    Figure CN123116871A_ABST
Patent Text Reader

Abstract

This invention provides a driving behavior simulation method and apparatus based on inverse reinforcement learning, relating to the field of traffic simulation technology. The method includes: modeling driver behavior of a target vehicle using maximum entropy deep inverse reinforcement learning, constructing a deep neural network reward function; constructing a Hessian matrix from real human driver trajectory data, quantifying the interaction intensity between the target vehicle and neighboring vehicles using the Hessian matrix, and constructing an interaction intensity space; performing soft-bucket clustering analysis on the interaction intensity space and extracting heterogeneous driving features of individual vehicles; fine-tuning the parameters of the deep neural network using vehicle samples with heterogeneous driving features from different ranges to obtain fine-tuned network parameters; and performing lightweight simulation of driver-similar trajectories using the maximum entropy deep inverse reinforcement learning method based on the fine-tuned network parameters. This invention solves the problem of simulation systems' difficulty in accurately predicting and responding to the complex intentions of human drivers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of traffic simulation technology, and more specifically, to a driving behavior simulation method and apparatus based on inverse reinforcement learning. Background Technology

[0002] In existing traffic simulation technologies, as the use of autonomous vehicles (AVs) gradually becomes a reality, AVs still need to share roads with human-driven vehicles. They must ensure stability and reliability in complex traffic environments where humans and machines coexist. However, existing research often focuses too much on the decision-making and control algorithms of AVs themselves, while neglecting the nonlinearity and uncertainty of human driving behavior. In-depth analysis and modeling of human driving behavior are relatively lacking, which makes it difficult for simulation systems to accurately predict and respond to the complex intentions of human drivers, thereby increasing the risk of accidents in real traffic applications.

[0003] Therefore, there is an urgent need for a driving behavior simulation method and device based on inverse reinforcement learning, which solves the problem that simulation systems have difficulty accurately predicting and responding to the complex intentions of human drivers. Summary of the Invention

[0004] The purpose of this invention is to provide a driving behavior simulation method and apparatus based on inverse reinforcement learning to improve the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0005] Firstly, this application provides a driving behavior simulation method based on inverse reinforcement learning, including:

[0006] Acquire the target vehicle;

[0007] The driver behavior of the target vehicle is modeled based on maximum entropy deep inverse reinforcement learning, and a deep neural network reward function is constructed.

[0008] Based on the deep neural network reward function, a Hessian matrix is ​​constructed from the real trajectory data of human drivers. The interaction intensity between the target vehicle and its neighboring vehicles is quantified by the Hessian matrix, and an interaction intensity space is constructed.

[0009] The interaction intensity space is subjected to soft bucket clustering analysis and the heterogeneous driving features of individual vehicles are extracted. The parameters of the deep neural network are fine-tuned by vehicle samples with heterogeneous driving features of individual vehicles in different ranges to obtain the fine-tuned network parameters.

[0010] The maximum entropy deep inverse reinforcement learning method based on the fine-tuned network parameters is used to perform lightweight simulation of driver-similar trajectories for the trajectory probability distribution model.

[0011] Secondly, this application also provides a driving behavior simulation device based on inverse reinforcement learning, comprising:

[0012] The acquisition module is used to acquire the target vehicle;

[0013] The modeling module is used to model the driver behavior of the target vehicle based on maximum entropy deep inverse reinforcement learning and construct a deep neural network reward function.

[0014] The matrix module is used to construct a Hessian matrix based on the reward function of the deep neural network on the real trajectory data of human drivers, and to quantify the interaction intensity between the target vehicle and its neighboring vehicles through the Hessian matrix, thereby constructing an interaction intensity space.

[0015] The analysis module is used to perform soft bucket clustering analysis on the interaction intensity space and extract vehicle individual heterogeneous driving features. The parameters of the deep neural network are fine-tuned by vehicle samples with vehicle individual heterogeneous driving features in different ranges to obtain the fine-tuned network parameters.

[0016] The simulation module is used to perform lightweight simulation of driver-like trajectories on the trajectory probability distribution model based on the maximum entropy deep inverse reinforcement learning method under the fine-tuned network parameters.

[0017] The beneficial effects of this invention are as follows:

[0018] This invention constructs a driver behavior model and generates similar trajectories through maximum entropy deep inverse reinforcement learning, filling the gap in microscopic simulation decision modeling. The simulated vehicle behavior is no longer mechanical but possesses the habits of real drivers. Furthermore, Hessian matrix is ​​used to quantify the dynamic interaction intensity between vehicles and construct an interaction space, accurately identifying high-conflict areas in the road network, overcoming the limitations of traditional methods in simplifying nonlinear interactions. Based on this, soft-bucket clustering extracts heterogeneous features of different drivers and fine-tunes network parameters, achieving a leap from common traffic flow to individual driving behavior. Finally, the lightweight simulation with fine-tuned network parameters avoids cumbersome online training during simulation runtime, directly using the fine-tuned network parameters for inference, and conducting simulation tests in complex human-machine mixed traffic environments for safety verification and risk prediction. In summary, this invention solves the problem of simulation systems' difficulty in accurately predicting and responding to the complex intentions of human drivers.

[0019] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the driving behavior simulation method based on inverse reinforcement learning as described in this embodiment of the invention;

[0022] Figure 2 This is a schematic diagram of the basic trajectory set in the basic trajectory combination described in the embodiments of the present invention;

[0023] Figure 3 This is a schematic diagram of the deep neural network reward function described in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the driving behavior simulation device based on inverse reinforcement learning as described in an embodiment of the present invention.

[0025] The diagram is labeled as follows: 800, Driving behavior simulation device based on inverse reinforcement learning; 801, Processor; 802, Memory; 803, Multimedia component; 804, I / O interface; 805, Communication component. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] In traditional traffic simulation technology, especially for autonomous vehicles in complex mixed traffic flow environments such as highways, autonomous vehicles need to coexist with human-driven vehicles that are dynamic, random, heterogeneous, and complex. However, traditional traffic simulation models cannot accurately reproduce the personalized behavior of human drivers, causing autonomous vehicles to make conservative decisions in mixed traffic flow, making it difficult to adapt to complex environments and lacking realism. At the same time, although game theory models that consider the heterogeneity of interaction need to solve Nash equilibrium online iteratively, the computational cost of online iterative solution is extremely high in multi-agent interaction scenarios. This not only makes it difficult to respond in real time, but also easily causes transient instability in the simulation, making it impossible to scale to large-scale traffic scenarios with thousands of vehicles. As a result, existing simulation systems are unable to accurately predict and respond to the complex intentions of human drivers while ensuring real-time response.

[0029] Example 1: This embodiment provides a driving behavior simulation method based on inverse reinforcement learning.

[0030] See Figure 1 The figure shows that the method includes steps S1 to S5, including: S1: Obtain the target vehicle; S2: Model the driver behavior of the target vehicle based on maximum entropy deep inverse reinforcement learning, and construct a deep neural network reward function; In this step, the driver behavior of the target vehicle is modeled based on Maximum Entropy Deep Inverse Reinforcement Learning (MEDIRL). The driver behavior is decomposed into lateral position (following, changing lanes left, changing lanes right) and longitudinal position (constant speed, acceleration, deceleration) action dimensions. The lateral and longitudinal position action dimensions and the vehicle's running state are input into a deep neural network containing two hidden layers (RELU activation function) and one output layer (Sigmoid activation function). Through the mechanism that the trajectory reward follows the Boltzmann distribution and the trajectory probability is proportional to the reward exponent, the deep neural network reward function is finally trained.

[0031] To clarify the specific method for obtaining similar trajectories of drivers, step S2 includes S21 to S24, specifically: S21: Based on maximum entropy deep inverse reinforcement learning, the trajectory state of the target vehicle is collected to obtain a candidate trajectory set; To clarify the specific method for obtaining the candidate trajectory set, step S21 includes S211 to S213, specifically: S211: Based on maximum entropy deep inverse reinforcement learning, the longitudinal and lateral positions of the target vehicle are sampled to obtain the collected trajectory state; In this step, the longitudinal and lateral positions of the target vehicle are sampled based on maximum entropy deep inverse reinforcement learning. The sampling is set within a 5-second time domain with a sampling interval of 0.1 seconds, resulting in a collected trajectory state. This trajectory state includes the longitudinal and lateral position coordinates of the target vehicle, longitudinal and lateral velocities, braking values, and longitudinal and lateral acceleration and steering angle strategy parameters. Simultaneously, considering the limited field of view of human drivers and to avoid state input dimension explosion, the distance to the target vehicle is... The trajectory status of the three nearest neighboring vehicles is collected, including longitudinal and lateral position coordinates, longitudinal and lateral velocities, as well as longitudinal acceleration and steering angle, as strategy parameters. This data is then used to determine the distance to the target vehicle. Neighboring vehicles beyond the three nearest cars Its state information is processed using pooling, and the unified format of the pooled state information is as follows: (in, , The target vehicles with neighbor's vehicle The longitudinal and lateral position distances are calculated, and the pooled results are added to the acquisition trajectory status.

[0032] S212: Construct a basic trajectory combination based on the collected trajectory status and the speed of the target vehicle;

[0033] like Figure 2 As shown in Table 1, based on the collected trajectory status and the speed of the target vehicle, the lateral actions (following, left lane change, right lane change) are coupled with the longitudinal actions (constant speed, acceleration, deceleration) to form constant speed following, acceleration following, deceleration following, constant speed left lane change, etc., and nine basic trajectory combinations are constructed to cover the core driving intentions of the driver in the highway scenario. The basic trajectory combinations fully consider the holistic nature of the driver's behavior, combining lateral position selection with longitudinal speed decision-making to ensure that the trajectory combinations realistically simulate the high-level driving intentions of human drivers.

[0034] Table 1 Basic Trajectory Combinations

[0035] S213: Construct a candidate trajectory set by combining the basic trajectory combination and the kinematic parameters of neighboring vehicles, and generate multiple initial state trajectories by combining maximum entropy inverse reinforcement learning.

[0036] In this step, the core principle of Maximum Entropy Inverse Reinforcement Learning (MEIRL) is that the probability distribution that matches the real demonstration data has the maximum entropy is used to generate multiple trajectories that are consistent with the initial state of the target vehicle, and a candidate trajectory set is constructed. By reasonably selecting the three nearest vehicles as the number of neighboring vehicles, overfitting is avoided.

[0037] S22: Input the candidate trajectory set and the actual driving trajectory into the deep neural network to calculate the reward for each trajectory and obtain the trajectory reward value;

[0038] To clarify the specific method for obtaining trajectory reward values, step S22 includes S221 to S223, which are as follows:

[0039] S221: Input the candidate trajectory set and the actual driving trajectory into a deep neural network for forward propagation inference, and construct a nonlinear reward function;

[0040] In this step, the candidate trajectory set and the actual driving trajectory are input into a deep neural network with a pre-defined structure for forward propagation inference. The deep neural network contains two hidden layers and one output layer. Hidden layer 1 has 10 neurons, hidden layer 2 has 8 neurons, both using the ReLU activation function. The output layer has one neuron using the Sigmoid activation function. After inputting the candidate trajectory set and the actual driving trajectory into the deep neural network, a nonlinear reward function is constructed by weighted summation and activation function operations of neurons in each layer during forward propagation. This constructs a function to fit the complex relationships in the driving scenario, comprehensively capturing various influencing factors in human driving behavior. The linear reward function expression used in traditional inverse reinforcement learning is:

[0041] (1);

[0042] Equation (1) above is: To determine the parameters of the reward function Lower trajectory The reward These are manually extracted trajectory feature vectors. These are the parameters for the reward function.

[0043] Wherein, the determination of reward function parameters Lower trajectory The reward is a non-linear reward function. Linear reward functions are easy to calculate, but they require manually designed features, which cannot fully represent the factors involved in human driving behavior. Linear structures also have limitations, failing to accurately simulate the impact of various complex situations on the reward in human driving scenarios. Therefore, a deep neural network is designed as the reward function to avoid manual design of trajectory features, allowing the network to extract them automatically and fit the trajectory reward function in a non-linear form.

[0044] The structure of the deep neural network is as follows:

[0045] Table 2 Deep Neural Network Structure

[0046]

[0047] S222: Based on the nonlinear reward function under the hidden layer, automatic feature extraction and nonlinear mapping are performed on the candidate trajectory set and the real driving trajectory to obtain a high-level trajectory feature representation;

[0048] In this step, the candidate trajectory set and the real driving trajectory are transformed and combined layer by layer based on the nonlinear reward function under the hidden layer. High-dimensional key features such as relative motion patterns and speed change trends are automatically mined and abstracted to obtain a high-level trajectory feature representation.

[0049] S223: Based on the high-level trajectory feature representation, the candidate trajectory set and the actual driving trajectory are calculated in the output layer of the deep neural network using a sigmoid activation function to obtain the trajectory reward value.

[0050] In this step, based on the high-level trajectory feature representation, the output range of the output layer is limited to between 0 and 1. The calculated reward value is controllable and the gradient is smooth, avoiding jumps in output. After the high-level trajectory feature representation is input into the output layer, it is processed by the sigmoid activation function to obtain the degree of fit of each candidate trajectory behavior.

[0051] The hyperparameters during training are set as follows: learning rate of 0.000125, number of iterations of 2000, and the Adam optimization algorithm is used to improve the efficiency of deep neural network training. The specific parameters in the Adam calculation formula are as follows: ; ; ; .

[0052] Table 3 Hyperparameters for Algorithm Training

[0053]

[0054] S23: Based on the principle of maximum entropy, the trajectory reward value is transformed into a probability distribution to generate a trajectory probability distribution model; To clarify the specific method for obtaining the trajectory probability distribution, step S23 includes S231 to S234, which are as follows:

[0055] S231: Based on the principle of maximum entropy, the vehicle trajectory probability is obtained by sampling generated trajectories with the same initial state in the continuous state space and combining multiple preset discrete subspace trajectories to approximate the continuous policy space.

[0056] In this step, based on the core idea of ​​the maximum entropy principle, multiple candidate trajectories are generated by sampling from the current strategy with the same initial state as the real trajectory data of human drivers in the continuous state space. The candidate trajectories include multiple types of driving behavior trends that may occur in the initial state. Then, by statistically analyzing the state feature distribution of multiple sampled trajectories in multiple preset discrete subspaces, the continuous strategy space is approximated using the maximum entropy principle, thereby obtaining the vehicle trajectory probability under the approximate generation strategy.

[0057] The expression for the probability of the vehicle trajectory is:

[0058] (2);

[0059] Equation (2) above To determine the parameters of the reward function Vehicle trajectory The probability, It is an exponential function. To determine the parameters of the reward function Lower trajectory The reward This is the partition function.

[0060] S232: Normalize the finite trajectories of the candidate trajectory set according to the vehicle trajectory probability to obtain the partition function;

[0061] In this step, since the theoretical integral form of the partition function in continuous space is difficult to solve directly and has extremely high computational complexity, the finite trajectories of the candidate trajectory set are normalized by the vehicle trajectory probability, which effectively avoids the probability distortion problem caused by the difference in the number of trajectories or the fluctuation of the reward value.

[0062] The expression for the partition function is:

[0063] (3);

[0064] In the above formula (3), For the partition function, To theoretically sum over the entire trajectory space, It is an exponential function. To determine the parameters of the reward function Lower trajectory The reward Let be a infinitesimal element of the trajectory space.

[0065] S233: Based on the partition function and the vehicle trajectory probability, a trajectory probability approximation model is constructed to obtain the trajectory probability approximation model;

[0066] This step requires the learned strategy to match the expected value of features with the real data. The constraints are handled using the Lagrange multiplier method. The vehicle trajectory probability is proportional to the exponential function of the total reward and is normalized by the partition function. Substituting the partition function into the analytical expression, an approximate trajectory probability model is constructed; the expression for the approximate trajectory probability model is:

[0067] (4);

[0068] In the above formula (4), For the partition function, For the reward function parameters, For the candidate trajectory set Each trajectory in Summation, For the first A trajectory, It is an exponential function. To determine the parameters of the reward function Lower trajectory The reward.

[0069] S234: Based on the trajectory probability approximation model and the vehicle trajectory probability, construct a trajectory probability distribution model by maximizing the probability proportion of the real human trajectory;

[0070] This step combines the approximation result of the partition function of the trajectory probability approximation model with the expression of the vehicle trajectory probability to derive the maximum proportion of the probability of the real human trajectory, thus obtaining the trajectory probability distribution model. The trajectory probability distribution model optimizes the reward function parameters to maximize the probability of the real human trajectory among all candidate trajectories, thereby accurately imitating human driving behavior.

[0071] The expression for the trajectory probability distribution model is:

[0072] (5);

[0073] In the above formula (5), To determine the parameters of the reward function Vehicle trajectory The probability, It is an exponential function. To determine the parameters of the reward function Lower trajectory The reward For the candidate trajectory set Each trajectory in Summation, For the first A trajectory, To determine the parameters of the reward function Lower trajectory The reward.

[0074] S24: Based on the log-likelihood objective function, the gradient of the reward function parameters is calculated in the backpropagation of the deep neural network through gradient ascent and the network weights are iteratively updated until convergence, thus obtaining the deep neural network reward function;

[0075] In this step, a deep neural network reward function is trained based on a log-likelihood objective function. The goal is to maximize this objective function. The gradient of the objective function with respect to the network parameters is calculated and optimized. During training, the gradient is solved by backpropagation through the deep neural network, and the gradient ascent strategy is used to calculate the gradient value of the objective function with respect to the network parameters. The Adam optimizer is used to iteratively update the weights and bias parameters of the neural network until the objective function converges. Finally, a deep neural network reward function for evaluating the quality of the trajectory is obtained.

[0076] The core of gradient calculation is the difference between the gradient of the reward function parameters of the expert trajectory and the gradient of the expected reward function parameters of all trajectories in the candidate trajectory set. The corresponding log-likelihood objective function expression is:

[0077] (6);

[0078] In the above formula (6), Let log-likelihood be the objective function. To determine the parameters of the reward function Lower trajectory The reward It is the natural logarithm function. For the candidate trajectory set Each trajectory in Summation, It is an exponential function. To determine the parameters of the reward function Lower trajectory The reward.

[0079] The formula for calculating the gradient of the objective function derived from the log-likelihood objective function is as follows:

[0080] (7);

[0081] In the above formula (7), Let be the gradient of the objective function with respect to the parameter θ. For partial derivatives, To determine the parameters of the reward function Lower trajectory The reward For the candidate trajectory set Each trajectory in Summation, It is an exponential function. To determine the parameters of the reward function Lower trajectory The reward To determine the parameters of the reward function Vehicle trajectory The probability of;

[0082] Wherein, the determination of reward function parameters Lower trajectory The trajectory in the reward For expert trajectory.

[0083] The core step in training the reward function of a deep neural network is backpropagation, and its corresponding weight gradient update formula is:

[0084] (8);

[0085] Equation (8) above is: Let x be the gradient of the weights in the x-th layer of the neural network. For expert trajectory The partial derivative of the corresponding reward function with respect to the linear output of the x-th layer of the neural network before activation function nonlinearization. Let x be the output activation value of the (x-1)th layer of the neural network. The number of candidate trajectories to be sampled. For the candidate trajectory set Each trajectory in Summation, To determine the parameters of the reward function Vehicle trajectory The probability, For trajectory The partial derivative of the corresponding reward function with respect to the linear output of the x-th layer of the neural network without activation function nonlinearization;

[0086] The bias gradient update formula is as follows:

[0087] (9);

[0088] Equation (9) above is: For the neural network The gradient of layer bias, For expert trajectory The partial derivative of the corresponding reward function with respect to the linear output of the x-th layer of the neural network before activation function nonlinearization. For the candidate trajectory set Each trajectory in Summation, To determine the parameters of the reward function Vehicle trajectory The probability, For trajectory The partial derivative of the corresponding reward function with respect to the linear output of the x-th layer of the neural network before activation function nonlinearization. This represents the number of candidate trajectories to be sampled.

[0089] S3: Construct a Hessian matrix based on the deep neural network reward function for real trajectory data of human drivers, quantify the interaction intensity between the target vehicle and neighboring vehicles through the Hessian matrix, and construct an interaction intensity space;

[0090] To clarify the specific method for obtaining the interaction intensity space, step S3 includes S31 to S34, specifically:

[0091] S31: Based on the deep neural network reward function, accumulate rewards at the end of the trajectory of the real trajectory data of the human driver to obtain the utility function;

[0092] like Figure 3 As shown, the deep neural network reward function uses the optimal nonlinear reward function (constructed by a deep neural network with two hidden layers and one output layer, the hidden layers using the ReLU activation function and the output layer using the Sigmoid activation function) after training and convergence. This reward function is used to obtain the driver's similar trajectory at time step [time value missing]. status With action The corresponding instantaneous reward value; summing the instantaneous reward values ​​in the time domain to obtain a utility function that reflects the overall benefit of the trajectory, the utility function being directly related to the target vehicle's driving decisions and environmental feedback.

[0093] The expression for the ReLU activation function is:

[0094] (10);

[0095] In the above formula (10), The output value of the ReLU activation function. To find the maximum value function, The input value for the neuron;

[0096] Specifically, the maximum value function is defined as follows: when the input x>0, the output is equal to x; when the input x≤0, the output is 0.

[0097] The expression for the Sigmoid activation function is:

[0098] (11);

[0099] In the above formula (11), The output value of the Sigmoid activation function. It is an exponential function. This is the input value for the neuron.

[0100] S32: Based on the utility function, solve for the second-order partial derivatives of the control inputs of the target vehicle and neighboring vehicles to form a Hessian matrix block;

[0101] In this step, the control input includes the target vehicle. Lateral and longitudinal acceleration and steering angle, as well as neighboring vehicles The longitudinal acceleration and steering angle, along with the Hessian matrix, are used to accurately reflect the target vehicle. The utility gradient of neighboring vehicles The degree of dependence on action input, in the context of... In a road environment with vehicles (including the target vehicle), the utility function at the trajectory endpoint is solved for the target vehicle. With neighboring vehicles The second-order partial derivatives of the control inputs form a dimension-matched Hessian matrix block. Each element in the matrix block corresponds to a set of interaction correlations between the control inputs of the vehicles. The expression for the Hessian matrix is ​​as follows:

[0102] (12);

[0103] In the above formula (12), For the reward function parameters With trajectory Next target vehicle The Hessian matrix, For the target vehicle control input The second-order partial derivative operator, For the end-of-trajectory effect, The Hessian matrix is ​​a two-dimensional matrix with 2 rows and 2 columns.

[0104] S33: Extract the interaction Hessian block norm of the target vehicle and each neighboring vehicle from the Hessian matrix block to obtain the quantized interaction strength scalar.

[0105] In this step, the target vehicle is first separated from the constructed Hessian matrix block. With a single neighbor vehicle The corresponding interactive Hessian sub-blocks (the dimensions of the interactive Hessian sub-blocks and the Hessian matrix) (Consistent), focusing on the second-order correlation of control inputs between the target vehicle and specific neighboring vehicles, then using the Frobenius norm to perform scalar transformation on the interactive Hessian sub-block, and then using square root operations to transform the two-dimensional matrix form of interactive information into a single numerical scalar; the magnitude of the norm is positively correlated with the interaction strength between vehicles, and a quantized interaction strength scalar is calculated, which is used to intuitively quantify the strength of the mutual influence between the target vehicle and its corresponding neighboring vehicles.

[0106] The expression for the Frobenius norm is:

[0107] (13);

[0108] In the above formula (13), For the target vehicle a neighbor's vehicle The Frobenius norm of interactive Hessian sub-blocks For the target vehicle The corresponding interactive Hessian sub-block, It is the arithmetic square root. For the target vehicle with neighbor's vehicle The first interactive Hessian sub-block Line number The column contains the calculation results of the second-order partial derivatives between the two sets of vehicle control inputs. This involves summing the squares of all elements in a 2×2 matrix.

[0109] S34: Construct an interaction intensity space based on the quantized interaction intensity scalar of the target vehicle and all the neighboring vehicles in the same time period.

[0110] In this step, the selection of neighboring vehicles follows the principle of "limited field of view," that is, only dynamic traffic participants within the perception range of the target vehicle are considered, which conforms to the modeling logic of local dependence in multi-agent interactions. The three core neighboring vehicles closest to the target vehicle are prioritized for inclusion. Meanwhile, for neighboring vehicles further away within the perception range, the interaction information is integrated and calculated using the mean pooling method. Subsequently, the quantized interaction intensity scalar between the target vehicle and each neighboring vehicle in the 5-second time domain is collected. Using the target vehicle-neighboring vehicle as the basic interaction pair and the corresponding quantized scalar as the interaction intensity value, a continuous data set of all effective interaction relationships is constructed. This continuous data set is then used as the interaction intensity space.

[0111] S4: Perform soft-bucket clustering analysis on the interaction intensity space and extract vehicle individual heterogeneous driving features. Fine-tune the parameters of the deep neural network using vehicle samples with vehicle individual heterogeneous driving features in different ranges to obtain fine-tuned network parameters.

[0112] To clarify the specific method for obtaining the fine-tuned network parameters, step S4 includes S41 to S44, specifically:

[0113] S41: The interaction intensity values ​​in the interaction intensity space are serialized, and the number of clustering buckets of the interaction intensity sequence is determined by kernel density estimation to obtain the bucketing result;

[0114] In this step, the interaction intensity values ​​in the interaction intensity space are serialized according to vehicle pairing relationships to form an ordered interaction intensity sequence. Then, a logarithmic transformation is performed on all interaction intensity values ​​in the interaction intensity sequence, and the kernel density estimation method is used to analyze the distribution density characteristics of the logarithmically transformed data. Based on the density peaks and valleys, the optimal number of cluster buckets is automatically determined. At the same time, the core of each cluster bucket is calculated. With bandwidth The final result is a bucketing result containing the number of buckets, the bucket center, and the bandwidth. The bucketing result is used to adaptively match the continuous distribution characteristics of the interaction intensity, avoiding the subjective error of manual bucketing.

[0115] S42: Based on the bucketing results, perform soft bucketing clustering analysis on the interaction intensity space to obtain soft bucketing analysis results;

[0116] In this step, soft-bucket clustering differs from the rigid "either / or" partitioning rule of hard-bucket clustering. It achieves flexible classification by calculating the soft weights of each interaction intensity sample to each bucket. The Gaussian kernel function is used to measure the similarity between the sample and the bucket center. The larger the weight value, the higher the fit between the sample and the corresponding bucket. Finally, each interaction intensity sample will obtain a set of weight distributions corresponding to all buckets, forming the soft-bucket analysis results. This achieves both the classification and aggregation of interaction intensity and preserves the correlation between samples in different categories.

[0117] The expression for the soft-assignment weight is:

[0118] (14);

[0119] In the above formula (14), For the first The interaction strength sample corresponds to the first... The soft weighting of each cluster bucket It is a natural exponential function. This represents the logarithmic value of the interaction strength for the current sample. For the first The center of the bucket, For the first The bandwidth of each cluster bucket For bandwidth, For all Summation is performed on each cluster bucket. As a dummy variable for summation, This represents the total number of cluster buckets.

[0120] S43: Correct the latent variable perturbation term of the reward function in the driver's similar trajectory according to the preset random low-dimensional Gaussian latent variable, and extract the individual heterogeneous driving features of the vehicle.

[0121] In this step, the random low-dimensional Gaussian hidden variable is (Two-dimensional standard normal distribution) For hidden variables A column vector, with variances of independent and arbitrary directions for each sample, unbiased towards acceleration, left lane changes, or other specific driving directions, and latent variables are... 3D column vectors and fixed dimensions Neighbor pooling features Perform the inner product operation to obtain the scalar perturbation term. ( The scalar perturbation term is used to adjust the degree of heterogeneity influence. The corrected reward function output is associated with the trajectory behavior to accurately extract the individual heterogeneous driving characteristics of the vehicle.

[0122] The revised reward function is:

[0123] (15);

[0124] In the above formula (15), The modified reward function is the result of random latent variables. For drivers with similar trajectories, For the first Deep network parameters shared by soft-bucket clustering For the first Random low-dimensional Gaussian latent variables corresponding to each vehicle interaction sample For the first The basic reward function shared by all buckets outputs the following: For scalar disturbance terms, The disturbance coefficient is... For random latent variables The transpose operation, For trajectory The corresponding neighbor pooling features.

[0125] S44: Based on the soft bucket analysis results, select vehicle samples with heterogeneous driving characteristics of individual vehicles within different interaction intensity ranges to fine-tune the parameters of the deep neural network, and obtain the fine-tuned network parameters.

[0126] In this step, the soft bucket analysis results divide the interaction intensity into multiple continuous ranges. Each range corresponds to a group of vehicle samples with similar interaction characteristics, and each sample extracts individual heterogeneous driving features. Representative samples from each interaction intensity range are selected to form a fine-tuning dataset, and the fine-tuning network parameters are constructed.

[0127] S5: Based on the maximum entropy deep inverse reinforcement learning method under the fine-tuned network parameters, perform lightweight simulation of driver similar trajectories for the trajectory probability distribution model.

[0128] In this step, the simulation environment is custom-developed based on the underlying logic of the open-source highway environment highway_env in Python. The NGSIM dataset is used as the simulation object. The deep neural network with fine-tuned network parameters (containing two ReLU activation function hidden layers and one Sigmoid activation function output layer, with fine-tuned weights and biases adapted to different interaction intensity scenarios and vehicle individual heterogeneity) is deeply integrated with the core logic of maximum entropy deep inverse reinforcement learning with fine-tuned network parameters (trajectory probability calculation, objective function optimization, gradient update rules). At the same time, the interaction intensity quantification logic based on Hessian matrix and the soft bucket clustering mechanism are integrated. The driver-similar trajectory is generated through the trajectory probability distribution model, and lightweight simulation of driving behavior is achieved through the driver-similar trajectory.

[0129] Example 2:

[0130] This embodiment provides a driving behavior simulation device based on inverse reinforcement learning, the device comprising:

[0131] The acquisition module is used to acquire the target vehicle;

[0132] The modeling module is used to model the driver behavior of the target vehicle based on maximum entropy deep inverse reinforcement learning and construct a deep neural network reward function.

[0133] To clarify the specific methods for obtaining the modeling module, the following are included:

[0134] The acquisition unit is used to acquire the trajectory state of the target vehicle based on maximum entropy deep inverse reinforcement learning to obtain a candidate trajectory set.

[0135] The computing unit is used to input the candidate trajectory set and the real driving trajectory into the deep neural network to calculate the reward for each trajectory and obtain the trajectory reward value;

[0136] To clarify the specific methods for obtaining computing units, the following are included:

[0137] The inference subunit is used to input the candidate trajectory set and the actual driving trajectory into the deep neural network for forward propagation inference and to construct a nonlinear reward function.

[0138] The extraction subunit is used to automatically extract and nonlinearly map the candidate trajectory set and the real driving trajectory based on the nonlinear reward function under the hidden layer to obtain a high-level trajectory feature representation.

[0139] The computational subunit is used to calculate the trajectory reward value by using the sigmoid activation function on the candidate trajectory set and the real driving trajectory in the output layer of the deep neural network based on the high-level trajectory feature representation.

[0140] A probability unit is used to transform the trajectory reward value into a probability distribution based on the maximum entropy principle, thereby generating a trajectory probability distribution model.

[0141] To clarify the specific methods for obtaining probability units, the following are included:

[0142] The sampling subunit is used to sample generated trajectories with the same initial state in a continuous state space based on the maximum entropy principle, and approximate the continuous policy space by combining multiple preset discrete subspace trajectories to obtain the vehicle trajectory probability.

[0143] The normalization processing subunit is used to normalize the finite trajectories of the candidate trajectory set according to the vehicle trajectory probability to obtain the partition function;

[0144] An approximation construction subunit is used to approximate the trajectory probability based on the partition function and the vehicle trajectory probability, thereby obtaining an approximate trajectory probability model.

[0145] The probability construction subunit is used to construct a trajectory probability distribution model by maximizing the probability proportion of the actual human trajectory based on the trajectory probability approximation model and the vehicle trajectory probability.

[0146] The iterative unit is used to calculate the gradient of the reward function parameters in the backpropagation of the deep neural network based on the log-likelihood objective function, and iteratively update the network weights until convergence, thereby obtaining the deep neural network reward function.

[0147] The matrix module is used to construct a Hessian matrix based on the reward function of the deep neural network on the real trajectory data of human drivers, and to quantify the interaction intensity between the target vehicle and its neighboring vehicles through the Hessian matrix, thereby constructing an interaction intensity space.

[0148] The analysis module is used to perform soft bucket clustering analysis on the interaction intensity space and extract vehicle individual heterogeneous driving features. The parameters of the deep neural network are fine-tuned by vehicle samples with vehicle individual heterogeneous driving features in different ranges to obtain the fine-tuned network parameters.

[0149] To clarify the specific methods for obtaining the analysis module, the following are included:

[0150] Clustering unit is used to serialize the interaction intensity values ​​in the interaction intensity space, and to determine the number of clustering buckets for the interaction intensity sequence by kernel density estimation to obtain the bucketing result;

[0151] The soft bucketing unit is used to perform soft bucketing clustering analysis on the interaction intensity space based on the bucketing results, and obtain soft bucketing analysis results.

[0152] The correction unit is used to correct the latent variable perturbation term of the reward function in the driver's similar trajectory according to the preset random low-dimensional Gaussian latent variable, and extract the individual heterogeneous driving features of the vehicle.

[0153] The fine-tuning unit is used to select vehicle samples with heterogeneous driving characteristics of individual vehicles within different interaction intensity ranges based on the soft bucket analysis results to fine-tune the parameters of the deep neural network, thereby obtaining the fine-tuned network parameters.

[0154] The simulation module is used to perform lightweight simulation of driver-like trajectories on the trajectory probability distribution model based on the maximum entropy deep inverse reinforcement learning method under the fine-tuned network parameters.

[0155] It should be noted that the specific manner in which each module performs its operation in the apparatus described in the above embodiments has been described in detail in the embodiments of the method, and will not be elaborated here.

[0156] Example 3:

[0157] Corresponding to the above method embodiments, this embodiment also provides a driving behavior simulation device based on inverse reinforcement learning. The driving behavior simulation device based on inverse reinforcement learning described below and the driving behavior simulation method based on inverse reinforcement learning described above can be referred to each other.

[0158] Figure 4 This is a block diagram illustrating a driving behavior simulation device 800 based on inverse reinforcement learning, according to an exemplary embodiment. Figure 4 As shown, the driving behavior simulation device 800 based on inverse reinforcement learning may include: a processor 801 and a memory 802. The driving behavior simulation device 800 based on inverse reinforcement learning may also include one or more of the following: a multimedia component 803, an I / O interface 804, and a communication component 805.

[0159] The processor 801 controls the overall operation of the inverse reinforcement learning-based driving behavior simulation device 800 to complete all or part of the steps in the aforementioned inverse reinforcement learning-based driving behavior simulation method. The memory 802 stores various types of data to support the operation of the inverse reinforcement learning-based driving behavior simulation device 800. This data may include, for example, instructions for any application or method operating on the inverse reinforcement learning-based driving behavior simulation device 800, as well as application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 802 or transmitted via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the inverse reinforcement learning-based driving behavior simulation device 800 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, and an NFC module.

[0160] In an exemplary embodiment, the driving behavior simulation device 800 based on inverse reinforcement learning may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the driving behavior simulation method based on inverse reinforcement learning described above.

[0161] Example 4:

[0162] Corresponding to the above method embodiments, this embodiment also provides a medium. The medium described below can be referred to in correspondence with the driving behavior simulation method based on inverse reinforcement learning described above.

[0163] A medium storing a computer program, which, when executed by a processor, implements the steps of the driving behavior simulation method based on inverse reinforcement learning described in the above method embodiments.

[0164] The medium can specifically be any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0165] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A driving behavior simulation method based on inverse reinforcement learning, characterized in that, include: Acquire the target vehicle; The driver behavior of the target vehicle is modeled based on maximum entropy deep inverse reinforcement learning, and a deep neural network reward function is constructed. Based on the deep neural network reward function, a Hessian matrix is ​​constructed from the real trajectory data of human drivers. The interaction intensity between the target vehicle and its neighboring vehicles is quantified by the Hessian matrix, and an interaction intensity space is constructed. The interaction intensity space is subjected to soft bucket clustering analysis and the heterogeneous driving features of individual vehicles are extracted. The parameters of the deep neural network are fine-tuned by vehicle samples with heterogeneous driving features of individual vehicles in different ranges to obtain the fine-tuned network parameters. The maximum entropy deep inverse reinforcement learning method based on the fine-tuned network parameters is used to perform lightweight simulation of driver-similar trajectories for the trajectory probability distribution model.

2. The driving behavior simulation method based on inverse reinforcement learning according to claim 1, characterized in that, The driver behavior of the target vehicle is modeled based on maximum entropy deep inverse reinforcement learning, and a deep neural network reward function is constructed, including: The trajectory state of the target vehicle is collected based on maximum entropy deep inverse reinforcement learning to obtain a candidate trajectory set; The candidate trajectory set and the actual driving trajectory are input into a deep neural network to calculate the reward for each trajectory and obtain the trajectory reward value; Based on the principle of maximum entropy, the trajectory reward value is transformed into a probability distribution to generate a trajectory probability distribution model; Based on the log-likelihood objective function, the gradient of the reward function parameters is calculated during the backpropagation of the deep neural network through gradient ascent and the network weights are iteratively updated until convergence, thus obtaining the deep neural network reward function.

3. The driving behavior simulation method based on inverse reinforcement learning according to claim 2, characterized in that, The candidate trajectory set and the actual driving trajectory are input into a deep neural network to calculate the reward for each trajectory, resulting in a trajectory reward value, including: The candidate trajectory set and the actual driving trajectory are input into a deep neural network for forward propagation inference to construct a nonlinear reward function. Based on the nonlinear reward function under the hidden layer, automatic feature extraction and nonlinear mapping are performed on the candidate trajectory set and the real driving trajectory to obtain a high-level trajectory feature representation. Based on the high-level trajectory feature representation, the candidate trajectory set and the actual driving trajectory are calculated in the output layer of the deep neural network using a sigmoid activation function to obtain the trajectory reward value.

4. The driving behavior simulation method based on inverse reinforcement learning according to claim 2, characterized in that, Based on the principle of maximum entropy, the trajectory reward value is transformed into a probability distribution to generate a trajectory probability distribution model, including: Based on the maximum entropy principle, the vehicle trajectory probability is obtained by sampling generated trajectories with the same initial state in the continuous state space and combining multiple preset discrete subspace trajectories to approximate the continuous policy space. The finite trajectories in the candidate trajectory set are normalized based on the vehicle trajectory probability to obtain the partition function; Based on the partition function and the vehicle trajectory probability, a trajectory probability approximation model is constructed to obtain the trajectory probability approximation model. Based on the trajectory probability approximation model and the vehicle trajectory probability, a trajectory probability distribution model is constructed by maximizing the probability proportion of the actual human trajectory.

5. The driving behavior simulation method based on inverse reinforcement learning according to claim 1, characterized in that, Soft-bucket clustering analysis is performed on the interaction intensity space to extract heterogeneous driving features of individual vehicles. The parameters of the deep neural network are then fine-tuned using vehicle samples with heterogeneous driving features from different ranges, resulting in fine-tuned network parameters, including: The interaction intensity values ​​in the interaction intensity space are serialized, and the number of clustering buckets is determined by kernel density estimation to obtain the bucketing result. Based on the binning results, perform soft binning clustering analysis on the interaction intensity space to obtain soft binning analysis results; The hidden variable perturbation term of the reward function in the driver's similar trajectory is corrected according to the preset random low-dimensional Gaussian hidden variable, and the individual heterogeneous driving characteristics of the vehicle are extracted. Based on the soft bucket analysis results, vehicle samples with heterogeneous driving characteristics of individual vehicles within different interaction intensity ranges are selected to fine-tune the parameters of the deep neural network, thereby obtaining the fine-tuned network parameters.

6. A driving behavior simulation device based on inverse reinforcement learning, characterized in that, include: The acquisition module is used to acquire the target vehicle; The modeling module is used to model the driver behavior of the target vehicle based on maximum entropy deep inverse reinforcement learning and construct a deep neural network reward function. The matrix module is used to construct a Hessian matrix based on the reward function of the deep neural network on the real trajectory data of human drivers, and to quantify the interaction intensity between the target vehicle and its neighboring vehicles through the Hessian matrix, thereby constructing an interaction intensity space. The analysis module is used to perform soft bucket clustering analysis on the interaction intensity space and extract vehicle individual heterogeneous driving features. The parameters of the deep neural network are fine-tuned by vehicle samples with vehicle individual heterogeneous driving features in different ranges to obtain the fine-tuned network parameters. The simulation module is used to perform lightweight simulation of driver-like trajectories on the trajectory probability distribution model based on the maximum entropy deep inverse reinforcement learning method under the fine-tuned network parameters.

7. The driving behavior simulation device based on inverse reinforcement learning according to claim 6, characterized in that, The modeling module includes: The acquisition unit is used to acquire the trajectory state of the target vehicle based on maximum entropy deep inverse reinforcement learning to obtain a candidate trajectory set. The computing unit is used to input the candidate trajectory set and the real driving trajectory into the deep neural network to calculate the reward for each trajectory and obtain the trajectory reward value; A probability unit is used to transform the trajectory reward value into a probability distribution based on the maximum entropy principle, thereby generating a trajectory probability distribution model. The iterative unit is used to calculate the gradient of the reward function parameters in the backpropagation of the deep neural network based on the log-likelihood objective function, and iteratively update the network weights until convergence, thereby obtaining the deep neural network reward function.

8. The driving behavior simulation device based on inverse reinforcement learning according to claim 7, characterized in that, The computing unit includes: The inference subunit is used to input the candidate trajectory set and the actual driving trajectory into the deep neural network for forward propagation inference and to construct a nonlinear reward function. The extraction subunit is used to automatically extract and nonlinearly map the candidate trajectory set and the real driving trajectory based on the nonlinear reward function under the hidden layer to obtain a high-level trajectory feature representation. The computational subunit is used to calculate the trajectory reward value by using the sigmoid activation function on the candidate trajectory set and the real driving trajectory in the output layer of the deep neural network based on the high-level trajectory feature representation.

9. The driving behavior simulation device based on inverse reinforcement learning according to claim 7, characterized in that, The probability unit includes: The sampling subunit is used to sample generated trajectories with the same initial state in a continuous state space based on the maximum entropy principle, and approximate the continuous policy space by combining multiple preset discrete subspace trajectories to obtain the vehicle trajectory probability. The normalization processing subunit is used to normalize the finite trajectories of the candidate trajectory set according to the vehicle trajectory probability to obtain the partition function; An approximation construction subunit is used to approximate the trajectory probability based on the partition function and the vehicle trajectory probability, thereby obtaining an approximate trajectory probability model. The probability construction subunit is used to construct a trajectory probability distribution model by maximizing the probability proportion of the actual human trajectory based on the trajectory probability approximation model and the vehicle trajectory probability.

10. The driving behavior simulation device based on inverse reinforcement learning according to claim 7, characterized in that, The analysis module includes: Clustering unit is used to serialize the interaction intensity values ​​in the interaction intensity space, and to determine the number of clustering buckets for the interaction intensity sequence by kernel density estimation to obtain the bucketing result; The soft bucketing unit is used to perform soft bucketing clustering analysis on the interaction intensity space based on the bucketing results, and obtain soft bucketing analysis results. The correction unit is used to correct the latent variable perturbation term of the reward function in the driver's similar trajectory according to the preset random low-dimensional Gaussian latent variable, and extract the individual heterogeneous driving features of the vehicle. The fine-tuning unit is used to select vehicle samples with heterogeneous driving characteristics of individual vehicles within different interaction intensity ranges based on the soft bucket analysis results to fine-tune the parameters of the deep neural network, thereby obtaining the fine-tuned network parameters.