Multimodal scenario driving behavior modeling method and system based on inverse reinforcement learning

By constructing the inverse reinforcement learning-scene clustering model, the global cost function is optimized to match the driving scenario and model parameters, the problem of linear reward function limitation of the inverse reinforcement learning algorithm in intelligent driving is solved, and multimodality is realized in complex scenarios Driving behavior modeling and strategy output improve the robustness and adaptability of the model.

CN118560529BActive Publication Date: 2024-11-15XIDI SMART DRIVING (CHENGDU) TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411053068.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-15
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

In intelligent driving, the existing inverse reinforcement learning algorithm is difficult to adapt to complex mixed-programming interaction scenarios due to the limitation of linear reward functions, resulting in the generalization ability of the learned strategies in different scenarios, and it is impossible to effectively simulate human drivers. Decision-making process in complex traffic environments.

Method used

By constructing an inverse reinforcement learning-scene clustering model, using expert trajectory as training data, the global cost function is optimized to match driving scenarios and model parameters, the driving behavior modeling in multimodal scenarios is realized, and robust control parameters are output to imitate Expert driving habits.

Benefits of technology

Effectively self-supervised mining of multimodal driving trajectory data, extract interaction features, output global cost functions and multimodal scene adaptive driving strategies, improve the robustness and generalization capabilities of the model, and can be highly adaptable in different scenarios. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118560529B_ABST
    Figure CN118560529B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-modal scenario driving behavior modeling method and system based on inverse reinforcement learning, which relates to the field of intelligent driving technology. The scenarios of training data are classified based on model parameters through a self-supervised comparative learning algorithm, and the scenario labels corresponding to each training data are output. The data with the same scene label are grouped together, and the "inverse reinforcement learning-scenario clustering" process is repeated again until a stable "model parameter-training data" pairing is obtained. The global cost function is trained using the "model parameter-training data" pairing so that it can judge the corresponding scenario according to the state of the vehicle and output the corresponding model parameters. This solves the problem that the current driving behavior modeling method using inverse reinforcement learning in intelligent driving cannot adapt to complex mixed interaction scenarios due to the limitation of linear reward functions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a multi-modal scenario driving behavior modeling method and system based on inverse reinforcement learning. Background Art

[0002] Autonomous driving refers to the technology that enables the car to perceive the environment, plan the path and control the vehicle autonomously. At present, the demand for autonomous vehicles is to accurately simulate human driving behavior to adapt to mixed environments.

[0003] At present, the intelligent driving of unmanned vehicles usually adopts a uniform speed model or an intelligent driving model. The uniform speed model (such as the classic uniform speed following model) and the intelligent driving model (such as the intelligent driving model, IDM - Intelligent Driver Model) are mathematical models used to simulate the behavior of vehicles under different traffic conditions. However, these models are often based on simplified assumptions and may not fully capture the decision-making process and behavioral intentions of human drivers in complex traffic environments.

[0004] At present, there is an application of inverse reinforcement learning in intelligent driving. Inverse reinforcement learning (IRL) is a machine learning method that attempts to infer the goals or preferences of decision makers from observed behaviors. In the field of intelligent driving, IRL can be used to learn driving strategies from the behavior of human drivers. However, the current modeling algorithm that applies inverse reinforcement learning to intelligent driving approximates the linear reward function characteristics, which cannot fully express complex driving scenarios and the preferences of human drivers. In actual driving, the reward function may need to consider multiple factors, which are often nonlinear. In the absence of sufficient prior knowledge, the IRL algorithm may find it difficult to learn effective driving strategies from nonlinear reward functions.

[0005] Driving behaviors may show diversity (i.e., multimodality) across different driving scenarios and drivers. Existing IRL algorithms may have difficulty adapting to this diversity, resulting in limited generalization of learned policies across different scenarios. Control parameters learned in one specific scenario may not be directly applicable to other scenarios, which limits the robustness and practicality of the model.

[0006] Therefore, the current driving behavior modeling method that applies inverse reinforcement learning to intelligent driving cannot adapt to complex mixed interaction scenarios due to the limitations of linear reward functions. Summary of the invention

[0007] Based on the problems raised by the above background technology, the purpose of the present invention is to provide a multimodal scenario driving behavior modeling method and system based on inverse reinforcement learning, which solves the problem that the current driving behavior modeling method using inverse reinforcement learning in intelligent driving cannot adapt to complex mixed interaction scenarios due to the limitation of linear reward functions.

[0008] The present invention is achieved through the following technical solutions:

[0009] The first aspect of the present invention provides a multimodal scenario driving behavior modeling method based on inverse reinforcement learning, comprising the following steps:

[0010] Step S1, establishing an inverse reinforcement learning-scenario clustering model, inputting expert trajectories as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, and obtaining model parameter-training data pairing data;

[0011] Step S2, constructing a global cost function, and optimizing the global cost function using the model parameter-training data pairing data;

[0012] Step S3: Use the optimized global cost function to perform scene judgment, obtain a driving scene, and output model parameters corresponding to the driving scene.

[0013] In the above technical solution, by constructing an inverse reinforcement learning-scenario clustering model, the expert trajectory is used as training data to thrust the reward function from the expert trajectory through inverse reinforcement learning, and then the best agent of the expert trajectory is found from the sampled trajectory through the reward function. The best agent is used as the model parameter, and scene clustering is performed based on the model parameter. That is, it is possible to find the driving scene corresponding to the driving trajectory state under the state of the matched driving trajectory, aggregate them, match the driving trajectory with the driving scene, and obtain model parameter-training data pairing data.

[0014] In the existing inverse reinforcement learning model, the model parameters and feature weight vectors obtained by the inverse reinforcement learning model are directly used to train the optimization cost function, which leads to the neglect of the multimodality of driving behavior in different driving scenarios. In this method, the model parameters are matched with the training data, and the matched model parameter-training data pairing data is used to optimize the global cost function. Its core purpose is to imitate the driving habits of experts as much as possible when outputting the control parameters of the corresponding scene. After completing the training of the global cost function, the feature weight vector can be output for any initial state. According to the feature weight vector, the trajectory with the maximum reward can be selected from the sampled trajectory corresponding to the initial state. This trajectory satisfies the multimodality of driving behavior in different driving scenarios.

[0015] In an optional embodiment, the step S1 of inputting the expert trajectory as training data into the inverse reinforcement learning-scenario clustering model for cyclic training includes the following steps:

[0016] According to the initial state of the expert trajectory, generating a sampling trajectory corresponding to the expert trajectory;

[0017] Mapping the expert trajectory and the sampling trajectory to a feature space through a feature function to obtain a sampling feature;

[0018] The sampling characteristics are solved by using the maximum entropy solution method to obtain model parameters;

[0019] Performing scene clustering based on the model parameters to obtain expert trajectories in driving scenes;

[0020] The model parameters are paired with expert trajectories in a driving scenario to generate model parameter-training data paired data.

[0021] In an optional embodiment, generating a sampling trajectory corresponding to the expert trajectory according to the initial state of the expert trajectory includes the following steps:

[0022] Get in complex interactive environment Track status at the moment ,Will Track state of time length Integrate to generate Expert Tracks ,in, For the The initial state of the expert trajectory;

[0023] According to the initial state, generate the The sampled trajectories of expert trajectories ,in, is the total number of sampled trajectories, and The set of expert trajectories and sampled trajectories is denoted as .

[0024] In an optional embodiment, solving the sampling feature by using the maximum entropy solution method includes:

[0025] First The sampled trajectories of expert trajectories Construct the objective function;

[0026] The objective function is solved using the maximum entropy method to obtain The feature weight vector corresponding to the expert trajectory;

[0027] Wherein, the objective function is constructed as follows:

[0028] ;

[0029] ;

[0030] ;

[0031] In the above formula, represents the characteristic function, Indicates Expert track Sampled trajectories The sampling characteristics of represents the length of the feature vector, Indicates The feature weight vector corresponding to the expert trajectory.

[0032] In an optional embodiment, performing scene clustering based on the model parameters includes the following steps:

[0033] The expert trajectories are classified using a clustering algorithm according to the feature weight vectors corresponding to the expert trajectories, and the expert trajectories with the same labels are classified into the same set to obtain a scene trajectory set;

[0034] The data in the scene trajectory set is solved using the maximum entropy solution method to obtain the feature weight vector corresponding to the scene trajectory set;

[0035] The feature weight vector corresponding to the scene trajectory set is matched with the scene set corresponding to the feature weight vector corresponding to the scene trajectory set to obtain model parameter-training data pairing data.

[0036] In an optional embodiment, optimizing the global cost function using the model parameter-training data pairing data includes: using the initial state of each expert trajectory in the scene trajectory set as input data of the global cost function, and using the feature weight vector corresponding to the scene trajectory set as a label of the global cost function.

[0037] In an optional embodiment, the global cost function includes a loss function, wherein the loss function is expressed as follows:

[0038] ;

[0039] in, represents the loss function, represents the cross entropy function, Represents a set of scene trajectories Middle The feature weight vector for data prediction, Indicates The feature weight vector corresponding to a set of scene trajectories.

[0040] A second aspect of the present invention provides a multimodal scenario driving behavior modeling system based on inverse reinforcement learning, comprising:

[0041] A pairing module, wherein the pairing module is used to establish an inverse reinforcement learning-scenario clustering model, input the expert trajectory as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, and obtain model parameter-training data pairing data;

[0042] A function optimization module, wherein the function optimization module is used to construct a global cost function and optimize the global cost function using the model parameter-training data pairing data;

[0043] The output module is used to use the optimized global cost function to perform scene judgment, obtain a driving scene, and output model parameters corresponding to the driving scene.

[0044] In an optional embodiment, the pairing module includes:

[0045] A sampling submodule, the sampling submodule is used to generate a sampling trajectory corresponding to the expert trajectory according to the initial state of the expert trajectory;

[0046] A mapping submodule, wherein the mapping submodule is used to map the expert trajectory and the sampling trajectory into a feature space through a feature function to obtain a sampling feature;

[0047] A solution submodule, the solution submodule is used to solve the sampling characteristics by using a maximum entropy solution method to obtain model parameters;

[0048] A clustering submodule, the clustering submodule is used to perform scene clustering based on the model parameters to obtain expert trajectories in driving scenes;

[0049] The scenario pairing submodule is used to pair the model parameters with expert trajectories in the driving scenario to generate model parameter-training data pairing data.

[0050] In an optional embodiment, the clustering submodule includes:

[0051] A classification unit, wherein the classification unit is used to classify the expert trajectories using a clustering algorithm according to the feature weight vectors corresponding to the expert trajectories, and classify the expert trajectories with the same label into the same set to obtain a scene trajectory set;

[0052] A weight unit, the weight unit is used to solve the data in the scene trajectory set by using the maximum entropy solution method to obtain a feature weight vector corresponding to the scene trajectory set;

[0053] A pairing unit is used to match the feature weight vector corresponding to the scene trajectory set with the scene set corresponding to the feature weight vector corresponding to the scene trajectory set to obtain model parameter-training data pairing data.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] The present invention can effectively and self-supervisorily mine multiple modes in a large-scale driving trajectory data set, extract strategies through a unique interactive feature set, and finally output a robust global cost function and multi-modal scenario adaptive driving strategy. The global cost function is trained using the "model parameter-training data" pairing, so that it can judge the corresponding scenario according to the vehicle's state and output the corresponding model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0057] Figure 1 A schematic diagram of a flow chart of a multimodal scenario driving behavior modeling method based on inverse reinforcement learning provided in Example 1 of the present invention;

[0058] Figure 2 A schematic diagram of the process of inverse reinforcement learning provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0060] Embodiment 1 of the present invention provides a multi-modal scenario driving behavior modeling method based on inverse reinforcement learning, such as Figure 1 As shown, the multimodal scenario driving behavior modeling method based on inverse reinforcement learning includes the following steps:

[0061] Step S1, establishing an inverse reinforcement learning-scenario clustering model, inputting expert trajectories as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, and obtaining model parameter-training data pairing data;

[0062] Step S2, constructing a global cost function, and optimizing the global cost function using the model parameter-training data pairing data;

[0063] Step S3: Use the optimized global cost function to perform scene judgment, obtain a driving scene, and output model parameters corresponding to the driving scene.

[0064] It should be noted that the inverse reinforcement learning model currently used in intelligent driving is limited by linear functions and is difficult to meet the multimodality of driving behavior in different driving scenarios, resulting in limited generalization ability in different scenarios. Based on this problem, Example 1 of the present invention proposes a multimodal scene driving behavior modeling method based on inverse reinforcement learning. In this method, an inverse reinforcement learning-scene clustering model is constructed, and the expert trajectory is used as training data to infer the reward function from the expert trajectory through inverse reinforcement learning, and then the best proxy of the expert trajectory is found from the sampled trajectory through the reward function, and the best proxy is used as a model parameter. Based on the model parameter, scene clustering is performed, that is, it can be realized in the state of the matched driving trajectory, the driving scene corresponding to the driving trajectory state can be found, and it can be aggregated, and the driving trajectory is matched with the driving scene to obtain model parameter-training data pairing data.

[0065] In the existing inverse reinforcement learning model, the model parameters and feature weight vectors obtained by the inverse reinforcement learning model are directly used to train the optimization cost function, which leads to the neglect of the multimodality of driving behavior in different driving scenarios. In this method, the model parameters are matched with the training data, and the matched model parameter-training data pairing data is used to optimize the global cost function. Its core purpose is to imitate the driving habits of experts as much as possible when outputting the control parameters of the corresponding scene. After completing the training of the global cost function, the feature weight vector can be output for any initial state. According to the feature weight vector, the trajectory with the maximum reward can be selected from the sampled trajectory corresponding to the initial state. This trajectory satisfies the multimodality of driving behavior in different driving scenarios.

[0066] In an optional embodiment, the step S1 of inputting the expert trajectory as training data into the inverse reinforcement learning-scenario clustering model for cyclic training includes the following steps:

[0067] According to the initial state of the expert trajectory, generating a sampling trajectory corresponding to the expert trajectory;

[0068] Mapping the expert trajectory and the sampling trajectory to a feature space through a feature function to obtain a sampling feature;

[0069] The sampling characteristics are solved by using the maximum entropy solution method to obtain model parameters;

[0070] Performing scene clustering based on the model parameters to obtain expert trajectories in driving scenes;

[0071] The model parameters are paired with expert trajectories in a driving scenario to generate model parameter-training data paired data.

[0072] It should be noted that the purpose of step S1 is to train the expert trajectory through the inverse reinforcement learning-scenario clustering model to obtain stable model parameter-training data pairing data. In step S1, inverse reinforcement learning is first required, such as Figure 2 As shown, Example 1 of the present invention uses the expert trajectory within a period of time as the training data of inverse reinforcement learning. The expert trajectory includes the trajectory state within a period of time. The expert trajectory is sampled based on the initial state in the trajectory state to generate a sampled trajectory corresponding to the expert trajectory. In a multimodal scenario, sampling based on the initial state can generate multiple possible trajectories starting from the same initial state to cope with driving conditions in different driving scenarios.

[0073] The key features of the expert trajectory are extracted through the feature function, which can reflect the expert's driving style and decision-making logic; the sampled trajectory is sampled based on the initial state of the expert trajectory, and the same feature function is applied to the sampled trajectory to extract the corresponding features, ensuring that the sampled trajectory and the expert trajectory are compared in the same feature space. The extracted features are converted into vector form, each trajectory is represented by a feature vector, and the feature vectors of all trajectories are mapped to the feature space, in which the relationship between different trajectories and their similarity with the expert trajectory can be observed.

[0074] In a multimodal scenario, there are different ways to represent the same trajectory in the feature space. Therefore, in this embodiment 1, the maximum entropy solution method is used to solve the sampling features. A set of feature weight vectors is obtained by the maximum entropy solution method to maximize the probability of the expert trajectory appearing in the trajectory space. The inverse reinforcement algorithm maximizes the probability of the expert trajectory in the overall trajectory space so that the inverse reinforcement model can imitate the expert's driving habits as much as possible when outputting the control parameters of the corresponding scenario.

[0075] At the same time, the same scenes are clustered according to the model parameters obtained by inverse reinforcement learning to obtain expert trajectories under the same driving scenarios, which are used as new training data and matched with the model parameters to generate model parameter-training data pairing data. The model parameter-training data pairing data is then re-input into the inverse reinforcement learning-scenario clustering model for iterative training until a stable "model parameter-training data" pairing is obtained.

[0076] In an optional embodiment, generating a sampling trajectory corresponding to the expert trajectory according to the initial state of the expert trajectory includes the following steps:

[0077] Get in complex interactive environment Track status at the moment ,Will Track state of time length Integrate to generate Expert Tracks ,in, For the The initial state of the expert trajectory;

[0078] According to the initial state, generate the The sampled trajectories of expert trajectories ,in, is the total number of sampled trajectories, and The set of expert trajectories and sampled trajectories is denoted as .

[0079] It should be noted that the collected expert trajectory dataset is recorded as ,in, is the total number of expert trajectories, For the expert trajectory, for the Expert Tracks , which includes experts in The trajectory state within a certain time period, where the trajectory state includes position, speed, direction, etc.

[0080] For each expert trajectory, according to its initial state , generating a set of corresponding sampling trajectories , is the total number of sampled trajectories.

[0081] In an optional embodiment, solving the sampling feature by using the maximum entropy solution method includes:

[0082] First The sampled trajectories of expert trajectories Construct the objective function;

[0083] The objective function is solved using the maximum entropy method to obtain The feature weight vector corresponding to the expert trajectory;

[0084] Wherein, the objective function is constructed as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] In the above formula, represents the characteristic function, Indicates Expert track Sampled trajectories The sampling characteristics of represents the length of the feature vector, Indicates The feature weight vector corresponding to the expert trajectory.

[0089] In an optional embodiment, performing scene clustering based on the model parameters includes the following steps:

[0090] The expert trajectories are classified using a clustering algorithm according to the feature weight vectors corresponding to the expert trajectories, and the expert trajectories with the same labels are classified into the same set to obtain a scene trajectory set;

[0091] The data in the scene trajectory set is solved using the maximum entropy solution method to obtain the feature weight vector corresponding to the scene trajectory set;

[0092] The feature weight vector corresponding to the scene trajectory set is matched with the scene set corresponding to the feature weight vector corresponding to the scene trajectory set to obtain model parameter-training data pairing data.

[0093] It should be noted that the expert trajectories are classified using a clustering algorithm based on the feature weight vectors corresponding to the expert trajectories. The data with the same label is taken as a subset. In a subset, the expert trajectory data with the same label is included. The expert trajectory dataset is obtained by integrating the subsets , at this time, the expert trajectory dataset It can be recorded as ;

[0094] In the collection middle For the Subsets, is the total number of subsets. Data in the sub-collection , For the The total number of data in the subsets and , subset data The expert trajectories under the same driving trajectory scenario are included.

[0095] Among them, a clustering algorithm is used to cluster the set. The clustering algorithm will assign a cluster label to each data point to indicate which cluster it belongs to. According to the clustering results, the data points with the same label are grouped into different sets. Each set contains all the data points belonging to the same category.

[0096] Specifically, in Embodiment 1 of the present invention, the clustering process includes, for each expert trajectory and its corresponding sampling trajectory, finding The nearest point and The farthest points are calculated, and the distance between the nearest point and the farthest point is sorted.

[0097] The same set of data Through the same steps of inverse reinforcement learning, we can maximize the solution:

[0098]

[0099] Get the first The feature weight vector corresponding to the training data set:

[0100]

[0101] Continue to cycle the "inverse reinforcement learning-scenario clustering" process until a stable "model parameter-training data" pairing is obtained .

[0102] Specifically, the cycle process includes: using the gradient descent function to update the The feature weight vector corresponding to the training data set is calculated, and the loop stops until the expert trajectory is the optimal trajectory.

[0103] In an optional embodiment, optimizing the global cost function using the model parameter-training data pairing data includes: using the initial state of each expert trajectory in the scene trajectory set as input data of the global cost function, and using the feature weight vector corresponding to the scene trajectory set as a label of the global cost function.

[0104] In an optional embodiment, the global cost function includes a loss function, wherein the loss function is expressed as follows:

[0105]

[0106] in, represents the loss function, represents the cross entropy function, Represents a set of scene trajectories Middle The feature weight vector for data prediction, Indicates The feature weight vector corresponding to a set of scene trajectories.

[0107] The multi-modal scenario driving behavior modeling method based on inverse reinforcement learning proposed in Example 1 of the present invention is specifically applied to multi-scenario adaptive driving to reflect the effect of the method. Specifically, it includes three parts: data set standardization, multi-vehicle-in-the-loop interactive environment standardization, and multi-scenario adaptive driving strategy modeling based on inverse reinforcement learning.

[0108] For data set standardization, four data sets were used for verification in Example 1 of the present invention. First, a private data set will be used, which contains the positioning data of manually driven mining trucks. These data are thoroughly cleaned and preprocessed, including denoising, data amplification, and data format standardization. In addition, three public data sets, namely CitySim, Interaction, and Argoverse, will be used for comparison with the benchmark method. For these four data sets, preprocessing was performed in accordance with the NGSIM format to ensure the unification of the data structure. Among them, the end-to-end data preprocessing script is also included in Example 1 of the present invention. The trajectory data after preprocessing and denoising (NGSIM format) mainly includes the Cartesian coordinates of the main vehicle 、Convert coordinates to frenet coordinate system , and the high-order differentials in the two coordinate systems, respectively, the high-order differentials represent velocity, acceleration, and jerk. It is the road number in the frenet coordinate system.

[0109] The standardized multi-vehicle-in-the-loop interactive environment follows the format of OpenAI's GYM environment, providing a robust simulation platform for multiple vehicles in a complex interactive environment. The environment allows simulation of different traffic densities and various driving behaviors for training and evaluating driving strategy models.

[0110] In Example 1 of the present invention, a homogeneous state space and action space are uniformly designed for each environment, as well as an external reward mechanism that provides feedback to the agent. The driving agent plans a series of future trajectories based on its behavior strategy, and the environment returns new states, rewards, and completion status based on this action. In addition, a visualization kit is provided to facilitate real-time viewing of several possible trajectories and probabilities planned by the driving agent. The kit allows the user to pause and continue the simulation through controls, and provides an easy-to-operate user interface to set simulation hyperparameters.

[0111] Based on the standardization of data sets and multi-vehicle-in-the-loop interactive environments, a multimodal scenario driving behavior modeling method based on inverse reinforcement learning is used to process driving data, and the corresponding model parameters are obtained from the training data through inverse reinforcement learning.

[0112] Among them, the linear sum of features is used as the reward function ( ). First, a set of sampling trajectories is obtained from the initial state through a specific sampling method, and the expert trajectory and the sampling trajectory are mapped to the feature space using the feature function. The specific sampling method includes one of the open-loop strategy, forward reinforcement learning strategy, Laplace continuous strategy, and near-Gaussian optimization strategy.

[0113] The driving features of Example 1 of the present invention are automatically extracted from common features in related research, scene-based road features, and application-based interactive features through a neural network.

[0114] The scenes of the training data are classified based on the model parameters through the self-supervised contrastive learning algorithm, and the scene labels corresponding to each training data are output. The data with the same scene label are grouped together, and the "inverse reinforcement learning-scene clustering" process is repeated again until a stable "model parameter-training data" pairing is obtained. The global cost function is trained using the "model parameter-training data" pairing so that it can judge the corresponding scene according to the vehicle's state (position speed, etc.) and output the corresponding model parameters.

[0115] The present invention combines the characteristics of artificially designed specific complex interaction scenarios, the innovative points of contrastive inverse reinforcement learning and global cost function learning, and has the following advantages and positive effects:

[0116] Effectively capture drivers’ driving habits: By manually designing 17 features for interactive scenarios, the algorithm effectively embeds experts’ prior knowledge of driving habits in related scenarios, which promotes the effectiveness and convergence of the algorithm.

[0117] Accurate strategy differentiation: By calculating the Euclidean distance between the parameter vectors of each point, the algorithm can accurately distinguish between similar and dissimilar driving strategies, which helps to identify and aggregate strategies with common characteristics.

[0118] Effective regularization: By learning the global cost function, the algorithm can balance the influence of each local data point, reduce the impact of noise and the particularity of individual trajectories on the model, avoid overfitting, and improve the generalization ability of the model.

[0119] Dynamic adaptability: As the global cost function is continuously updated, the algorithm can dynamically adjust parameters to adapt to changes in the environment and data, helping the model remain robust in the face of different driving scenarios.

[0120] Global optimization: The algorithm promotes a comprehensive search of the parameter space by optimizing the global loss function, avoiding excessive concentration on local optimal solutions, thereby finding a better combination of strategy parameters.

[0121] The test results of the algorithm (SAIRL) proposed in this invention, the state-of-the-art (SOTA) algorithm in this field and the same type of inverse reinforcement learning algorithm on the same data set are compared as shown in the following table:

[0122]

[0123] The evaluation indicators selected in the above table are average distance deviation (ADE), final distance deviation (FDE), failure rate (MR) and similarity between planned trajectory and real trajectory (HP). It can be seen that the algorithm proposed in this invention is superior to the current SOTA algorithm in all indicators, proving its effectiveness.

[0124] Embodiment 2 of the present invention provides a multi-modal scenario driving behavior modeling system based on inverse reinforcement learning, including:

[0125] A pairing module, wherein the pairing module is used to establish an inverse reinforcement learning-scenario clustering model, input the expert trajectory as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, and obtain model parameter-training data pairing data;

[0126] A function optimization module, wherein the function optimization module is used to construct a global cost function and optimize the global cost function using the model parameter-training data pairing data;

[0127] The output module is used to use the optimized global cost function to perform scene judgment, obtain a driving scene, and output model parameters corresponding to the driving scene.

[0128] In an optional embodiment, the pairing module includes:

[0129] A sampling submodule, the sampling submodule is used to generate a sampling trajectory corresponding to the expert trajectory according to the initial state of the expert trajectory;

[0130] A mapping submodule, wherein the mapping submodule is used to map the expert trajectory and the sampling trajectory into a feature space through a feature function to obtain a sampling feature;

[0131] A solution submodule, the solution submodule is used to solve the sampling characteristics by using a maximum entropy solution method to obtain model parameters;

[0132] A clustering submodule, the clustering submodule is used to perform scene clustering based on the model parameters to obtain expert trajectories in driving scenes;

[0133] The scenario pairing submodule is used to pair the model parameters with expert trajectories in the driving scenario to generate model parameter-training data pairing data.

[0134] In an optional embodiment, the clustering submodule includes:

[0135] A classification unit, wherein the classification unit is used to classify the expert trajectories using a clustering algorithm according to the feature weight vectors corresponding to the expert trajectories, and classify the expert trajectories with the same label into the same set to obtain a scene trajectory set;

[0136] A weight unit, the weight unit is used to solve the data in the scene trajectory set by using the maximum entropy solution method to obtain a feature weight vector corresponding to the scene trajectory set;

[0137] A pairing unit is used to match the feature weight vector corresponding to the scene trajectory set with the scene set corresponding to the feature weight vector corresponding to the scene trajectory set to obtain model parameter-training data pairing data.

[0138] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-modal scenario driving behavior modeling method based on inverse reinforcement learning, characterized in that: The steps include: Step S1, establishing an inverse reinforcement learning-scenario clustering model, inputting expert trajectories as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, and obtaining model parameter-training data pairing data; Step S2, constructing a global cost function, and optimizing the global cost function using the model parameter-training data pairing data; Step S3: using the optimized global cost function to perform scene judgment, obtain a driving scene, and output model parameters corresponding to the driving scene; Wherein, the global cost function includes a loss function, wherein the loss function is expressed as follows: ; in, represents the loss function, represents the cross entropy function, Represents a set of scene trajectories Middle The feature weight vector for data prediction, Indicates The feature weight vector corresponding to a set of scene trajectories.

2. The multimodal scenario driving behavior modeling method based on inverse reinforcement learning according to claim 1 is characterized in that: In step S1, the expert trajectory is input as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, which includes the following steps: According to the initial state of the expert trajectory, generating a sampling trajectory corresponding to the expert trajectory; Mapping the expert trajectory and the sampling trajectory to a feature space through a feature function to obtain a sampling feature; The sampling characteristics are solved by using the maximum entropy solution method to obtain model parameters; Performing scene clustering based on the model parameters to obtain expert trajectories in driving scenes; The model parameters are paired with expert trajectories in a driving scenario to generate model parameter-training data paired data.

3. The multimodal scenario driving behavior modeling method based on inverse reinforcement learning according to claim 2 is characterized in that: According to the initial state of the expert trajectory, generating a sampling trajectory corresponding to the expert trajectory comprises the following steps: Get in complex interactive environment Track status at the moment ,Will Track state of time length Integrate to generate Expert Tracks ,in, For the The initial state of the expert trajectory; According to the initial state, generate the The sampled trajectories of expert trajectories ,in, is the total number of sampled trajectories, and The set of expert trajectories and sampled trajectories is denoted as .

4. The multimodal scenario driving behavior modeling method based on inverse reinforcement learning according to claim 3 is characterized in that: Solving the sampling characteristics using the maximum entropy solution method includes: First The sampled trajectories of expert trajectories Construct the objective function; The objective function is solved using the maximum entropy method to obtain The feature weight vector corresponding to the expert trajectory; Wherein, the objective function is constructed as follows: ; ; ; In the above formula, represents the characteristic function, Indicates Expert track Sampled trajectories The sampling characteristics of represents the length of the feature vector, Indicates The feature weight vector corresponding to the expert trajectory.

5. The multimodal scenario driving behavior modeling method based on inverse reinforcement learning according to claim 2 is characterized in that: Performing scene clustering based on the model parameters includes the following steps: The expert trajectories are classified using a clustering algorithm according to the feature weight vectors corresponding to the expert trajectories, and the expert trajectories with the same labels are classified into the same set to obtain a scene trajectory set; The data in the scene trajectory set is solved using the maximum entropy solution method to obtain the feature weight vector corresponding to the scene trajectory set; The feature weight vector corresponding to the scene trajectory set is matched with the scene set corresponding to the feature weight vector corresponding to the scene trajectory set to obtain model parameter-training data pairing data.

6. The multimodal scenario driving behavior modeling method based on inverse reinforcement learning according to claim 1 is characterized in that: Optimizing the global cost function using the model parameter-training data pairing data includes: using the initial state of each expert trajectory in the scene trajectory set as input data of the global cost function, and using the feature weight vector corresponding to the scene trajectory set as a label of the global cost function.

7. A multi-modal scenario driving behavior modeling system based on inverse reinforcement learning, characterized in that: include: A pairing module, wherein the pairing module is used to establish an inverse reinforcement learning-scenario clustering model, input the expert trajectory as training data into the inverse reinforcement learning-scenario clustering model for cyclic training, and obtain model parameter-training data pairing data; A function optimization module, wherein the function optimization module is used to construct a global cost function and optimize the global cost function using the model parameter-training data pairing data; An output module, the output module is used to use the optimized global cost function to perform scene judgment, obtain a driving scene, and output model parameters corresponding to the driving scene; Wherein, the global cost function includes a loss function, wherein the loss function is expressed as follows: ; in, represents the loss function, represents the cross entropy function, Represents a set of scene trajectories Middle The feature weight vector for data prediction, Indicates The feature weight vector corresponding to a set of scene trajectories.

8. The multi-modal scenario driving behavior modeling system based on inverse reinforcement learning according to claim 7 is characterized in that: The pairing module comprises: A sampling submodule, the sampling submodule is used to generate a sampling trajectory corresponding to the expert trajectory according to the initial state of the expert trajectory; A mapping submodule, the mapping submodule is used to map the expert trajectory and the sampling trajectory into a feature space through a feature function to obtain a sampling feature; A solution submodule, the solution submodule is used to solve the sampling characteristics by using a maximum entropy solution method to obtain model parameters; A clustering submodule, the clustering submodule is used to perform scene clustering based on the model parameters to obtain expert trajectories in driving scenes; The scenario pairing submodule is used to pair the model parameters with expert trajectories in the driving scenario to generate model parameter-training data pairing data.

9. The multimodal scenario driving behavior modeling system based on inverse reinforcement learning according to claim 8, characterized in that: The clustering submodule includes: A classification unit, wherein the classification unit is used to classify the expert trajectories using a clustering algorithm according to the feature weight vectors corresponding to the expert trajectories, and classify the expert trajectories with the same label into the same set to obtain a scene trajectory set; A weight unit, the weight unit is used to solve the data in the scene trajectory set by using the maximum entropy solution method to obtain a feature weight vector corresponding to the scene trajectory set; A pairing unit is used to match the feature weight vector corresponding to the scene trajectory set with the scene set corresponding to the feature weight vector corresponding to the scene trajectory set to obtain model parameter-training data pairing data.

Citation Information

Patent Citations

  • Method, equipment and device for controlling vehicle driving

    CN116203964A

  • Intelligent automobile standard test scene generation method based on inverse reinforcement learning

    CN118228591A

  • Unmanned aerial vehicle target tracking method and system based on improved machine learning

    CN118334085A

  • Systems and methods to learn constraints from expert demonstrations

    US20230376749A1

  • Trajectory generation for mobile agents

    WO2023148297A1