Autonomous driving decision model training method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for training an autonomous driving decision model, the method comprising: obtaining the interaction data between the initial autonomous driving main vehicle model and the environment vehicle, the initial autonomous driving main vehicle model comprising an initial embodied world model and an initial behavior model; using the interaction data as training samples, so that the initial embodied world model respectively models various types of environment vehicles based on the training samples; imagining the driving trajectory of the initial autonomous driving main vehicle model corresponding to the autonomous driving vehicle based on the modeling results and the initial behavior model, and training the initial behavior model based on the driving trajectory; terminating the training when the current training reaches a preset termination condition, and obtaining an autonomous driving decision model; the types of environment vehicles are divided based on the interaction distance between the environment vehicle and the autonomous driving vehicle, and the modeling methods of various types of environment vehicles are different. The method and device provided by the present application improve the training efficiency, decision efficiency and decision accuracy of the model.

Description

Technical Field

[0001] The present application relates to the field of learning algorithm technology, and more specifically, to an autonomous driving decision model training method, device and storage medium. Background Art

[0002] The application and popularization of self-driving cars will help improve traffic safety, improve traffic efficiency, promote sustainable development and promote new business opportunities. Among them, the lag in the development of intelligent decision-making technology in car-to-car high-interaction driving scenarios is one of the main bottlenecks restricting the intelligence level of self-driving cars. There are two main types of reinforcement learning technology in intelligent decision-making technology, one is model-free and the other is model-based. Model-based reinforcement learning can improve data utilization efficiency by modeling the environment to generate virtual data training strategies, or use models for planning to learn driving strategies more effectively.

[0003] However, existing model-based reinforcement learning technology can only model driving scenarios at the scenario level, that is, it does not classify vehicles according to the interaction distance between vehicles, and cannot model driving scenarios from the perspective of individual traffic participants. This confuses the states of different vehicles in the driving scenarios, which not only increases the computational burden of describing similar scenarios and reduces the learning efficiency of the algorithm; it also increases the decision-making difficulty of the autonomous driving decision model, making it difficult for the autonomous driving decision model to focus on vehicles that directly interact with the autonomous driving vehicle when making decisions.

[0004] Therefore, how to model driving scenarios from the perspective of individual traffic participants and train autonomous driving decision-making models has become a technical problem that the industry urgently needs to solve. Summary of the invention

[0005] The present application provides an autonomous driving decision model training method, device and storage medium to solve the technical problem of how to model driving scenarios from the perspective of individual traffic participants and train the autonomous driving decision model in the prior art.

[0006] In a first aspect, the present application provides an autonomous driving decision model training method, comprising:

[0007] Acquire interaction data between an initial autonomous driving main vehicle model and an environment vehicle, wherein the initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model;

[0008] Using the interaction data as training samples of the initial embodied world model, so that the initial embodied world model respectively models various types of environment vehicles based on the training samples;

[0009] Imagining a driving trajectory of the automatic driving vehicle corresponding to the initial automatic driving main vehicle model based on the modeling result and the initial behavior model, and training the initial behavior model based on the driving trajectory;

[0010] When the current training reaches a preset termination condition, the training is terminated to obtain the autonomous driving decision model;

[0011] Among them, the type of the environmental vehicle is divided based on the interaction distance between the environmental vehicle and the autonomous driving vehicle, and the modeling method of each type of environmental vehicle is different.

[0012] In some embodiments, obtaining the interaction data between the initial autonomous driving main vehicle model and the environment vehicle includes:

[0013] Creating a driving simulator based on historical driving data, and running the driving simulator in a set container to simulate a virtual driving scenario;

[0014] The initial autonomous driving main vehicle model outside the container interacts with the environment vehicle in the virtual driving scene simulated by the driving simulator in the container to obtain interaction data, wherein the interaction data includes the vehicle state, driving action, reward, vehicle state at the next moment and round termination signal of the current vehicle at the current moment.

[0015] In some embodiments, obtaining the interaction data includes:

[0016] Create a coordinate system based on the autonomous driving vehicle, and obtain trajectory information of the autonomous driving vehicle and various environmental vehicles in the coordinate system within a historical time period;

[0017] The trajectory information includes the coordinates of the autonomous driving vehicle and various environmental vehicles in the coordinate system, and their orientations in the coordinate system; the vehicle state includes the trajectory information;

[0018] The driving action includes a longitudinal target speed selected by the initial automatic driving host vehicle model.

[0019] In some embodiments, the interaction data further includes a reward function, wherein the reward function includes a safety reward r safe and efficiency reward efficient :

[0020] The safety reward r safe The calculation formula is: safe =-d1*(1+v norm );

[0021] The efficiency reward r efficient The calculation formula is: efficient =d2*vnorm -d3;

[0022] Among them, v norm is the speed value of the autonomous driving vehicle after normalization; d1, d2 and d3 are all set fixed values; when the autonomous driving vehicle collides with the environmental vehicle, the safety reward takes effect.

[0023] In some embodiments, the using the interaction data as a training sample of the initial embodied world model includes:

[0024] Initializing a preset experience replay pool, where the experience replay pool is used to store the interaction data;

[0025] Randomly collecting a group of interaction data in the experience replay pool as the training sample;

[0026] Wherein, in the case of a storage space alarm in the experience replay pool, the interaction data in the experience replay pool is processed based on the storage time of the interaction data in the experience replay pool.

[0027] In some embodiments, modeling each type of environment vehicle based on the training samples includes:

[0028] Setting a shared encoder, wherein the shared encoder is used to extract underlying features in the training sample;

[0029] Setting the initial autonomous driving main vehicle model and a branch network of various environmental vehicles;

[0030] Modeling various types of environment vehicles and the autonomous driving vehicle based on the shared encoder and each branch network;

[0031] Among them, each branch network includes a branch encoder, a gated recurrent unit and an observation reconstructor, the branch encoder is used to encode the vehicle state based on the vehicle category, the gated recurrent unit is used to update the vehicle state, and the observation reconstructor is used to reconstruct the vehicle state; the vehicles include the autonomous driving vehicles and various types of environmental vehicles.

[0032] In some embodiments, the modeling of various types of environment vehicles and the autonomous driving vehicle includes:

[0033] Updating the deterministic latent state of any vehicle at the current moment based on the deterministic latent state of any vehicle at the previous moment, the stochastic latent state of the previous moment, and the action of the previous moment;

[0034] Obtaining the random latent state of any vehicle at the current moment based on a priori and a posteriori methods;

[0035] Reconstructing the vehicle state of any one of the vehicles based on the deterministic latent state and the posterior of the stochastic latent state;

[0036] Mixing the latent states of each vehicle based on a cross-attention mechanism, wherein the latent states include the deterministic latent state and the stochastic latent state;

[0037] The embodied world model is trained based on the mixed latent state, the set predicted reward and the episode termination signal.

[0038] In some embodiments, predicting the driving trajectory of the initial autonomous driving main vehicle model based on the modeling result and the initial behavior model, and training the initial behavior model based on the driving trajectory includes:

[0039] Identifying the latent state of the direct interaction vehicle in the mixed latent state based on a cross attention mechanism; the environment vehicle includes the direct interaction vehicle and the potential interaction vehicle;

[0040] Based on the current potential state of each vehicle, the controller evaluator module in the initial behavior model is input to obtain the value estimation of the corresponding action at the current moment output by the controller evaluator module;

[0041] The initial behavior model is trained based on the value estimates.

[0042] In a second aspect, the present application provides an autonomous driving decision model training device, comprising:

[0043] An acquisition module, used to acquire interaction data between an initial autonomous driving main vehicle model and an environment vehicle, wherein the initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model;

[0044] A modeling module, configured to use the interaction data as training samples of the initial embodied world model, so that the initial embodied world model respectively models various types of environment vehicles based on the training samples;

[0045] A training module, configured to imagine a driving trajectory of the automatic driving vehicle corresponding to the initial automatic driving main vehicle model based on the modeling result and the initial behavior model, and to train the initial behavior model based on the driving trajectory;

[0046] A termination module, used to terminate the training when the current training reaches a preset termination condition, and obtain the autonomous driving decision model;

[0047] Among them, the type of the environmental vehicle is divided based on the interaction distance between the environmental vehicle and the autonomous driving vehicle, and the modeling method of each type of environmental vehicle is different.

[0048] In a third aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above method when executed by a processor.

[0049] The autonomous driving decision model training method, device and storage medium provided in the present application obtain interaction data, classify environmental vehicles, and model driving scenarios from the individual perspective of traffic participants, so that various types of environmental vehicles can be distinguished during the training of the initial autonomous driving main vehicle model. This can not only effectively improve the training efficiency of the initial autonomous driving main vehicle model, but also improve the decision-making efficiency and accuracy of the autonomous driving decision model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 One of the flowcharts of the autonomous driving decision model training method provided in the embodiment of the present application;

[0053] Figure 2 A schematic diagram of a virtual driving scene provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a neural network of an embodied world model provided by an embodiment of the present invention;

[0055] Figure 4 A neural network diagram of a behavior model provided by an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a training framework based on an autonomous driving decision model training method provided by an embodiment of the present invention;

[0057] Figure 6 A second flowchart of the autonomous driving decision model training method provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of an autonomous driving decision model training device provided in an embodiment of the present application;

[0059] Figure 8A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0061] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or modules need not be limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] The autonomous driving decision model training method provided in the embodiment of the present application is applicable to a terminal, which may be a variety of electronic devices with a display screen and supporting web browsing, including but not limited to servers, smart phones, tablet computers, laptops, and desktop computers.

[0063] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0064] Figure 1 One of the flow charts of the autonomous driving decision model training method provided in the embodiment of the present application is as follows: Figure 1 As shown, the method includes step 110, step 120, step 130 and step 140. The steps of the method flow are only a possible implementation of the present application.

[0065] Step 110: Obtain interaction data between an initial autonomous driving main vehicle model and an environment vehicle, where the initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model.

[0066] Specifically, the executor of the autonomous driving decision model training method provided in the embodiment of the present application is an autonomous driving decision model training device, which can be a hardware device independently set in the terminal, or a software program running in the terminal. For example, when the terminal is a desktop computer, the autonomous driving decision model training device can be embodied as an application such as model training software in the desktop computer.

[0067] The initial autonomous driving main vehicle model is an autonomous driving decision model that has not been trained or has not completed training. The autonomous driving decision model is a computer model that simulates and implements the vehicle's autonomous driving function by using perception, decision-making and control algorithms.

[0068] The vehicle controlled by the autonomous driving decision model is its corresponding autonomous driving vehicle, which can be called an intelligent agent.

[0069] The initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model. Training the initial autonomous driving main vehicle model can obtain an autonomous driving decision model for controlling the autonomous driving vehicle to drive. That is, training the initial embodied world model and the initial behavior model can obtain an autonomous driving decision model for controlling the autonomous driving vehicle to drive, which is composed of the embodied world model and the behavior model.

[0070] The environment car is a vehicle simulated by a driving simulator.

[0071] A driving simulator is a software tool or system that simulates and reproduces real-world driving scenarios and provides a virtual driving experience. It uses technologies such as computer graphics, physics engines, and traffic models to simulate elements such as vehicles, roads, traffic rules, and environments, so that the created autonomous driving car can perform driving operations in a virtual driving scenario.

[0072] The virtual driving scene may include multiple environment vehicles. The autonomous driving vehicle created by the autonomous driving decision model will interact with these environment vehicles during driving, thereby generating interaction data. The interaction data may include the vehicle status, driving actions, reward functions, and preset termination conditions of the vehicle.

[0073] Step 120: Use the interaction data as training samples of the initial embodied world model, so that the initial embodied world model models each type of environment vehicle based on the training samples. The types of environment vehicles are divided based on the interaction distance between the environment vehicle and the autonomous driving vehicle, and each type of environment vehicle is modeled in a different way.

[0074] Specifically, the embodied world model can be used to reconstruct the driving scene from the perspective of individual traffic participants, and provide the latent state of individual traffic participants to the behavior model.

[0075] Modeling driving scenarios from a scenario perspective means that when modeling each vehicle, the type of vehicle will not be distinguished, and the data processing and modeling methods for each vehicle are the same. However, different types of vehicles have different degrees of influence on the driving behavior of the autonomous vehicle.

[0076] Therefore, if the driving scenario is modeled from a scenario perspective, key data is not identified based on the actual driving scenario during the initial training of the autonomous driving main vehicle model and the use of the autonomous driving decision model, resulting in poor training results and poor decision results output during final use.

[0077] The driving scenario is modeled from the perspective of individual traffic participants. That is, the environmental vehicles with a higher degree of influence on the driving behavior of the autonomous driving vehicle are classified from the interaction distance between the environmental vehicles and the autonomous driving vehicle, and the environmental vehicles with different degrees of influence are distinguished. Therefore, in the training process of the initial autonomous driving main vehicle model and the use of the autonomous driving decision model, the relevant data of the environmental vehicles with a higher degree of influence are focused on. This can not only effectively improve the training efficiency, but also improve the decision-making efficiency and accuracy of the autonomous driving decision-making model.

[0078] The interaction distance can be the straight-line physical distance between the autonomous driving vehicle and the environment vehicle; it can also be the interaction time interval between the environment vehicles, that is, the time interval for the environment vehicle to catch up with the autonomous driving vehicle, or for the autonomous driving vehicle to catch up with the environment vehicle, determined by the speed difference between the two; it can also be a combination of the straight-line physical distance and the time interval. It can be set according to actual conditions.

[0079] The environmental vehicles can be divided according to the interaction distance between them, and the autonomous driving vehicles can also be classified as a separate category.

[0080] For example, the autonomous driving vehicles controlled by the autonomous driving decision model are classified into one category; the environmental vehicles that interact with the autonomous driving vehicles at a closer distance are classified into one category, and these vehicles are called direct interaction vehicles; the environmental vehicles that interact with the autonomous driving vehicles at a farther distance are classified into one category, and these vehicles are called potential interaction vehicles.

[0081] Before training the initial autonomous driving main vehicle model, the model parameters of the initial embodied world model and the initial behavior model can be initialized through a parameter initializer.

[0082] Step 130: imagine the driving trajectory of the autonomous driving vehicle corresponding to the initial autonomous driving main vehicle model based on the modeling results and the initial behavior model, and train the initial behavior model based on the driving trajectory.

[0083] Specifically, the behavior model is used to introduce an attention mechanism to adaptively identify directly interacting vehicles and make optimal decisions based on the latent state of each vehicle in the current driving scene.

[0084] Through the initial embodied world model, each type of vehicle is modeled, and the potential state of each vehicle in the current driving scene is modeled. The modeling process of each type of vehicle is different.

[0085] The initial embodied world model is interacted with the initial behavior model, that is, the latent state input value of each vehicle is input into the initial behavior model. The initial behavior model imagines the driving trajectory of the autonomous driving vehicle based on the obtained latent state. The trajectory information corresponding to the driving trajectory may include the position information of the autonomous driving vehicle and the position information of various environmental vehicles.

[0086] The driving trajectory is used as the training sample of the initial behavior model to train the initial behavior model.

[0087] The model parameters of the initial embodied world model and the initial behavior model can be optimized using the gradient descent method.

[0088] Step 140: When the current training reaches a preset termination condition, the training is terminated to obtain an autonomous driving decision model.

[0089] Specifically, the preset termination condition is a set termination rule for controlling the termination of the initial autonomous driving main vehicle model training. The preset termination condition may include terminating the training when the number of interaction steps between the initial autonomous driving main vehicle model and the driving simulator is equal to a preset threshold.

[0090] The autonomous driving vehicle controlled by the autonomous driving decision model moves from one action to another as one step of interaction. The preset threshold can be determined based on whether the model converges, that is, when the number of interaction steps between the initial autonomous driving main vehicle model and the driving simulator is equal to the preset threshold, the state of the initial autonomous driving main vehicle model is a converged state. For example, the preset threshold can be set to 500,000 steps.

[0091] It can be judged in real time whether the current training has reached the preset termination condition, that is, whether the number of interaction steps between the initial autonomous driving main vehicle model and the driving simulator has reached the preset threshold. If not, the initial autonomous driving main vehicle model continues to be controlled to interact with the driving simulator, and the model parameters of the initial embodied world model and the initial behavior model are further tuned and updated until the current training reaches the preset termination condition.

[0092] After the training is completed, an autonomous driving decision model that can accurately control the autonomous driving vehicle to drive can be obtained. The model parameters of the last updated initial embodied world model can be used as the training result to obtain the autonomous driving decision model.

[0093] The autonomous driving decision model training method provided in the embodiment of the present application obtains interaction data, classifies environmental vehicles, and models driving scenarios from the individual perspective of traffic participants, so that various types of environmental vehicles can be distinguished during the training process of the initial autonomous driving main vehicle model. This can not only effectively improve the training efficiency of the initial autonomous driving main vehicle model, but also improve the decision-making efficiency and accuracy of the autonomous driving decision model.

[0094] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0095] In some embodiments, step 110 includes:

[0096] Create a driving simulator based on historical driving data and run the driving simulator in a set container to simulate virtual driving scenarios;

[0097] The initial autonomous driving main vehicle model outside the container interacts with the environment vehicle in the virtual driving scene simulated by the driving simulator inside the container to obtain interaction data, which includes the vehicle state, driving action, reward, vehicle state at the next moment and round termination signal of the current vehicle at the current moment.

[0098] Get interaction data, including:

[0099] Create a coordinate system based on the autonomous driving vehicle, and obtain the trajectory information of the autonomous driving vehicle and various environmental vehicles in the coordinate system during the historical time period;

[0100] Among them, the trajectory information includes the coordinates of the autonomous driving vehicle and various environmental vehicles in the coordinate system, as well as the orientation in the coordinate system; the vehicle state includes the trajectory information;

[0101] The driving action includes the longitudinal target speed selected by the initial autonomous driving host vehicle model.

[0102] The interaction data also includes a reward function, which includes a safety reward r safe and efficiency reward efficient :

[0103] Safety Reward safe The calculation formula is: safe =-d1*(1+v norm );

[0104] Efficiency Reward efficient The calculation formula is: efficient =d2*v norm -d3;

[0105] Among them, v normis the normalized speed value of the autonomous driving vehicle; d1, d2 and d3 are all set fixed values; when the autonomous driving vehicle collides with the environment vehicle, the safety reward takes effect.

[0106] Specifically, the historical driving data is a real data set collected historically. The real driving data in the real data set can be obtained from the INTERACTION data set.

[0107] The driving simulator can replay virtual driving scenarios based on real data sets and define the interaction parameters required for the model algorithm to run, such as vehicle status, reward function, and termination judgment, that is, the interaction data includes the vehicle status of the current vehicle at the current moment, driving action, reward, vehicle status at the next moment, and round termination signal.

[0108] The driving simulator can be run in a Docker container. Socket communication can be used to exchange information between the driving simulator in the Docker container and the autonomous driving decision model outside the Docker container. The driving simulator can use the matplotlib library to render the driving scene in real time and provide a visual driving scene interface.

[0109] For example, Figure 2 A schematic diagram of a virtual driving scene provided by an embodiment of the present invention; Figure 2 As shown, the main vehicle is an autonomous driving vehicle in a virtual driving scene, and the planned path of the main vehicle is rendered as curves in the virtual driving scene.

[0110] The vehicle state includes trajectory information. The trajectory information includes the coordinates of the autonomous driving vehicle and various environmental vehicles in the coordinate system constructed by the autonomous driving vehicle, as well as the orientation in the coordinate system. The number of various environmental vehicles can be preset.

[0111] For example, the number of environmental vehicles is set to 10, and the trajectory information includes the autonomous driving vehicle controlled by the autonomous driving decision model and the trajectory information of the 10 nearest environmental vehicles within 60 meters around the autonomous driving vehicle within the past 2 seconds. The trajectory information of each vehicle within the past 2 seconds is obtained at intervals of 0.01 seconds.

[0112] The vehicle status can be divided into different categories according to the type of vehicle. For example, the vehicle status of the autonomous driving vehicle is one category; the vehicle status of the five nearest environmental vehicles among the environmental vehicles is used as the vehicle status of the direct interaction vehicle; the vehicle status of the five farther environmental vehicles among the environmental vehicles is used as the vehicle status of the potential interaction vehicle; if the number of environmental vehicles is less than 10, the corresponding vacancies in the vehicle status will be filled with 0.

[0113] The action space of the autonomous driving decision model is a discrete action space, which can be represented by a set. The set can contain four optional driving actions. The driving action represents the longitudinal target speed that the autonomous driving decision model expects the autonomous driving vehicle to achieve.

[0114] For example, the four longitudinal target speeds are set to (0, 3, 6, 9) meters per second. After the longitudinal target speed is given, the simulator uses a proportional-integral-derivative (PID) controller to control the autonomous vehicle to complete the longitudinal speed tracking and lateral path tracking according to the longitudinal target speed and the preset planned path.

[0115] The simulated virtual driving scene obtains the coordinates, orientation and other dynamic information of each vehicle in the virtual driving scene at the next moment according to the driving actions of the autonomous driving decision model, and feeds back the reward value to the autonomous driving decision model according to the reward function r.

[0116] The reward function r contains the safety reward r safe and efficiency reward efficient .

[0117] Safety Reward safe It is mainly designed based on whether a collision occurs. The reward will only take effect when a collision occurs.

[0118] We can define d1 as 30, then r safe =-30*(1+v norm ).

[0119] Among them, v norm It refers to the normalized speed value. Since the speed may be a relatively large value, reducing the speed range can reduce the reward space, thereby improving the training effect of the autonomous driving decision model. By dynamically adjusting the size of the safety reward, the autonomous driving vehicle controlled by the autonomous driving decision model can be effectively trained to avoid active collisions with the surrounding vehicles.

[0120] Efficiency Reward efficiemt It takes effect at every moment, that is, the efficiency reward has a value at every moment, and a penalty is given at every moment, so it can effectively avoid negative driving behavior of autonomous driving vehicles.

[0121] We can define d2 and d3 to be 0.3, then r efficient =0.3*v norm -0.3.

[0122] The reward function r of the model-based reinforcement learning decision-making method in the embodiment of the present application can be expressed as r=r safe +r efficient .

[0123] The autonomous driving decision model training method provided in the embodiment of the present application can effectively extract relevant data by defining vehicle status, driving actions and reward functions, thereby improving the training efficiency of the autonomous driving decision model.

[0124] In some embodiments, step 120 includes:

[0125] Initialize the preset experience replay pool, which is used to store interaction data;

[0126] A set of interaction data is randomly collected from the experience replay pool as training samples;

[0127] In the case of a storage space alarm in the experience replay pool, the interaction data in the experience replay pool is processed based on the storage time of the interaction data in the experience replay pool.

[0128] Specifically, the experience replay pool is used to store the interaction data obtained by the interaction between the autonomous driving decision model and the driving simulator.

[0129] You can initialize the experience replay pool to store interaction data.

[0130] The interaction data at time t stored in the experience replay pool can be represented by (s t ,a t ,r t ,s t+1 ,c t ) is composed of a five-tuple; where s is the vehicle state; a is the driving action; c t It is the termination information of whether the interaction round ends at time t.

[0131] For example, the termination information includes that the autonomous driving car runs out of the lane boundary or collides with other environment cars, which means that the current interaction round ends. After receiving it, the autonomous driving decision model can be re-initialized and re-interacted and trained and the reward value can be accumulated.

[0132] The agent network model consists of two parts: an embodied world model and a behavior model based on an attention mechanism. The embodied world model is used to model the driving scene from the perspective of individual vehicles, and the behavior model is used to calculate the value of the agent and obtain a feasible driving strategy.

[0133] The autonomous driving decision model can interact with the driving simulator and save the interaction data as (s t ,a t ,r t ,s t+1 ,c t ) in the experience replay pool as training samples for the autonomous driving decision model.

[0134] Because the capacity of the experience replay pool is limited, it is necessary to set storage rules. The storage rules can be first-in-first-out, that is, when the storage space of the experience replay pool is alarmed, the interaction data with the earliest storage time will be processed.

[0135] The autonomous driving decision model training method provided in the embodiment of the present application can effectively manage the interactive data by storing the interactive data in the experience replay pool, and can obtain training samples at any time, thereby improving the training efficiency of the autonomous driving decision model.

[0136] In some embodiments, step 120 includes:

[0137] Set a shared encoder, which is used to extract the underlying features in the training samples;

[0138] Set up the initial autonomous driving main vehicle model and the branch network of various environment vehicles;

[0139] Model various environmental vehicles and autonomous vehicles based on shared encoders and various branch networks;

[0140] Among them, each branch network includes a branch encoder, a gated recurrent unit and an observation reconstructor. The branch encoder is used to encode the vehicle state based on the vehicle category, the gated recurrent unit is used to update the vehicle state, and the observation reconstructor is used to reconstruct the vehicle state; the vehicles include autonomous driving vehicles and various environmental vehicles.

[0141] Modeling of various environmental vehicles and autonomous vehicles, including:

[0142] Update the deterministic latent state of any vehicle at the current moment based on the deterministic latent state of any vehicle at the previous moment, the stochastic latent state of any vehicle at the previous moment, and the action of any vehicle at the previous moment;

[0143] Obtain the random latent state of any vehicle at the current moment based on a priori and a posteriori methods;

[0144] Reconstructing the vehicle state of any vehicle based on the posterior of the deterministic latent state and the stochastic latent state;

[0145] The latent states of each vehicle are mixed based on the cross-attention mechanism, and the latent states include deterministic latent states and random latent states;

[0146] The embodied world model is trained based on the mixed latent states, the set predicted rewards, and the episode termination signal.

[0147] Specifically, a batch of training samples can be randomly collected from the experience replay pool, and these samples can be used to train the model parameters of the initial embodied world model, so that the embodied world model can model the driving scene from an individual perspective in the latent space and predict the reward.

[0148] Figure 3 A neural network diagram of an embodied world model provided in an embodiment of the present invention; Figure 3 As shown, the embodiment of the present application adopts a branch structure network to process different types of vehicles in the vehicle state, and models each vehicle in the driving scene separately.

[0149] Use and Represents the vehicle state s at time t t The vehicle states of the autonomous vehicle (ego, corresponding to the subscript e), the direct interaction vehicle (direct influence, corresponding to the subscript d), and the potential interaction vehicle (potential influence, corresponding to the subscript p) in the example are encoded using a shared encoder (shared trajectory encoder) and three branch encoders to obtain the following formula:

[0150]

[0151]

[0152]

[0153] in, Represents the shared parameter trajectory encoder, Enc θ1 ,Enc θ2 and Enc θ3 The branch encoders representing the autonomous driving vehicle branch, the direct interaction vehicle branch, and the potential interaction vehicle branch respectively; and is the feature obtained from the vehicle state. The encoder can be composed of a 2-layer fully connected network with 200 neurons.

[0154] The data of the branch encoder is the same as the number of vehicle types. The shared encoder is used to extract the underlying features in the training samples. The underlying features usually refer to the basic and abstract feature representations in the input data.

[0155] The embodied world model models all vehicles in the latent space and maintains a latent state for each vehicle. The latent state consists of two parts: deterministic latent state and random latent state. Determinism can be understood as the state data of people's fixed driving habits, and randomness can be understood as the state data of sudden behaviors. By combining deterministic latent states and random latent states, the autonomous driving decision model can be applied to more application scenarios.

[0156] Use and Denote the deterministic latent states of the autonomous driving vehicle, direct interaction vehicle, and potential interaction vehicle at time t, respectively. and represents the random latent state of the autonomous driving car, direct interaction car and potential interaction car at the current time t. It is known that the action a of the autonomous driving car at the previous moment t-1 As well as the latent state at the previous moment, the deterministic latent state at the current moment t is obtained using a branched gated recurrent unit (GRU). The calculation formula is as follows:

[0157]

[0158]

[0159]

[0160] Among them, GRU e , GRU d and GRU p The gated recurrent units representing the autonomous driving vehicle branch, the direct interaction vehicle branch, and the potential interaction vehicle branch respectively.

[0161] There are two ways to obtain the random latent state at the current moment, the difference lies in whether the features encoded according to the current state are considered in the input. For input without considering the features encoded according to the current state, only the deterministic latent state h at the current moment is considered. t , the random latent state obtained at this moment is called the prior of the random latent state, and is expressed as Denotes, otherwise known as the posterior, z t The specific calculation formula is as follows:

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168] Among them, p and q are the prior and posterior probability distributions of the random latent state, respectively.

[0169] After obtaining the random latent state, the vehicle state is reconstructed using the posterior of the deterministic latent state and the random latent state, respectively. and It represents the vehicle state reconstructed according to the latent state. The specific formula is as follows:

[0170]

[0171]

[0172]

[0173] Among them, Rec e , Rec d and Rec p They are the observation reconstructors of each branch structure respectively.

[0174] The initial embodied world model also learns to predict rewards and the end-of-round signal End of turn signal Rewards can be predicted for parameters related to the preset termination conditions. is a parameter related to the prediction reward function. For all vehicles including the self-driving car, the cross attention mechanism is used to mix the latent states of each vehicle. The query vector of the cross attention comes only from the self-driving car, while the key vector and value vector come from all vehicles. This latent state mixing method can effectively avoid the difference in prediction value caused by different vehicle state sorting. t After that, this value is combined with the potential state of the autonomous vehicle as input and the predicted reward is determined and termination The relevant formula is as follows:

[0175]

[0176]

[0177] Among them, Rew and Cont represent the reward prediction network and the termination prediction network respectively, and both can be composed of a 4-layer fully connected network with 400 neurons. The reward prediction network is the reward predictor, and the termination prediction network is the termination predictor.

[0178] The ultimate training goal of the initial embodied world model can be described as: learning meaningful latent state representation by minimizing the reconstruction error; improving the embodied world model's ability to fit vehicle state transitions by minimizing the KL divergence of the prior and posterior probability distributions of random latent states; and enabling the autonomous driving decision model to obtain accurate value estimation capabilities by minimizing the error between predicted rewards and actual rewards, and the error between predicted termination and actual termination.

[0179] The autonomous driving decision model training method provided in the embodiment of the present application can solve the problem of vehicle latent state confusion through the embodied world model, can independently model the vehicle state of each vehicle, and can effectively improve the training efficiency of the embodied world model by training the initial embodied world model through a branch structure, thereby enhancing the modeling effect for individual vehicles.

[0180] In some embodiments, step 130 includes:

[0181] Based on the cross attention mechanism, the latent state of the direct interaction vehicle is identified in the mixed latent state; the environmental vehicles include the direct interaction vehicle and the potential interaction vehicle;

[0182] Based on the current potential state of each vehicle, the controller evaluator module in the initial behavior model is input to obtain the value estimate of the corresponding action at the current moment output by the controller evaluator module;

[0183] An initial behavior model is trained based on the value estimates.

[0184] Specifically, since the scene-level world model mixes the latent states of all vehicles in the driving scene, the corresponding behavior model cannot focus on the directly interacting vehicles. Therefore, the embodiment of the present application uses a behavior model based on the attention mechanism on the basis of individual vehicle modeling.

[0185] Figure 4 A neural network diagram of a behavior model provided by an embodiment of the present invention; Figure 4 As shown, the behavior model consists of two parts, one is the cross-attention mechanism and the other is the controller-evaluator module.

[0186] The latent states of all vehicles in the current virtual driving scene are input into the initial behavior model. The cross attention mechanism can be used to mix the vehicle latent states and adaptively identify the directly interacting vehicles in the process. The query vector of the cross attention mechanism comes only from the autonomous driving vehicle, and the key value vector and value vector come from all vehicles. t Then, this value and the latent state of the autonomous vehicle are used as the input of the controller-evaluator module. The controller-evaluator module is used to output the driving action a according to the latent state at time t.t and value estimate v t , the relevant formula is as follows:

[0187]

[0188]

[0189]

[0190] Among them, π ω is a controller network with parameters ω, u δ is the evaluator network with parameter δ, G t is the reward value, H is the end time of the round, In the embodiment of the present application, λ-return is used to calculate the return value, where γ is a discount factor and λ is a parameter that balances long-term return and short-term return.

[0191] The training goal of the evaluator is to accurately evaluate the value estimation of different latent states under the current strategy, and the training goal of the controller is to maximize the state evaluation value of the evaluator. In an embodiment of the present application, the gradients of the training controller and the evaluator are not back-propagated to the latent state, so that the training of the latent state comes entirely from the embodied world model. The controller and the evaluator can both be composed of a fully connected neural network with 4 layers and 400 neurons.

[0192] The autonomous driving decision model training method provided in the embodiment of the present application improves the training efficiency of the initial behavior model by interacting with the embodied world model based on the attention mechanism and paying attention to the directly interacting vehicle.

[0193] Figure 5 A schematic diagram of a training framework based on an autonomous driving decision model training method provided by an embodiment of the present invention; Figure 6 The second flowchart of the autonomous driving decision model training method provided in the embodiment of the present application; Figure 5 and Figure 6 As shown, in this embodiment, the autonomous driving decision model is referred to as an intelligent agent, and the method includes:

[0194] Step 610: Build a driving simulator for autonomous driving in a high-interaction scenario based on historical driving data; define the vehicle state s, driving action a, and reward function r of the intelligent agent.

[0195] Step 620: Initialize the experience replay pool, which is used to store interaction data; initialize the network model parameters of the agent, including initializing the initial embodied world model and the initial behavior model.

[0196] Step 630: Enable the agent to interact with the driving simulator and save the interaction data as (at ,s t ,r t ,s t+1 ,c t ) in the experience replay pool as model training samples.

[0197] Step 640: randomly collect a batch of training samples from the experience replay pool, use these samples to train the initial embodied world model and update the model parameters, so that the embodied world model can model the driving scene from an individual perspective in the latent space and predict the reward.

[0198] Step 650: Use the attention-based behavior model to interact with the embodied world model to generate driving trajectories imagined in the latent space, and use these driving trajectories as samples required for updating the behavior model to train the behavior model and update the model parameters. The behavior model includes a cross-attention mechanism and a controller-evaluator module.

[0199] Step 660: determine whether the preset training termination condition is met, for example, determine whether the current number of interaction steps reaches a set threshold, that is, the maximum number of interaction steps. If so, execute the next step, otherwise repeat steps 630 to 650.

[0200] Step 670: After the training is completed, the last updated agent network parameters are obtained as the training results.

[0201] In the autonomous driving decision model training method provided in the embodiment of the present application, the intelligent agent uses the behavior model to interact with the driving simulator and stores experience data in the experience replay pool; the embodied world model samples from the experience replay pool and trains and updates its own network parameters; the behavior model is trained and updated in the imagination of the embodied world model; and a new round of interaction with the driving simulator is performed using the updated behavior model, which can effectively improve the learning efficiency of the intelligent agent based on the model reinforcement learning algorithm and enhance the policy performance of the intelligent agent output.

[0202] The following is a description of the autonomous driving decision model training device provided in an embodiment of the present application. The autonomous driving decision model training device described below and the autonomous driving decision model training method described above can be referenced to each other.

[0203] Figure 7 A schematic diagram of the structure of the autonomous driving decision model training device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device includes an acquisition module 710, a modeling module 720, a training module 730 and a termination module 740.

[0204] An acquisition module is used to acquire the interaction data between the initial autonomous driving main vehicle model and the environment vehicle, where the initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model;

[0205] A modeling module is used to use the interaction data as training samples of the initial embodied world model, so that the initial embodied world model can model various types of environment vehicles based on the training samples;

[0206] A training module, used to imagine the driving trajectory of the initial autonomous driving main vehicle model corresponding to the autonomous driving vehicle based on the modeling results and the initial behavior model, and train the initial behavior model based on the driving trajectory;

[0207] The termination module is used to terminate the training when the current training reaches the preset termination condition and obtain the autonomous driving decision model;

[0208] Among them, the types of environmental vehicles are divided based on the interaction distance between the environmental vehicles and the autonomous driving vehicles, and the modeling methods of various types of environmental vehicles are different.

[0209] Specifically, according to an embodiment of the present application, any multiple modules among the acquisition module, the modeling module, the training module and the termination module can be combined into one module for implementation, or any one of the modules can be split into multiple modules.

[0210] Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module.

[0211] According to an embodiment of the present application, at least one of the acquisition module, modeling module, training module and termination module can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them.

[0212] Alternatively, at least one of the acquisition module, the modeling module, the training module and the termination module may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0213] The autonomous driving decision model training device provided in the embodiment of the present application obtains interaction data, classifies environmental vehicles, and models driving scenarios from the individual perspective of traffic participants, so that various types of environmental vehicles can be distinguished during the training process of the initial autonomous driving main vehicle model. This can not only effectively improve the training efficiency of the initial autonomous driving main vehicle model, but also improve the decision-making efficiency and accuracy of the autonomous driving decision model.

[0214] In some embodiments, the acquisition module is specifically used to:

[0215] Create a driving simulator based on historical driving data and run the driving simulator in a set container to simulate virtual driving scenarios;

[0216] The initial autonomous driving main vehicle model outside the container interacts with the environment vehicle in the virtual driving scene simulated by the driving simulator inside the container to obtain interaction data, which includes the vehicle state, driving action, reward, vehicle state at the next moment and round termination signal of the current vehicle at the current moment.

[0217] In some embodiments, the acquisition module includes a creation submodule, and the creation submodule is specifically used to:

[0218] Create a coordinate system based on the autonomous driving vehicle, and obtain the trajectory information of the autonomous driving vehicle and various environmental vehicles in the coordinate system during the historical time period;

[0219] Among them, the trajectory information includes the coordinates of the autonomous driving vehicle and various environmental vehicles in the coordinate system, as well as the orientation in the coordinate system; the vehicle state includes the trajectory information;

[0220] The driving action includes the longitudinal target speed selected by the initial autonomous driving host vehicle model.

[0221] In some embodiments, the interaction data also includes a reward function, the reward function includes a safety reward r safe and efficiency reward efficient :

[0222] Safety Reward safe The calculation formula is: safe =-d1*(1+v norm );

[0223] Efficiency Reward efficient The calculation formula is: efficient =d2*v norm -d3;

[0224] Among them, v norm is the normalized speed value of the autonomous driving vehicle; d1, d2 and d3 are all set fixed values; when the autonomous driving vehicle collides with the environment vehicle, the safety reward takes effect.

[0225] In some embodiments, the modeling module is specifically configured to:

[0226] Initialize the preset experience replay pool, which is used to store interaction data;

[0227] A set of interaction data is randomly collected from the experience replay pool as training samples;

[0228] In the case of a storage space alarm in the experience replay pool, the interaction data in the experience replay pool is processed based on the storage time of the interaction data in the experience replay pool.

[0229] In some embodiments, the modeling module is further specifically configured to:

[0230] Set a shared encoder, which is used to extract the underlying features in the training samples;

[0231] Set up the initial autonomous driving main vehicle model and the branch network of various environment vehicles;

[0232] Model various environmental vehicles and autonomous vehicles based on shared encoders and various branch networks;

[0233] Among them, each branch network includes a branch encoder, a gated recurrent unit and an observation reconstructor. The branch encoder is used to encode the vehicle state based on the vehicle category, the gated recurrent unit is used to update the vehicle state, and the observation reconstructor is used to reconstruct the vehicle state; the vehicles include autonomous driving vehicles and various environmental vehicles.

[0234] In some embodiments, the modeling module includes a reconstruction submodule, and the reconstruction submodule is specifically used to:

[0235] Update the deterministic latent state of any vehicle at the current moment based on the deterministic latent state of any vehicle at the previous moment, the stochastic latent state of any vehicle at the previous moment, and the action of any vehicle at the previous moment;

[0236] Obtain the random latent state of any vehicle at the current moment based on a priori and a posteriori methods;

[0237] Reconstructing the vehicle state of any vehicle based on the posterior of the deterministic latent state and the stochastic latent state;

[0238] The latent states of each vehicle are mixed based on the cross-attention mechanism, and the latent states include deterministic latent states and random latent states;

[0239] The embodied world model is trained based on the mixed latent states, the set predicted rewards, and the episode termination signal.

[0240] In some embodiments, the training module is specifically used to:

[0241] Based on the cross attention mechanism, the latent state of the direct interaction vehicle is identified in the mixed latent state; the environmental vehicles include the direct interaction vehicle and the potential interaction vehicle;

[0242] Based on the current potential state of each vehicle, the controller evaluator module in the initial behavior model is input to obtain the value estimate of the corresponding action at the current moment output by the controller evaluator module;

[0243] An initial behavior model is trained based on the value estimates.

[0244] It should be noted here that the autonomous driving decision model training device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned autonomous driving decision model training method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0245] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus (Communications Bus) 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic command in the memory 830 to execute the above method, which includes:

[0246] Obtaining interaction data between an initial autonomous driving vehicle model and an environment vehicle, wherein the initial autonomous driving vehicle model includes an initial embodied world model and an initial behavior model;

[0247] The interaction data is used as training samples of the initial embodied world model, so that the initial embodied world model can model various types of environment vehicles based on the training samples;

[0248] Based on the modeling results and the initial behavior model, imagine the driving trajectory of the initial autonomous driving main vehicle model corresponding to the autonomous driving vehicle, and train the initial behavior model based on the driving trajectory;

[0249] When the current training reaches the preset termination condition, the training is terminated to obtain the autonomous driving decision model;

[0250] Among them, the types of environmental vehicles are divided based on the interaction distance between the environmental vehicles and the autonomous driving vehicles, and the modeling methods of various types of environmental vehicles are different.

[0251] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software function modules and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0252] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0253] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0254] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0255] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0256] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0257] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0258] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an autonomous driving decision model, characterized in that: include: Acquire interaction data between an initial autonomous driving main vehicle model and an environment vehicle, wherein the initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model; Using the interaction data as training samples of the initial embodied world model, so that the initial embodied world model respectively models various types of environment vehicles based on the training samples; Imagining a driving trajectory of the automatic driving vehicle corresponding to the initial automatic driving main vehicle model based on the modeling result and the initial behavior model, and training the initial behavior model based on the driving trajectory; When the current training reaches a preset termination condition, the training is terminated to obtain the autonomous driving decision model; Among them, the type of the environmental vehicle is divided based on the interaction distance between the environmental vehicle and the autonomous driving vehicle, and the modeling methods of various types of environmental vehicles are different. The embodied world model is used to reconstruct the driving scene from the perspective of individual traffic participants, and provide the latent state of individual traffic participants to the behavior model.

2. The autonomous driving decision model training method according to claim 1, characterized in that: The obtaining of interaction data between the initial autonomous driving main vehicle model and the environment vehicle includes: Creating a driving simulator based on historical driving data, and running the driving simulator in a set container to simulate a virtual driving scenario; The initial autonomous driving main vehicle model outside the container interacts with the environment vehicle in the virtual driving scene simulated by the driving simulator in the container to obtain interaction data, wherein the interaction data includes the vehicle state, driving action, reward, vehicle state at the next moment and round termination signal of the current vehicle at the current moment.

3. The autonomous driving decision model training method according to claim 2, characterized in that: The obtaining of the interaction data includes: Create a coordinate system based on the autonomous driving vehicle, and obtain trajectory information of the autonomous driving vehicle and various environmental vehicles in the coordinate system within a historical time period; The trajectory information includes the coordinates of the autonomous driving vehicle and various environmental vehicles in the coordinate system, and their orientations in the coordinate system; the vehicle state includes the trajectory information; The driving action includes a longitudinal target speed selected by the initial automatic driving host vehicle model.

4. The autonomous driving decision model training method according to claim 2, characterized in that: The interaction data also includes a reward function, wherein the reward function includes a safety reward and efficiency rewards : The safety reward The calculation formula is: ; The efficiency reward The calculation formula is: ; in, is the speed value of the automatic driving vehicle after normalization; and are all set fixed values; when the autonomous driving vehicle collides with the environmental vehicle, the safety reward takes effect.

5. The autonomous driving decision model training method according to claim 1, characterized in that: The using the interaction data as training samples of the initial embodied world model includes: Initializing a preset experience replay pool, where the experience replay pool is used to store the interaction data; Randomly collecting a group of interaction data in the experience replay pool as the training sample; Wherein, in the case of a storage space alarm in the experience replay pool, the interaction data in the experience replay pool is processed based on the storage time of the interaction data in the experience replay pool.

6. The autonomous driving decision model training method according to claim 2, characterized in that: The modeling of various types of environment vehicles based on the training samples includes: Setting a shared encoder, wherein the shared encoder is used to extract underlying features in the training sample; Setting the initial autonomous driving main vehicle model and a branch network of various environmental vehicles; Modeling various types of environment vehicles and the autonomous driving vehicle based on the shared encoder and each branch network; Among them, each branch network includes a branch encoder, a gated recurrent unit and an observation reconstructor, the branch encoder is used to encode the vehicle state based on the vehicle category, the gated recurrent unit is used to update the vehicle state, and the observation reconstructor is used to reconstruct the vehicle state; the vehicles include the autonomous driving vehicles and various types of environmental vehicles.

7. The autonomous driving decision model training method according to claim 6, characterized in that: The modeling of various environmental vehicles and the autonomous driving vehicle includes: Updating the deterministic latent state of any vehicle at the current moment based on the deterministic latent state of any vehicle at the previous moment, the stochastic latent state of the previous moment, and the action of the previous moment; Obtaining the random latent state of any vehicle at the current moment based on a priori and a posteriori methods; Reconstructing the vehicle state of any one of the vehicles based on the deterministic latent state and the posterior of the stochastic latent state; Mixing the latent states of each vehicle based on a cross-attention mechanism, wherein the latent states include the deterministic latent state and the stochastic latent state; The embodied world model is trained based on the mixed latent state, the set predicted reward and the episode termination signal.

8. The autonomous driving decision model training method according to claim 7, characterized in that: The predicting the driving trajectory of the initial automatic driving main vehicle model based on the modeling result and the initial behavior model, and training the initial behavior model based on the driving trajectory, includes: Identifying the latent state of the direct interaction vehicle in the mixed latent state based on a cross attention mechanism; the environment vehicle includes the direct interaction vehicle and the potential interaction vehicle; Based on the current potential state of each vehicle, the controller evaluator module in the initial behavior model is input to obtain the value estimation of the corresponding action at the current moment output by the controller evaluator module; The initial behavior model is trained based on the value estimates.

9. An automatic driving decision model training device, characterized in that: include: An acquisition module, used to acquire interaction data between an initial autonomous driving main vehicle model and an environment vehicle, wherein the initial autonomous driving main vehicle model includes an initial embodied world model and an initial behavior model; A modeling module, configured to use the interaction data as training samples of the initial embodied world model, so that the initial embodied world model respectively models various types of environment vehicles based on the training samples; A training module, configured to imagine a driving trajectory of the automatic driving vehicle corresponding to the initial automatic driving main vehicle model based on the modeling result and the initial behavior model, and to train the initial behavior model based on the driving trajectory; A termination module, used to terminate the training when the current training reaches a preset termination condition, and obtain the autonomous driving decision model; Among them, the type of the environmental vehicle is divided based on the interaction distance between the environmental vehicle and the autonomous driving vehicle, and the modeling methods of various types of environmental vehicles are different. The embodied world model is used to reconstruct the driving scene from the perspective of individual traffic participants, and provide the latent state of individual traffic participants to the behavior model.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the autonomous driving decision model training method as described in any one of claims 1 to 7 is implemented.

Images