Driving training teaching method and system based on imitation learning

Abstract

The present invention provides a driving training teaching method and system based on imitation learning, which relates to the field of intelligent driving technology, including obtaining driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extracting driving behavior features in the driving behavior demonstration data; calculating a first feature expectation of an actual driving behavior trajectory of the human driver based on the driving behavior demonstration data; optimizing the driving strategy of an intelligent agent using a reinforcement learning algorithm, and calculating a second feature expectation by obtaining a simulated driving behavior trajectory of the intelligent agent by executing the driving strategy in a simulated environment; updating the parameters of the implicit reward function by minimizing the difference between the first feature expectation and the second feature expectation and combining a generative adversarial network to alternately optimize; and using the intelligent agent driving strategy for the assisted driving decision of the intelligent agent.

Description

Technical Field

[0001] The present invention relates to intelligent driving technology, and in particular to a driving training teaching method and system based on imitation learning. Background Art

[0002] Traditional driving training mainly adopts manual teaching, where experienced coaches instruct students in actual road environments. This training method has the following shortcomings:

[0003] (1) Low training efficiency and difficult to ensure teaching quality. Due to the complexity and variability of roads and traffic environments, it is difficult for coaches to provide systematic and comprehensive guidance to trainees, and the learning effect of trainees depends largely on the experience and level of the coaches.

[0004] (2) The learning process is monotonous and it is difficult to gain practical experience. Trainees can only be exposed to limited scenarios during road practice, making it difficult to improve their driving skills comprehensively. It is even more difficult for trainees to gain practical experience in some dangerous working conditions and extreme weather.

[0005] (3) There are great safety risks and high training costs. Training on actual roads has great safety risks. Once an accident occurs, it will cause casualties and economic losses. In addition, road training also requires a lot of vehicle and fuel costs.

[0006] In order to overcome the above problems, people have begun to explore the use of computer vision and machine learning technologies to build intelligent driving training systems. Among them, simulators based on virtual reality (VR) can provide trainees with a realistic driving environment and support repeated practice in a variety of scenarios. However, traditional VR simulators mainly rely on preset traffic scenes and rules, and are difficult to cope with complex and changing real driving environments. Summary of the invention

[0007] The embodiments of the present invention provide a driving training teaching method and system based on imitation learning, which can solve the problems in the prior art.

[0008] According to a first aspect of the embodiments of the present invention,

[0009] Provide driving training teaching methods based on imitation learning, including:

[0010] Acquire driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extract driving behavior features from the driving behavior demonstration data; calculate a first feature expectation of an actual driving behavior trajectory of the human driver based on the driving behavior demonstration data;

[0011] According to the parameters of the initialized implicit reward function, a reinforcement learning algorithm is used to optimize the driving strategy of the intelligent agent, a simulated driving behavior trajectory of the intelligent agent is obtained by the intelligent agent executing the driving strategy in a simulated environment, and a second feature expectation of the simulated driving behavior trajectory of the intelligent agent is calculated based on the simulated driving behavior trajectory; the parameters of the implicit reward function are updated by minimizing the difference between the first feature expectation and the second feature expectation and combining a generative adversarial network to alternately optimize and update the parameters of the implicit reward function, and the process of updating the parameters of the implicit reward function is repeated until the parameters of the implicit reward function converge or a preset number of iterations is reached, thereby obtaining the driving strategy of the intelligent agent when the iteration is terminated;

[0012] The intelligent agent driving strategy is used for the assisted driving decision of the intelligent agent, so that the intelligent agent can make driving decisions similar to those of human drivers according to the learned implicit reward function when facing a new driving scenario.

[0013] In an optional embodiment,

[0014] Updating the parameters of the implicit reward function by minimizing the difference between the first feature expectation and the second feature expectation and combining the generation adversarial network to alternately optimize includes:

[0015] Introducing a probability distribution distance as an optimization target, measuring the difference between the feature distribution of the actual driving behavior trajectory and the feature distribution of the simulated driving behavior trajectory by the probability distribution distance, and replacing the difference between the first feature expectation and the second feature expectation by minimizing the probability distribution distance;

[0016] Constructing a discriminator network, and enabling the discriminator network to distinguish the feature distribution of the actual driving behavior trajectory from the feature distribution of the simulated driving behavior trajectory by training the discriminator network to maximize the discriminator objective function, and outputting estimated values ​​of probability distribution distances representing the feature distribution of the actual driving behavior trajectory and the feature distribution of the simulated driving behavior trajectory respectively;

[0017] Alternately optimizing the parameters of the implicit reward function and the parameters of the discriminator network, fixing the parameters of the implicit reward function, optimizing the parameters of the discriminator network, so that the discriminator network can accurately estimate the probability distribution distance; fixing the parameters of the discriminator network, optimizing the parameters of the implicit reward function, and updating the parameters of the implicit reward function by minimizing the weighted sum of the estimated value of the probability distribution distance and the agent strategy entropy, so that the characteristic distribution of the simulated driving behavior trajectory approaches the characteristic distribution of the actual driving behavior trajectory;

[0018] The alternating optimization step is repeatedly performed until the parameters of the implicit reward function converge or a preset number of iterations is reached.

[0019] In an optional embodiment,

[0020] Replacing the difference between the first feature expectation and the second feature expectation by minimizing the probability distribution distance comprises:

[0021] ;

[0022] in, Indicates that given parameters The loss function value under The characteristic distribution representing the actual driving behavior trajectory and simulated driving behavior trajectory The probability distribution distance between the feature distributions of represents the balance factor, represents the agent strategy entropy;

[0023] Minimizing the weighted sum of the estimated value of the probability distribution distance and the agent's policy entropy includes:

[0024] ;

[0025] in, Indicates that given parameters The loss function value under represents the agent strategy entropy, represents the transpose of the parameters of the implicit reward function, represents the characteristic function, which is used to map the state s to the feature space, , Respectively in the strategy and expert strategies Expectations of the following characteristics.

[0026] In an optional embodiment,

[0027] The intelligent agent driving strategy is used for the intelligent agent's assisted driving decision, so that the intelligent agent can make driving decisions similar to those of a human driver according to the learned implicit reward function when facing a new driving scenario, including:

[0028] The vehicle status information and environmental perception information are obtained through vehicle-mounted sensors. The status information includes vehicle speed, acceleration and steering angle. The environmental perception information includes relative position and speed with respect to surrounding vehicles, relative position and angle with respect to road boundaries, and traffic sign and marking information.

[0029] Inputting the state information and the environment perception information into the learned intelligent agent driving strategy model to generate a set of candidate driving decision behaviors, wherein the candidate driving decision behaviors include acceleration, deceleration and steering;

[0030] Using the learned implicit reward function, the instantaneous reward value and the expected long-term cumulative reward value of each candidate driving decision behavior in the current state are calculated. The instantaneous reward value measures the rationality and safety of the candidate driving decision behavior at the current moment, and the expected long-term cumulative reward value measures the impact of the candidate driving decision behavior on vehicle driving in the future.

[0031] When calculating the expected value of the long-term cumulative reward, the Monte Carlo tree search algorithm is used to estimate the expected return of each decision node by performing multiple random simulations on the decision tree; based on the instant reward value, the candidate driving decision behavior with the highest expected value of the overall reward is selected as the optimal driving decision behavior;

[0032] The selected optimal driving decision behavior is converted into specific vehicle control instructions, including throttle, brake and steering, and real-time control of the vehicle is achieved through on-board actuators.

[0033] In an optional embodiment,

[0034] When calculating the expected value of long-term cumulative rewards, the Monte Carlo tree search algorithm is used to estimate the expected return of each decision node by performing multiple random simulations on the decision tree, including:

[0035] The process of calculating the expected value of long-term cumulative rewards is modeled as a decision tree, where each node represents a decision state and each edge represents a decision behavior;

[0036] The decision tree is updated by iteratively performing four steps of selection, expansion, simulation and backtracking;

[0037] In the selection step, starting from the root node, the child node corresponding to the root node is recursively selected by the upper confidence bound algorithm;

[0038] In the expansion step, the selected child node is randomly expanded to a new node as the new expansion node;

[0039] In the simulation step, starting from the new extended node, a random default strategy is executed until a termination state or a predetermined simulation depth is reached, and the accumulated reward obtained by the simulation is used as an initial estimate of the value of the new extended node;

[0040] In the backtracking step, the simulated cumulative reward is propagated back to each node in the decision tree, and the average reward estimate and the number of visits of the node are updated;

[0041] The above four steps are iteratively executed until the decision tree gradually converges to the optimal decision path.

[0042] In an optional embodiment,

[0043] Starting from the root node, the child nodes corresponding to the root node are recursively selected by the upper confidence bound algorithm, including:

[0044] ;

[0045] in, represents the upper confidence interval value of node j, represents the average reward estimate of node j, C The constant that represents the degree of exploration is called the exploration coefficient. N Indicates the total number of times the parent node of node j is visited, M represents the number of nodes, Indicates the number of times node j is visited.

[0046] According to a second aspect of the embodiments of the present invention,

[0047] Provide a driving training teaching system based on imitation learning, including:

[0048] A first unit is used to obtain driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extract driving behavior features in the driving behavior demonstration data; and calculate a first feature expectation of an actual driving behavior trajectory of the human driver based on the driving behavior demonstration data;

[0049] The second unit is used to optimize the driving strategy of the intelligent agent using a reinforcement learning algorithm according to the parameters of the initialized implicit reward function, obtain a simulated driving behavior trajectory of the intelligent agent by executing the driving strategy in a simulated environment, and calculate a second feature expectation of the simulated driving behavior trajectory of the intelligent agent based on the simulated driving behavior trajectory; minimize the difference between the first feature expectation and the second feature expectation and alternately optimize and update the parameters of the implicit reward function in combination with a generative adversarial network, and repeatedly execute the process of updating the parameters of the implicit reward function until the parameters of the implicit reward function converge or reach a preset number of iterations, thereby obtaining the driving strategy of the intelligent agent when the iteration is terminated;

[0050] The third unit is used to apply the intelligent agent driving strategy to the assisted driving decision of the intelligent agent, so that the intelligent agent can make driving decisions similar to those of human drivers according to the learned implicit reward function when facing a new driving scenario.

[0051] According to a third aspect of the embodiments of the present invention,

[0052] An electronic device is provided, comprising:

[0053] processor;

[0054] a memory for storing processor-executable instructions;

[0055] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0056] A fourth aspect of the embodiments of the present invention is:

[0057] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0058] This application constructs a driving strategy of an intelligent agent optimized by reinforcement learning, as well as the expected driving behavior trajectory characteristics of the intelligent agent in a simulated environment under this strategy. By comparing the expected simulated trajectory characteristics of the intelligent agent with the expected trajectory characteristics of a human driver, the imitation performance of the intelligent agent's driving strategy can be evaluated, that is, to what extent the intelligent agent has learned and reproduced the driving behavior pattern of a human driver.

[0059] By introducing the probability distribution distance as the optimization target and combining the idea of ​​generative adversarial networks, alternately optimizing the discriminator network and the implicit reward function, we can more effectively narrow the gap between the simulated driving behavior of the agent and the real behavior of human drivers, thereby learning a more realistic and reasonable driving strategy. Compared with the traditional feature expectation matching method, this inverse reinforcement learning method based on distribution matching can more comprehensively characterize the feature distribution of the behavior trajectory and improve the accuracy and robustness of imitation learning.

[0060] Applying the implicit reward function and agent driving strategy obtained based on inverse reinforcement learning to the actual assisted driving system can enable the agent to autonomously generate reasonable driving decisions based on the environmental state and vehicle state when facing new driving scenarios, and control the vehicle to perform smooth and safe driving behaviors. This end-to-end assisted driving method avoids explicit modeling of the driving environment and rules, and directly learns implicit driving preferences and strategies from the demonstration data of human drivers, with stronger adaptability and generalization capabilities. At the same time, the Monte Carlo tree search algorithm is introduced to estimate the long-term impact of decisions, making the driving decisions of the agent more forward-looking and globally optimal. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a flow chart of a driving training teaching method based on imitation learning according to an embodiment of the present invention;

[0062] Figure 2 The figure is a schematic diagram of the structure of a driving training teaching system based on imitation learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0065] Figure 1 FIG. 1 is a flow chart of a driving training teaching method based on imitation learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0066] S101. Acquire driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extract driving behavior features in the driving behavior demonstration data; calculate a first feature expectation of an actual driving behavior trajectory of a human driver based on the driving behavior demonstration data;

[0067] For example, in order to learn intelligent driving strategies from the actual driving experience of human drivers, it is first necessary to obtain driving behavior demonstration data of multiple human drivers in various driving scenarios and extract key driving behavior features from them. Then, the extracted features are used to calculate the expected driving behavior trajectory of human drivers in a given scenario as a reference standard for the intelligent agent to learn and optimize driving strategies. The specific steps are as follows:

[0068] Build a driving behavior data collection platform. In a real road environment, select multiple experienced human drivers and install data recording devices on their vehicles, such as on-board cameras, GPS locators, inertial measurement units (IMUs) and other sensors. These devices can record the driver's operating behavior (such as steering, acceleration, braking, etc.) and the vehicle's motion status (such as position, speed, acceleration, etc.) in real time.

[0069] Collect multi-scenario driving data. Require human drivers to drive in different driving scenarios (such as highways, urban roads, rural roads, etc.), different traffic flows and weather conditions. Collect data for at least 30 minutes continuously for each driving scenario to ensure data diversity and coverage.

[0070] Preprocess the collected driving data. Perform preprocessing operations such as time synchronization, coordinate system 1, and noise filtering on the collected raw sensor data to improve data quality. Divide the continuous sensor data into a series of driving state sequences, each of which contains information such as the vehicle's position, speed, acceleration, heading angle, and the driver's operation signals (such as throttle, brake, steering, etc.).

[0071] Extract driving behavior features. From the preprocessed driving state sequence, extract key features that reflect the driver's decision-making behavior, such as:

[0072] Average vehicle speed: reflects the driver’s preference and control ability for speed;

[0073] Average acceleration: reflects the driver's driving style (such as smooth, aggressive, etc.);

[0074] Average lateral deviation: reflects the driver's ability to control the lateral position of the vehicle;

[0075] Average heading angle deviation: reflects the stability of the driver's control over the vehicle direction;

[0076] Average accelerator and brake pedal pressure: reflects the driver's habit of longitudinal control of the vehicle;

[0077] Average steering wheel angle: reflects the driver's habit of lateral control of the vehicle.

[0078] Through statistical analysis of a large amount of driving data, we can obtain the driving behavior characteristic distribution of different drivers in different scenarios.

[0079] Calculate the expected driving behavior trajectory. For a given driving scenario (such as a specific road type, traffic flow, and weather conditions), use the extracted driving behavior features to calculate the average driving behavior trajectory of human drivers in the scenario as the target expectation for the agent to learn. Specifically, the driving state sequences of all drivers in the scenario can be aligned by time step, and then the mean and variance of all driver states can be calculated at each time step to obtain an average driving behavior trajectory and its confidence interval.

[0080] For example, in a certain urban road scenario, there are demonstration data of 10 drivers. For the tth time step, the speeds of these 10 drivers are: 25.1km / h, 27.8km / h, 24.6km / h, ..., 26.3km / h, with an average of 26.2km / h and a variance of 1.5km / h. Similarly, the average position, average acceleration and other state quantities of the vehicle at this time step can be calculated. By connecting the average state quantities of all time steps, the average driving behavior trajectory of human drivers in this scenario is obtained.

[0081] Smooth and interpolate the average driving behavior trajectory. Since the time step of the collected data does not necessarily match the time step of the agent training and decision-making, the calculated average trajectory needs to be processed. You can use methods such as spline interpolation to make discrete trajectory points continuous and adjust the time resolution of the trajectory as needed. In addition, in order to remove high-frequency oscillations and noise in the trajectory, you can apply smoothing filters to the trajectory, such as moving average filtering or Kalman filtering.

[0082] After the above steps, the characteristic statistics of human drivers' driving behaviors and the average driving behavior trajectory in a given driving scenario are obtained. These statistical information reflects the control preferences and decision-making habits of human drivers for vehicle states in actual driving, and can serve as an important reference for the intelligent driving system to learn and optimize driving strategies. In the subsequent imitation learning and reinforcement learning process, the intelligent agent will take the behavioral characteristics and trajectory expectations of these human drivers as the target, and through continuous trial and error and optimization, gradually master driving skills close to those of human drivers, and achieve safe, smooth and efficient autonomous driving decisions.

[0083] S102. According to the parameters of the initialized implicit reward function, a reinforcement learning algorithm is used to optimize the driving strategy of the intelligent agent, a simulated driving behavior trajectory of the intelligent agent is obtained by the intelligent agent executing the driving strategy in a simulated environment, and a second feature expectation of the simulated driving behavior trajectory of the intelligent agent is calculated based on the simulated driving behavior trajectory; the parameters of the implicit reward function are updated by minimizing the difference between the first feature expectation and the second feature expectation and combining a generative adversarial network to alternately optimize and update the parameters of the implicit reward function, and the process of updating the parameters of the implicit reward function is repeated until the parameters of the implicit reward function converge or a preset number of iterations is reached, thereby obtaining the driving strategy of the intelligent agent when the iteration is terminated;

[0084] For example, after obtaining the driving behavior characteristics and trajectory expectations of human drivers, the next step is to optimize the driving strategy of the agent through the reinforcement learning algorithm so that its driving behavior in the simulated environment is as close as possible to that of human drivers. At the same time, we need to calculate the characteristic expectations of the driving behavior trajectory obtained by the agent executing the current strategy in the simulated environment, which are used for comparison and correction with the trajectory expectations of human drivers. The specific steps are as follows:

[0085] Step 1: Initialize the parameters of the implicit reward function. The implicit reward function is a function used to evaluate the similarity between the agent's driving behavior and human driving behavior, and can be in the form of a deep neural network. The input of the network is the state-behavior pair of the agent (such as speed, acceleration, steering angle, etc.), and the output is a scalar reward value indicating the degree of similarity between the state-behavior pair and human driving behavior. Initially, the network parameters of the implicit reward function can be randomly initialized or use pre-trained parameters.

[0086] Step 2: Build a driving simulation environment. Use a high-fidelity driving simulator (such as CARLA, SUMO, etc.) to build a virtual environment similar to the real road scene, including road network, traffic flow, weather conditions and other elements. The simulation environment should be able to respond to the driving behavior of the agent and return the state information of the agent in real time.

[0087] Step 3: Initialize the driving strategy of the agent. The driving strategy of the agent can also be represented by a deep neural network. The input of the network is the state of the agent, and the output is the driving behavior of the agent (such as throttle, brake, steering, etc.). Initially, the driving strategy network can use random parameters or pre-trained parameters.

[0088] Step 4: The agent executes the current driving strategy in the simulated environment. In each simulated episode, the agent starts from the initial state, generates driving behavior based on the current state and driving strategy network, and executes the behavior to interact with the environment to obtain the state at the next moment and the output of the implicit reward function (i.e., behavior score). This process is repeated until the episode ends (such as reaching the target location or a collision occurs, etc.).

[0089] Step 5: Calculate the driving behavior trajectory features of the agent in the current episode. For each time step in the episode, record the state (position, speed, acceleration, heading angle, etc.) and behavior (throttle, brake, steering, etc.) of the agent to form a complete driving behavior trajectory. Then, in the same way as human driving behavior feature extraction, calculate the average speed, average acceleration, average lateral offset and other feature quantities on the trajectory.

[0090] Step 6: Calculate the average driving behavior trajectory characteristics of the agent over multiple episodes to obtain the second feature expectation of the simulated driving behavior trajectory of the agent. In order to reduce the random influence of a single episode, the agent is usually required to perform multiple episodes, record the driving behavior trajectory characteristics of each episode, and then calculate the mean and variance of all episode characteristics as the second feature expectation of the agent under the current strategy.

[0091] For example, assuming that the average speed of the agent in 10 episodes is 60.2km / h, 58.7km / h, 61.5km / h, ..., 59.6km / h, then the average speed of the agent under the current strategy is expected to be 60.1km / h, with a variance of 1.2km / h. Similarly, the expected value and confidence interval of the agent's other driving behavior characteristics can be calculated.

[0092] Step 7: Use reinforcement learning algorithms to optimize the agent's driving strategy and implicit reward function. Based on the state-behavior-reward data collected by the agent in the simulation environment, use reinforcement learning algorithms such as policy gradient, Q-learning, and Actor-Critic to adjust the parameters of the driving strategy network and implicit reward function network through gradient ascent or descent, so that the agent's driving behavior is closer to that of a human driver and obtains a higher implicit reward. Common optimization objectives include maximizing cumulative rewards, minimizing policy gradient variance, etc.

[0093] Step 8: Repeat steps 4-7 until the agent's driving strategy converges or reaches the preset performance index. In each iteration, the agent uses the updated driving strategy and implicit reward function to execute in the simulation environment, calculates the expected characteristics of the simulated trajectory, and further optimizes the strategy and reward function with the newly collected data. As the number of iterations increases, the driving behavior of the agent will gradually approach that of a human driver, and the expected characteristics of the simulated trajectory will become more and more similar to those of a human driver.

[0094] After the above steps, we obtained an intelligent driving strategy optimized by reinforcement learning, as well as the expected driving behavior trajectory characteristics of the intelligent agent in the simulated environment under this strategy. By comparing the expected simulated trajectory characteristics of the intelligent agent with the expected trajectory characteristics of human drivers, we can evaluate the imitation performance of the intelligent driving strategy, that is, to what extent the intelligent agent has learned and reproduced the driving behavior patterns of human drivers.

[0095] S103. Using the intelligent agent driving strategy for the assisted driving decision of the intelligent agent, so that when the intelligent agent faces a new driving scenario, it can make driving decisions similar to those of a human driver based on the learned implicit reward function.

[0096] In an optional embodiment,

[0097] Updating the parameters of the implicit reward function by minimizing the difference between the first feature expectation and the second feature expectation and combining the generation adversarial network to alternately optimize includes:

[0098] Introducing a probability distribution distance as an optimization target, measuring the difference between the feature distribution of the actual driving behavior trajectory and the feature distribution of the simulated driving behavior trajectory by the probability distribution distance, and replacing the difference between the first feature expectation and the second feature expectation by minimizing the probability distribution distance;

[0099] Constructing a discriminator network, and enabling the discriminator network to distinguish the feature distribution of the actual driving behavior trajectory from the feature distribution of the simulated driving behavior trajectory by training the discriminator network to maximize the discriminator objective function, and outputting estimated values ​​of probability distribution distances representing the feature distribution of the actual driving behavior trajectory and the feature distribution of the simulated driving behavior trajectory respectively;

[0100] Alternately optimizing the parameters of the implicit reward function and the parameters of the discriminator network, fixing the parameters of the implicit reward function, optimizing the parameters of the discriminator network, so that the discriminator network can accurately estimate the probability distribution distance; fixing the parameters of the discriminator network, optimizing the parameters of the implicit reward function, and updating the parameters of the implicit reward function by minimizing the weighted sum of the estimated value of the probability distribution distance and the agent strategy entropy, so that the characteristic distribution of the simulated driving behavior trajectory approaches the characteristic distribution of the actual driving behavior trajectory;

[0101] The alternating optimization step is repeatedly performed until the parameters of the implicit reward function converge or a preset number of iterations is reached.

[0102] In an optional embodiment,

[0103] Replacing the difference between the first feature expectation and the second feature expectation by minimizing the probability distribution distance comprises:

[0104] ;

[0105] in, Indicates that given parameters The loss function value under The characteristic distribution representing the actual driving behavior trajectory and simulated driving behavior trajectory The probability distribution distance between the feature distributions of represents the balance factor, represents the agent strategy entropy;

[0106] Minimizing the weighted sum of the estimated value of the probability distribution distance and the agent's policy entropy includes:

[0107] ;

[0108] in, Indicates that given parameters The loss function value under represents the agent strategy entropy, represents the transpose of the parameters of the implicit reward function, represents the characteristic function, which is used to map the state s to the feature space, , Respectively in the strategy and expert strategies Expectations of the following characteristics.

[0109] For example, in order to make the driving behavior of the intelligent agent closer to the real behavior pattern of human drivers, it is necessary to optimize the parameters of the implicit reward function by minimizing the difference between the characteristic distribution of the simulated driving behavior trajectory of the intelligent agent and the characteristic distribution of the actual driving behavior trajectory of the human driver. Traditional inverse reinforcement learning methods usually directly compare the characteristic expectation values ​​of the two trajectories, but this approach ignores the overall difference in the characteristic distribution. In order to more accurately characterize the difference between the two trajectory distributions, this paper introduces the probability distribution distance as the optimization target, and combines the idea of ​​generative adversarial network (GAN) to alternately optimize the parameters of the implicit reward function and the discriminator network to achieve effective matching of trajectory distribution. The specific steps are as follows:

[0110] Introduce probability distribution distance measurement. The actual driving behavior trajectory characteristics of human drivers are represented as a probability distribution P_real, and the simulated driving behavior trajectory characteristics of the intelligent agent are represented as another probability distribution P_sim. Select a suitable probability distribution distance measurement method (such as Jensen-Shannon divergence, Wasserstein distance, etc.) to measure the difference between the two distributions. The optimization goal is changed from minimizing the difference in feature expected values ​​to minimizing the probability distribution distance D(P_real, P_sim).

[0111] Construct a discriminator network. Introduce a discriminator network D, whose input is a given trajectory feature and whose output is the probability that the trajectory feature comes from the actual distribution P_real. The discriminator network usually adopts a structure such as a multi-layer perceptron (MLP) or a convolutional neural network (CNN), and uses a sigmoid function as the activation function of the output layer to output a probability value between 0 and 1.

[0112] Train the discriminator network. Fix the parameters of the implicit reward function and optimize the parameters of the discriminator network by maximizing the discriminator objective function. Update the parameters of the discriminator network through the gradient ascent algorithm so that it can distinguish the feature distribution of the actual trajectory and the simulated trajectory as accurately as possible.

[0113] Optimize the parameters of the implicit reward function. Fix the parameters of the discriminator network and optimize the parameters of the implicit reward function by minimizing the weighted sum of the estimated value of the probability distribution distance and the agent's policy entropy. The estimated value of the probability distribution distance can be approximated by the output of the discriminator network, that is, the mean of D(P_real, P_sim) ≈ log(D(x_real))+ log(1 -D(x_sim)). The policy entropy term is used to encourage the agent's strategy to have a certain degree of randomness to avoid premature convergence to a suboptimal solution. The parameters of the implicit reward function are updated through the gradient descent algorithm, so that the simulated trajectory distribution generated by the agent under the guidance of the optimized reward function gradually approaches the real trajectory distribution.

[0114] Alternating optimization. Repeat the above steps to alternately optimize the parameters of the discriminator network and the implicit reward function until the preset number of iterations or convergence conditions are reached. During the optimization process, the discriminator network and the implicit reward function compete with each other to jointly promote the simulated trajectory distribution to continuously approach the real trajectory distribution.

[0115] The following is a specific data case to illustrate the above optimization process:

[0116] Suppose we extract 1,000 trajectories from the actual driving data of human drivers, each trajectory contains 10 time steps, and 5 feature dimensions (such as vehicle speed, acceleration, steering angle, etc.) are recorded at each time step. At the same time, we use the agent to generate 1,000 simulated trajectories in a simulated environment, and the feature dimensions are the same as the real trajectory.

[0117] We construct an MLP with 3 hidden layers (128 neurons per layer) as the discriminator network, with an input feature dimension of 50 (10 time steps × 5 feature dimensions) and an output probability value between 0 and 1. The implicit reward function is initialized as a linear function with a 50-dimensional weight vector and a scalar bias term as parameters.

[0118] In each round of iteration, we sampled 100 trajectories from the real trajectory and the simulated trajectory for training the discriminator network. The parameters of the discriminator network were updated using the Adam optimizer (learning rate 0.0001) by maximizing the discriminator objective function and training for 100 epochs.

[0119] Next, we sample 100 trajectories from the simulated environment, calculate their feature representations, and estimate the probability distribution distance through the current discriminator network. At the same time, we calculate the average policy entropy of these 100 trajectories under the current implicit reward function. The weighted sum of the estimated value of the probability distribution distance and the average policy entropy is used as the optimization target, and the parameters of the implicit reward function are updated using the Adam optimizer (learning rate 0.0001) for 100 epochs.

[0120] Repeat the above alternating optimization process until 1000 iterations are completed or convergence conditions are reached. During the optimization process, we can record the estimated value of the probability distribution distance after each iteration, as well as the difference in characteristic distribution between the simulated trajectory and the real trajectory generated by the agent under the current implicit reward function, to evaluate the convergence and effectiveness of the optimization algorithm.

[0121] By introducing the probability distribution distance as the optimization target and combining the idea of ​​generative adversarial networks, alternately optimizing the discriminator network and the implicit reward function, the difference between the simulated driving behavior of the intelligent agent and the real behavior of human drivers can be more effectively narrowed, thereby learning a more realistic and reasonable driving strategy. Compared with the traditional feature expectation matching method, this inverse reinforcement learning method based on distribution matching can more comprehensively characterize the feature distribution of the behavior trajectory and improve the accuracy and robustness of imitation learning.

[0122] In an optional embodiment,

[0123] The intelligent agent driving strategy is used for the intelligent agent's assisted driving decision, so that the intelligent agent can make driving decisions similar to those of a human driver according to the learned implicit reward function when facing a new driving scenario, including:

[0124] The vehicle status information and environmental perception information are obtained through vehicle-mounted sensors. The status information includes vehicle speed, acceleration and steering angle. The environmental perception information includes relative position and speed with respect to surrounding vehicles, relative position and angle with respect to road boundaries, and traffic sign and marking information.

[0125] Inputting the state information and the environment perception information into the learned intelligent agent driving strategy model to generate a set of candidate driving decision behaviors, wherein the candidate driving decision behaviors include acceleration, deceleration and steering;

[0126] Using the learned implicit reward function, the instantaneous reward value and the expected long-term cumulative reward value of each candidate driving decision behavior in the current state are calculated. The instantaneous reward value measures the rationality and safety of the candidate driving decision behavior at the current moment, and the expected long-term cumulative reward value measures the impact of the candidate driving decision behavior on vehicle driving in the future.

[0127] When calculating the expected value of the long-term cumulative reward, the Monte Carlo tree search algorithm is used to estimate the expected return of each decision node by performing multiple random simulations on the decision tree; based on the instant reward value, the candidate driving decision behavior with the highest expected value of the overall reward is selected as the optimal driving decision behavior;

[0128] The selected optimal driving decision behavior is converted into specific vehicle control instructions, including throttle, brake and steering, and real-time control of the vehicle is achieved through on-board actuators.

[0129] For example, after completing the learning of the implicit reward function and optimizing the agent's driving strategy, we can apply the trained agent model to the actual assisted driving system. When the agent faces a new driving scenario, it can make driving decisions similar to those of a human driver based on the learned implicit reward function, thereby achieving a smooth, safe, and comfortable assisted driving function. The specific steps are as follows:

[0130] Obtain real-time driving data through on-board sensors. Various sensors installed on the vehicle, such as speed sensors, acceleration sensors, inertial measurement units (IMUs), lidars, millimeter-wave radars, surround-view cameras, etc., are used to collect vehicle status information and environmental perception information in real time. Status information includes current vehicle speed, acceleration, steering angle, etc. Environmental perception information includes relative position and speed with surrounding vehicles, relative position and angle with road boundaries, and traffic sign and marking information.

[0131] The collected real-time data is input into the intelligent driving policy model. The vehicle state information and environmental perception information are organized and preprocessed according to the predefined feature format to form a feature vector representing the current driving scene. The feature vector is input into the trained intelligent driving policy model to generate a set of candidate driving decision behaviors. These candidate decision behaviors usually include basic operations such as acceleration, deceleration and steering, as well as their different combinations and parameter changes.

[0132] The implicit reward function is used to evaluate the quality of candidate decisions. For each candidate driving decision behavior, the learned implicit reward function is used to calculate its immediate reward value and long-term cumulative reward expectation in the current state. The immediate reward value measures the rationality and safety of the decision behavior at the current moment, such as collision risk, comfort and other factors. The long-term cumulative reward expectation measures the impact of the decision behavior on vehicle driving in the future, such as average speed, fuel consumption, passenger experience and other factors.

[0133] The Monte Carlo tree search algorithm is used to estimate the expected value of long-term cumulative rewards. Due to the uncertainty of the future vehicle's driving trajectory, it is difficult to directly calculate the long-term cumulative rewards, so the Monte Carlo tree search (MCTS) algorithm is used for estimation. Specifically, starting from the current state, multiple random simulations are performed on the decision tree. Each simulation performs state transitions and reward calculations based on the candidate decision behavior and the environment transition probability until the preset search depth or termination state is reached. The average value of the cumulative reward values ​​obtained through multiple simulations is used as the expected value of the long-term cumulative reward for the candidate decision behavior in the current state.

[0134] Taking a simple lane keeping scenario as an example, assuming the current vehicle speed is 60km / h, the distance to the vehicle in front is 50m, and the lateral deviation from the centerline of the road is 0.3m. The intelligent driving policy model generates three candidate decisions: maintain the current speed and direction, accelerate to 65km / h and correct 0.1m to the right, and decelerate to 55km / h and correct 0.2m to the left.

[0135] Through the calculation of the implicit reward function, the immediate reward values ​​of the three candidate decisions are 0.8, 0.6, and 0.7, respectively, indicating that the first decision is the most reasonable and safe at the current moment. Then, the MCTS algorithm is used to perform 50 random simulations, each with 50 time steps, and the expected long-term cumulative rewards of the three candidate decisions in the next 5 seconds are estimated to be 8.2, 7.5, and 8.0, respectively.

[0136] Combine immediate rewards and long-term rewards to select the best decision and execute it. Perform a weighted sum of the immediate reward value and the expected value of the long-term cumulative reward to obtain a comprehensive evaluation score for each candidate driving decision behavior. Select the decision with the highest evaluation score as the optimal driving decision at the current moment. Convert the optimal decision into specific vehicle control instructions, such as throttle opening, brake pressure, steering angle, etc., and achieve real-time control of the vehicle through on-board actuators (such as electronic throttle, anti-lock braking system, electric power steering, etc.), so that the vehicle can smoothly and safely execute the decision behavior.

[0137] In the above lane keeping scenario, assuming that the weights of immediate reward and long-term reward are 0.6 and 0.4 respectively, the comprehensive evaluation scores of the three candidate decisions are 0.728, 0.66, and 0.72 respectively. The first decision (maintaining the current speed and direction) has the highest score and is therefore selected as the optimal decision. The agent converts this decision into a control command of 60% throttle opening and 0° steering angle, and controls the vehicle to drive smoothly in the current lane through the actuator.

[0138] Since the driving environment is dynamically changing, the agent needs to continuously perceive the state of the environment at a high frequency (e.g., 10 times per second), generate candidate decisions, evaluate the pros and cons of the decisions, select the best decision and execute it. Through this continuous decision-making execution process, the agent can make driving behaviors similar to those of human drivers based on the learned implicit reward function, achieve smooth, safe, and comfortable assisted driving functions, reduce the burden on the driver, and improve the driving experience.

[0139] In summary, applying the implicit reward function and agent driving strategy obtained based on inverse reinforcement learning to the actual assisted driving system can enable the agent to autonomously generate reasonable driving decisions based on the environmental state and vehicle state when facing new driving scenarios, and control the vehicle to perform smooth and safe driving behaviors. This end-to-end assisted driving method avoids explicit modeling of the driving environment and rules, and directly learns implicit driving preferences and strategies from the demonstration data of human drivers, with stronger adaptability and generalization capabilities. At the same time, the Monte Carlo tree search algorithm is introduced to estimate the long-term impact of decisions, making the driving decisions of the agent more forward-looking and globally optimal.

[0140] In an optional embodiment,

[0141] When calculating the expected value of long-term cumulative rewards, the Monte Carlo tree search algorithm is used to estimate the expected return of each decision node by performing multiple random simulations on the decision tree, including:

[0142] The process of calculating the expected value of long-term cumulative rewards is modeled as a decision tree, where each node represents a decision state and each edge represents a decision behavior;

[0143] The decision tree is updated by iteratively performing four steps of selection, expansion, simulation and backtracking;

[0144] In the selection step, starting from the root node, the child node corresponding to the root node is recursively selected by the upper confidence bound algorithm;

[0145] In the expansion step, the selected child node is randomly expanded to a new node as the new expansion node;

[0146] In the simulation step, starting from the new extended node, a random default strategy is executed until a termination state or a predetermined simulation depth is reached, and the accumulated reward obtained by the simulation is used as an initial estimate of the value of the new extended node;

[0147] In the backtracking step, the simulated cumulative reward is propagated back to each node in the decision tree, and the average reward estimate and the number of visits of the node are updated;

[0148] The above four steps are iteratively executed until the decision tree gradually converges to the optimal decision path.

[0149] In an optional embodiment,

[0150] Starting from the root node, the child nodes corresponding to the root node are recursively selected by the upper confidence bound algorithm, including:

[0151] ;

[0152] in, represents the upper confidence interval value of node j, represents the average reward estimate of node j, C The constant that represents the degree of exploration is called the exploration coefficient. N Indicates the total number of times the parent node of node j is visited, M represents the number of nodes, Indicates the number of times node j is visited.

[0153] For example, in the process of intelligent agent-assisted driving decision-making, in order to more accurately estimate the expected value of the long-term cumulative reward of each candidate driving decision behavior, this paper adopts the Monte Carlo Tree Search (MCTS) algorithm. MCTS is a heuristic search algorithm based on random sampling. It gradually estimates and optimizes the expected return of each decision node by performing multiple random simulations on the decision tree. The following is a detailed introduction to the specific steps and implementation details of the MCTS algorithm in calculating the expected value of the long-term cumulative reward.

[0154] Construct a decision tree. The calculation process of the expected value of the long-term cumulative reward is modeled as a decision tree. Each node of the decision tree represents a decision state, including the current vehicle state, the environment state, and the previously executed decision sequence. The child nodes of each node correspond to the optional decision behaviors in the current state, such as acceleration, deceleration, steering, etc. The edges of the decision tree represent the transition from the parent node state to the child node state, and also correspond to the state change after executing a certain decision behavior.

[0155] Iterate the four core steps of MCTS. The MCTS algorithm updates the decision tree and estimates the expected return of each node by repeatedly iterating the four steps of selection, expansion, simulation, and backpropagation.

[0156] Selection phase. Starting from the root node of the decision tree, recursively select the most promising child nodes until reaching a node or leaf node that is not fully expanded. The selection process uses the upper confidence bound (UCB) algorithm to weigh exploration and utilization, and selects child nodes that take into account the current benefit estimate and access frequency. The calculation formula for the UCB value is: UCB = node average reward estimate + exploration coefficient × square root (total number of parent node visits / number of node visits). Among them, the exploration coefficient controls the balance between exploring new nodes and utilizing existing estimates, and is usually taken as sqrt(2).

[0157] Expansion phase. If the selected node is a non-leaf node that has not been fully expanded, a child node of the node that has not been visited is randomly expanded and added to the decision tree. The expanded new node represents the new state reached after executing an optional decision behavior in the current state.

[0158] Simulation phase. Starting from the newly expanded node, a random default policy is executed, and random sampling and state transfer are performed in the decision space until the terminal state is reached or the preset maximum simulation depth is reached. The default policy usually adopts a simple random decision rule, such as randomly selecting a feasible decision behavior in each state. The reward value accumulated during the simulation process is used as the initial estimate of the value of the newly expanded node.

[0159] Backtracking phase. The accumulated reward value obtained by simulation is propagated back to each node in the decision tree, and the average reward estimate and the number of visits of the node are updated. For each node, its average reward estimate is: updated average reward estimate = (original average reward estimate × original number of visits + current simulation reward value) / (original number of visits + 1). At the same time, the number of visits to the node is increased by 1. Through backtracking updates, the node statistics in the decision tree are improved and can more accurately reflect the long-term value of each decision behavior.

[0160] Repeat the iteration until convergence. Repeat the selection, expansion, simulation and backtracking process to continuously update and optimize the decision tree until the preset number of iterations is met or the decision tree gradually converges to the optimal decision path. Through multiple iterations, the MCTS algorithm can find the optimal decision sequence in the search space of the decision tree and estimate the expected value of the long-term cumulative reward for each candidate driving decision behavior.

[0161] For example, suppose the agent is currently faced with three candidate driving decision behaviors: A, B, and C. We want to estimate the expected value of the long-term cumulative reward for each decision. First, build a decision tree with only a root node, where the root node represents the current state. Then, perform MCTS search iterations:

[0162] 1. Selection phase: According to the UCB algorithm, the most promising child node is selected, such as the child node corresponding to decision A.

[0163] 2. Expansion phase: Expand an unvisited child node of decision A, such as the new state A1 reached after executing decision A.

[0164] 3. Simulation phase: Starting from node A1, execute the random default strategy, perform 10 steps of simulation, and the cumulative reward value is 5.

[0165] 4. Backtracking phase: Backtrack the reward value 5 to update the statistics of node A and the root node in the decision tree.

[0166] Repeat the above iterative process several times, such as 50 times, and the search and evaluation results of the decision tree gradually converge. Finally, the average reward estimates of the three decisions at the root node are: decision A: 7.8, decision B: 6.5, decision C: 8.2, and the corresponding number of visits is: decision A: 20 times, decision B: 15 times, decision C: 15 times.

[0167] Therefore, through the MCTS algorithm, we get the estimated expected value of the long-term cumulative reward for each candidate driving decision behavior: decision A: 7.8, decision B: 6.5, decision C: 8.2. These estimated values ​​provide an important basis for the agent to choose the best decision, enabling it to take into account both short-term benefits and long-term interests and make more reasonable and globally optimal driving decisions.

[0168] In summary, the Monte Carlo tree search algorithm effectively solves the problem of calculating the expected value of long-term cumulative rewards by performing random simulation and statistical estimation on the decision tree. It can efficiently search in a massive decision space, weigh exploration and utilization, gradually optimize the decision tree, and finally obtain the long-term value estimate of each candidate driving decision behavior. This provides key evaluation information for the intelligent agent's assisted driving decision, enabling it to comprehensively consider immediate rewards and long-term benefits based on the learned implicit reward function, select the optimal driving decision, and achieve safe, smooth, and comfortable vehicle control.

[0169] Figure 2 FIG. 1 is a schematic diagram of the structure of a driving training teaching system based on imitation learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0170] A first unit is used to obtain driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extract driving behavior features in the driving behavior demonstration data; and calculate a first feature expectation of an actual driving behavior trajectory of the human driver based on the driving behavior demonstration data;

[0171] The second unit is used to optimize the driving strategy of the intelligent agent using a reinforcement learning algorithm according to the parameters of the initialized implicit reward function, obtain a simulated driving behavior trajectory of the intelligent agent by executing the driving strategy in a simulated environment, and calculate a second feature expectation of the simulated driving behavior trajectory of the intelligent agent based on the simulated driving behavior trajectory; minimize the difference between the first feature expectation and the second feature expectation and alternately optimize and update the parameters of the implicit reward function in combination with a generative adversarial network, and repeatedly execute the process of updating the parameters of the implicit reward function until the parameters of the implicit reward function converge or reach a preset number of iterations, thereby obtaining the driving strategy of the intelligent agent when the iteration is terminated;

[0172] The third unit is used to apply the intelligent agent driving strategy to the assisted driving decision of the intelligent agent, so that the intelligent agent can make driving decisions similar to those of human drivers according to the learned implicit reward function when facing a new driving scenario.

[0173] According to a third aspect of the embodiments of the present invention,

[0174] An electronic device is provided, comprising:

[0175] processor;

[0176] a memory for storing processor-executable instructions;

[0177] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0178] A fourth aspect of the embodiments of the present invention is:

[0179] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0180] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driving training teaching method based on imitation learning, characterized in that: include: Acquire driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extract driving behavior features from the driving behavior demonstration data; Calculating a first characteristic expectation of an actual driving behavior trajectory of a human driver based on the driving behavior demonstration data; According to the parameters of the initialized implicit reward function, a reinforcement learning algorithm is used to optimize the driving strategy of the intelligent agent, a simulated driving behavior trajectory of the intelligent agent is obtained by the intelligent agent executing the driving strategy in a simulated environment, and a second feature expectation of the simulated driving behavior trajectory of the intelligent agent is calculated based on the simulated driving behavior trajectory; the parameters of the implicit reward function are updated by minimizing the difference between the first feature expectation and the second feature expectation and combining a generative adversarial network to alternately optimize and update the parameters of the implicit reward function, and the process of updating the parameters of the implicit reward function is repeated until the parameters of the implicit reward function converge or a preset number of iterations is reached, thereby obtaining the driving strategy of the intelligent agent when the iteration is terminated; Using the intelligent agent driving strategy for the intelligent agent's assisted driving decision-making, so that the intelligent agent can make driving decisions similar to those of a human driver according to the learned implicit reward function when facing a new driving scenario; When calculating the expected value of long-term cumulative rewards, the Monte Carlo tree search algorithm is used to estimate the expected return of each decision node by performing multiple random simulations on the decision tree, including: The process of calculating the expected value of long-term cumulative rewards is modeled as a decision tree, where each node represents a decision state and each edge represents a decision behavior; The decision tree is updated by iteratively performing four steps of selection, expansion, simulation and backtracking; In the selection step, starting from the root node, the child node corresponding to the root node is recursively selected by the upper confidence bound algorithm; In the expansion step, the selected child node is randomly expanded to a new node as the new expansion node; In the simulation step, starting from the new extended node, a random default strategy is executed until a termination state or a predetermined simulation depth is reached, and the accumulated reward obtained by the simulation is used as an initial estimate of the value of the new extended node; In the backtracking step, the simulated cumulative reward is propagated back to each node in the decision tree, and the average reward estimate and the number of visits of the node are updated; The above four steps are iteratively executed until the decision tree gradually converges to the optimal decision path.

2. The method according to claim 1, characterized in that Updating the parameters of the implicit reward function by minimizing the difference between the first feature expectation and the second feature expectation and combining the generation adversarial network to alternately optimize includes: Introducing a probability distribution distance as an optimization target, measuring the difference between the feature distribution of the actual driving behavior trajectory and the feature distribution of the simulated driving behavior trajectory by the probability distribution distance, and replacing the difference between the first feature expectation and the second feature expectation by minimizing the probability distribution distance; Constructing a discriminator network, and enabling the discriminator network to distinguish the feature distribution of the actual driving behavior trajectory from the feature distribution of the simulated driving behavior trajectory by training the discriminator network to maximize the discriminator objective function, and outputting estimated values ​​of probability distribution distances representing the feature distribution of the actual driving behavior trajectory and the feature distribution of the simulated driving behavior trajectory respectively; Alternately optimizing the parameters of the implicit reward function and the parameters of the discriminator network, fixing the parameters of the implicit reward function, optimizing the parameters of the discriminator network, so that the discriminator network can accurately estimate the probability distribution distance; fixing the parameters of the discriminator network, optimizing the parameters of the implicit reward function, and updating the parameters of the implicit reward function by minimizing the weighted sum of the estimated value of the probability distribution distance and the agent strategy entropy, so that the characteristic distribution of the simulated driving behavior trajectory approaches the characteristic distribution of the actual driving behavior trajectory; The alternating optimization step is repeatedly performed until the parameters of the implicit reward function converge or a preset number of iterations is reached.

3. The method according to claim 2, characterized in that Replacing the difference between the first feature expectation and the second feature expectation by minimizing the probability distribution distance comprises: ； in, Indicates that given parameters The loss function value under The characteristic distribution representing the actual driving behavior trajectory and simulated driving behavior trajectory The probability distribution distance between the feature distributions of represents the balance factor, represents the agent strategy entropy; Minimizing the weighted sum of the estimated value of the probability distribution distance and the agent's policy entropy includes: ； in, Indicates that given parameters The loss function value under represents the agent strategy entropy, represents the transpose of the parameters of the implicit reward function, represents the characteristic function, which is used to map the state s to the feature space, , Respectively in the strategy and expert strategies Expectations of the following characteristics.

4. The method according to claim 1, characterized in that: The intelligent agent driving strategy is used for the intelligent agent's assisted driving decision, so that the intelligent agent can make driving decisions similar to those of a human driver according to the learned implicit reward function when facing a new driving scenario, including: The vehicle status information and environmental perception information are obtained through vehicle-mounted sensors. The status information includes vehicle speed, acceleration and steering angle. The environmental perception information includes relative position and speed with respect to surrounding vehicles, relative position and angle with respect to road boundaries, and traffic sign and marking information. Inputting the state information and the environment perception information into the learned intelligent agent driving strategy model to generate a set of candidate driving decision behaviors, wherein the candidate driving decision behaviors include acceleration, deceleration and steering; Using the learned implicit reward function, calculate the instantaneous reward value and the expected long-term cumulative reward value of each candidate driving decision behavior in the current state. The instantaneous reward value measures the rationality and safety of the candidate driving decision behavior at the current moment, and the expected long-term cumulative reward value measures the impact of the candidate driving decision behavior on vehicle driving in the future. When calculating the expected value of the long-term cumulative reward, the Monte Carlo tree search algorithm is used to estimate the expected return of each decision node by performing multiple random simulations on the decision tree; based on the instant reward value, the candidate driving decision behavior with the highest expected value of the overall reward is selected as the optimal driving decision behavior; The selected optimal driving decision behavior is converted into specific vehicle control instructions, including throttle, brake and steering, and real-time control of the vehicle is achieved through on-board actuators.

5. The method according to claim 1, characterized in that Starting from the root node, the child nodes corresponding to the root node are recursively selected by the upper confidence bound algorithm, including: ； in, represents the upper confidence interval value of node j, represents the average reward estimate of node j, C represents a constant of the degree of exploration, called the exploration coefficient, N represents the total number of times the parent node of node j is visited, and M represents the number of nodes. Indicates the number of times node j is visited.

6. A driving training teaching system based on imitation learning, used to implement the method according to any one of claims 1 to 5, characterized in that: include: A first unit is used to obtain driving behavior demonstration data of multiple human drivers in multiple driving scenarios, and extract driving behavior features from the driving behavior demonstration data; Calculating a first characteristic expectation of an actual driving behavior trajectory of a human driver based on the driving behavior demonstration data; The second unit is used to optimize the driving strategy of the intelligent agent using a reinforcement learning algorithm according to the parameters of the initialized implicit reward function, obtain a simulated driving behavior trajectory of the intelligent agent by executing the driving strategy in a simulated environment, and calculate a second feature expectation of the simulated driving behavior trajectory of the intelligent agent based on the simulated driving behavior trajectory; minimize the difference between the first feature expectation and the second feature expectation and alternately optimize and update the parameters of the implicit reward function in combination with a generative adversarial network, and repeatedly execute the process of updating the parameters of the implicit reward function until the parameters of the implicit reward function converge or reach a preset number of iterations, thereby obtaining the driving strategy of the intelligent agent when the iteration is terminated; The third unit is used to apply the intelligent agent driving strategy to the assisted driving decision of the intelligent agent, so that the intelligent agent can make driving decisions similar to those of human drivers according to the learned implicit reward function when facing a new driving scenario.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images