A human-in-the-loop autonomous driving vehicle interactive learning control method and device

Through the combination of DQN-MPC dual-layer control model and human-in-ring interactive information, the problem of insufficient decision-making strategy optimization capabilities in complex traffic scenarios is solved, the learning ability and generalization ability are improved, and the safety of autonomous driving is ensured. Sex and comfort.

CN115257809BActive Publication Date: 2024-11-26TONGJI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210900448.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-28
Publication Date
2024-11-26
Estimated Expiration
2042-07-28

AI Technical Summary

Technical Problem

Inadequate decision-making strategy optimization capabilities in complex traffic scenarios, resulting in limited awareness of the scene, reduced learning ability and generalization performance, and reinforcement learning algorithms may generate unsuitable decision-making instructions during training, affecting safety. Comfort.

Method used

Using DQN-MPC dual-layer control model, combining deep Q learning and model prediction control, introduce driver interaction information, generate decision instructions through Q value network and model prediction controller, and use punishment factors to correct parameters to ensure that autonomous vehicles are complex Drive safely in the environment.

Benefits of technology

It improves the learning ability and generalization ability of autonomous vehicles, ensures the safety and comfort of the driving process, reduces manual operations, and enhances the robustness and adaptability of decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115257809B_ABST
    Figure CN115257809B_ABST
Patent Text Reader

Abstract

The present invention relates to a human-in-the-loop interactive learning control method and device for an autonomous driving vehicle, the method comprising the following steps: collecting state perception information of the autonomous driving vehicle at the current moment, obtaining the expected acceleration and steering wheel angle of the next moment based on a trained DQN-MPC two-layer control model; the DQN-MPC two-layer control model is updated based on human-in-the-loop control, and when receiving a driver's instruction, the driver's instruction replaces the current decision instruction, updates the experience playback pool as a training data set, and generates a penalty factor at the same time, which is used to correct the update of the Q-value network parameters. In addition, the present invention has a human feedback prediction function, and when the credibility reaches the set standard, the model-in-the-loop control module will replace the long-term human-in-the-loop control to liberate the driver. Compared with the prior art, the present invention has the advantages of improving driving safety and comfort.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of behavioral decision-making and control of autonomous driving vehicles and the field of machine learning algorithms of artificial intelligence, and to a vehicle control method, and in particular to an interactive learning control method and device for an autonomous driving vehicle with a human in the loop. Background Art

[0002] In recent years, with the continuous development of artificial intelligence algorithms, the technology of autonomous vehicles has been rapidly improved. In the research of learning-based driving systems, reinforcement learning algorithms can enable autonomous vehicles to autonomously search for feasible control measures and optimize decision-making strategies in the driving environment, and have broad application prospects. However, although reinforcement learning algorithms have been effective in solving simple traffic scene cases, the actual types of vehicle driving roads are diverse, and the traffic flow is complex and changeable. It is very likely that the intelligent agent has limited scene cognition, and the learning ability and generalization performance are significantly reduced. In addition, during the exploration process of neural network training, the reinforcement learning algorithm will inevitably make decision instructions that are not suitable for the current driving state, and obtain unreasonable state-action pairs, thereby affecting the parameter update of the neural network. Therefore, ensuring the safety and comfort of autonomous vehicles during driving, improving the playback experience quality of the intelligent agent learning process and the generalization ability of multi-scenario transplantation are technical difficulties that technicians in this field need to solve urgently. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a human-in-the-loop autonomous driving vehicle interactive learning control method and equipment to improve driving safety and comfort.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A human-in-the-loop autonomous driving vehicle interactive learning control method, comprising the following steps:

[0006] Collecting the state perception information of the autonomous driving vehicle at the current moment, and obtaining the expected acceleration and steering wheel angle at the next moment based on the trained DQN-MPC two-layer control model, wherein the DQN-MPC two-layer control model includes a DQN layer and an MPC layer, the DQN layer obtains a decision instruction through a Q value network based on the state perception information, and the MPC layer performs longitudinal planning tracking and lateral planning tracking based on the decision instruction to obtain the expected acceleration and steering wheel angle;

[0007] The DQN-MPC two-layer control model is updated based on human-in-the-loop control. When receiving a driver's instruction, the driver's instruction replaces the current decision instruction, updates the experience replay pool as a training data set, and generates a penalty factor, which is used to correct the update of the Q-value network parameters.

[0008] Furthermore, the training process of the DQN-MPC two-layer control model includes:

[0009] Initialize the capacity of the experience replay pool to N, randomly initialize the weight value θ of the action value function Q network and the target action value function The weights of the network

[0010] Obtain state perception information of the autonomous driving vehicle over a period of time, form data pairs of state-action-reward value, and store them in the experience replay pool;

[0011] Sampling from the experience replay pool, constructing a training dataset, based on the target action value function updated last time The estimated target value of the network is then updated by performing gradient descent.

[0012] Furthermore, the autonomous driving vehicle updates the DQN-MPC two-layer control model once every M steps, and each update uniformly and randomly extracts M data pairs from the experience replay pool to construct a training data set.

[0013] Furthermore, the target action value function of the previous update The estimated target value of the network is specifically:

[0014] Determine whether the next step of the autonomous driving car has reached the destination. If so, calculate the target value Y = r i ,otherwise, Among them, γ is the reward discount coefficient, Refers to the maximum value selected according to the next state s′ Value, this The value corresponds to the action a′ at the next moment;

[0015] If the penalty factor exists, then r i Replace with r i e H , where e H is the penalty factor.

[0016] Furthermore, the driver command is obtained through an input device or generated by a model-in-the-loop control neural network.

[0017] Furthermore, when the model is training the in-loop control neural network, a training data set is constructed based on state perception information, decision instruction information, and driver instructions obtained through input devices during the automatic driving process.

[0018] Furthermore, when the credibility of the model-in-the-loop control neural network is greater than a set value, the driver command is generated by the model-in-the-loop control neural network.

[0019] Furthermore, the DQN layer selects an instruction with the largest Q value or a random instruction as the decision instruction according to the probability of the ε-greedy algorithm.

[0020] Furthermore, the MPC layer utilizes a PID controller to perform acceleration tracking.

[0021] The present invention also provides an electronic device comprising one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the human-in-the-loop autonomous driving vehicle interactive learning control method as described above.

[0022] The present invention targets complex and changeable road environments, introduces driver interaction information to train a safe and reliable value neural network, and enables the self-driving car to select appropriate decision instructions according to different vehicle spacing, relative speed, road curvature and other conditions, thereby controlling the vehicle to drive safely. Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention puts the driver's decision-making judgment when necessary into the value network training process of the Deep Q-Learning Net (DQN), and adopts the Model Predictive Control (MPC) method to realize the lower-level control, making full use of the robustness and adaptability of humans in the face of complex driving scenarios, improving the learning ability and generalization ability of the autonomous driving vehicle intelligent body, and the constraints of its MPC controller ensure safety and comfort during the autonomous driving process.

[0024] 2. The present invention is also provided with a model-in-the-loop control neural network for learning real-person driving instructions, which ensures reliability while reducing manual operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the principle of the present invention;

[0026] Figure 2 It is a control flow diagram of the MPC layer of the present invention;

[0027] Figure 3 It is a schematic diagram of the process of loop control by the inventor;

[0028] Figure 4 It is a schematic diagram of the flow chart of feedback prediction of the present invention. DETAILED DESCRIPTION

[0029] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0030] like Figure 1 As shown, this embodiment provides an interactive learning control method for an autonomous driving vehicle with a human in the loop, comprising the following steps: collecting state perception information of the autonomous driving vehicle at the current moment, and obtaining the expected acceleration and steering wheel angle at the next moment based on a trained DQN-MPC two-layer control model, wherein the DQN-MPC two-layer control model includes a DQN layer and an MPC layer, the DQN layer obtains a decision instruction through a Q-value network based on the state perception information, and the MPC layer performs longitudinal planning tracking and lateral planning tracking based on the decision instruction to obtain the expected acceleration and steering wheel angle, wherein the DQN-MPC two-layer control model is updated based on human-in-the-loop control, and when a driver instruction is received, the driver instruction replaces the current decision instruction, updates the experience replay pool as a training data set, and generates a penalty factor at the same time, and the penalty factor is used to correct the update of the Q-value network parameters.

[0031] Specifically, the above method includes:

[0032] 1) Generate decision instructions, obtain the expected acceleration and steering wheel angle based on the decision instructions, and realize autonomous driving vehicle control.

[0033] The self-driving car collects the current state set s during driving and transmits the information of the state set s to the convergent Q value network; the Q value network calculates the lane change and lane keeping instructions a according to the state s of the self-driving car and the possible lane change and lane keeping instructions a. Q The Q value is calculated, and the action instruction with the maximum Q value or the random instruction is selected as the decision instruction according to the probability of the ε-greedy algorithm; the decision instruction will be used as the input of the MPC layer to obtain the expected acceleration and steering wheel angle of the autonomous vehicle at the next moment, and the PID controller is used for acceleration tracking, such as Figure 2 As shown in the figure, the MPC controller is not only a lower-level controller that executes upper-level decisions, but also ensures the safety and comfort of the autonomous vehicle. Its specific structure is designed as follows:

[0034] The longitudinal kinematic model is:

[0035]

[0036]

[0037]

[0038] Where x is the longitudinal displacement of the vehicle, v x is the longitudinal velocity of the vehicle, Δx z is the relative velocity, select x, v x , Δx zis the state variable, a x is the input variable, y lon =Δx z is the output variable, and its cost function can be defined as:

[0039]

[0040] The constraints are set as:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] (i=1,2,…,N1; j=0,1,…,N1-1)

[0047] The lateral dynamics model is:

[0048]

[0049]

[0050]

[0051]

[0052] Where r represents the yaw rate of the vehicle at the center of gravity, l f and l r Indicates the distance from the center of gravity to the front and rear axles, F yf and F yr The expression is as follows:

[0053]

[0054]

[0055] Select v y , r, y are state variables, u lat =δ is the input variable, y lat =y is the output variable, and the cost function can be defined as:

[0056]

[0057] Constraints can be set as:

[0058]

[0059]

[0060]

[0061]

[0062] (i=1,2,…,N2; j=0,1,…,N2-1)

[0063] 2) Model update based on driver instructions.

[0064] The driver needs to observe the road environment, the driving status of surrounding vehicles and the behavioral decisions made by the self-driving car in real time during the driving process. If the self-driving car makes an unreasonable or unnecessary action decision at time t, the human-in-the-loop control module receives the reasonable lane change or lane keeping instructions provided by the driver through the embedded control screen in the car at time t+1. H Covering the automatic control decision instructions of self-driving carsa Q , to correct the car's behavior, and at the same time, the corresponding state-action-reward value information (s, a H ,r,s′) will be stored in the experience replay pool; when the autonomous vehicle runs to the Q network update step, the driver's command information (s,a H ,r,s′) will be used as candidate information to correct the update of the Q network. In addition, the human-in-the-loop control mechanism is also equivalent to a backward prediction penalty module. When the driver takes over the self-driving car, the reward value r in the target value Y estimation function i Multiply by the penalty factor e H , to further correct the update of Q network parameters, such as Figure 3 As shown in Figure 2, the human-in-the-loop mechanism not only improves the quality of experience in the experience replay pool, but also further improves the robustness and adaptability of the value Q neural network in the DQN algorithm. i is the reward value of the set obtained after sampling from the experience replay pool.

[0065] The above method introduces human-in-the-loop control information during model training, which can correct inaccurate and unreasonable decisions, thereby correcting the updating process of model parameters, improving model reliability, and thus improving the reliability and safety of autonomous driving vehicle control.

[0066] In this embodiment, in the DQN-MPC two-layer control model, the DQN layer includes a Q value network and an experience replay pool, and its training process includes:

[0067] 1) Initialize the capacity of the experience replay pool to N, randomly initialize the weight value θ of the action value function Q network and the target action value function The weights of the network

[0068] 2) Obtain the state perception information of the autonomous driving vehicle over a period of time to form a data pair of state-action-reward value, expressed as (s, a Q ,r,s′), s represents the state, a Q Represents the action, r represents the reward value, s′ is the state at the next moment, and the reward value r is calculated by the reward function. r includes speed limit reward, collision reward, line crossing reward, comfortable driving reward, stop reward and courtesy reward. The data pair is stored in the experience replay pool.

[0069] The vehicle's onboard sensors are used to observe the road environment and traffic flow at the current time t, and the vehicle's displacement x, y, and vehicle speed v are calculated. x 、v y , the distance between the vehicle and the center line d1, the displacement of the surrounding vehicle x z ,y z , the speed of the vehicle v xz 、v yz , and the relative position of the pericycle Δx z , Δy z and the relative velocity Δv xz , Δv yz Packaged as state s at time t; use Q neural network to calculate action value Q(s,a); select lane change and lane keeping instructions a according to ε-greedy algorithm Q , a Q It acts on the MPC layer to control the acceleration and steering wheel angle of the autonomous vehicle from time t to time t+1, and obtains the environmental observation state s′ and the corresponding reward function value r at the next moment.

[0070] 3) Sampling from the experience replay pool to construct a training data set based on the target action value function updated last time The estimated target value Y of the network is used to perform the gradient descent method to update the parameter θ. The self-driving car is set to update the DQN-MPC two-layer control model once every M steps. Each update uniformly randomly extracts M data pairs from the experience replay pool to construct a training data set. The mean square error is used to calculate the loss L = (YQ) 2 , and then perform the gradient descent method to update the parameters θ.

[0071] Among them, the target action value function of the previous update is The estimated target value of the network is specifically:

[0072] Determine whether the next step of the autonomous driving car has reached the destination. If so, calculate the target value Y = r i ,otherwise, Among them, γ is the reward discount coefficient, Refers to the maximum value selected according to the next state s′ Value, this The value corresponds to the action a′ at the next moment.

[0073] If the penalty factor exists, then r i Replace with r i e H , where e H is the penalty factor.

[0074] The above driver instructions can be obtained through input devices or generated by model-in-the-loop control neural networks. In another embodiment, a human feedback prediction mechanism is introduced, and the driver instructions obtained through the input device are learned by setting a human feedback prediction model credibility module and a model-in-the-loop supervisory control module to replace long-term human-in-the-loop control and free the driver. Figure 4 As shown, during training, the state s perceived by the autonomous vehicle and the decision behavior a Q and the driver's intervention decision a H It will be used as the input of the neural network and the LSTM neural network will be trained to simulate the driver's supervision behavior; when the human feedback prediction model credibility module determines that the credibility of the model-in-the-loop supervisory control module is greater than ξ, the model-in-the-loop supervisory control module will replace the human-in-the-loop control module and output appropriate decision-making instructions when necessary to predict human feedback and compensate for the poor decision-making behavior of the autonomous driving vehicle.

[0075] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0076] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A human-in-the-loop autonomous driving vehicle interactive learning control method, characterized in that: The following steps are involved: Collecting the state perception information of the autonomous driving vehicle at the current moment, and obtaining the expected acceleration and steering wheel angle at the next moment based on the trained DQN-MPC two-layer control model, wherein the DQN-MPC two-layer control model includes a DQN layer and an MPC layer, the DQN layer obtains a decision instruction through a Q value network based on the state perception information, and the MPC layer performs longitudinal planning tracking and lateral planning tracking based on the decision instruction to obtain the expected acceleration and steering wheel angle; The DQN-MPC two-layer control model is updated based on human-in-the-loop control. When receiving a driver's instruction, the driver's instruction replaces the current decision instruction, updates the experience playback pool as a training data set, and generates a penalty factor, which is used to correct the update of the Q value network parameters; The training process of the DQN-MPC two-layer control model includes: Initialize the capacity of the experience replay pool to N, and randomly initialize the weight value of the action value function Q network and the target action-value function The weights of the network ; Obtain state perception information of the autonomous driving vehicle over a period of time, form data pairs of state-action-reward value, and store them in the experience replay pool; Sampling from the experience replay pool, constructing a training dataset, based on the target action value function updated last time Estimated target value of the network, perform gradient descent, and update parameters ; The autonomous vehicle updates the DQN-MPC two-layer control model once every M steps. Each time the update is performed, M data pairs are uniformly and randomly extracted from the experience replay pool to construct a training data set. The target action-value function of the previous update The estimated target value of the network is specifically: Determine whether the next step of the autonomous driving car has reached the destination. If so, calculate the target value Y= , otherwise, Y= ,in, is the reward discount factor, According to the next state Select the maximum Value, this Value and next moment action correspond; If the penalty factor exists, then Replace with ,in, is the penalty factor.

2. The human-in-the-loop autonomous driving vehicle interactive learning control method according to claim 1, characterized in that: The driver command is obtained through an input device or generated by a model-in-the-loop control neural network.

3. The human-in-the-loop autonomous driving vehicle interactive learning control method according to claim 2, characterized in that: When the model-in-the-loop control neural network is trained, a training data set is constructed based on state perception information, decision instruction information, and driver instructions obtained through input devices during the automatic driving process.

4. The human-in-the-loop autonomous driving vehicle interactive learning control method according to claim 2, characterized in that: When the credibility of the model-in-the-loop control neural network is greater than a set value, the driver command is generated by the model-in-the-loop control neural network.

5. The human-in-the-loop autonomous driving vehicle interactive learning control method according to claim 1, characterized in that: The DQN layers are based on The algorithm selects the instruction with the largest Q value or a random instruction as the decision instruction according to probability.

6. The human-in-the-loop autonomous driving vehicle interactive learning control method according to claim 1, characterized in that: The MPC layer uses a PID controller for acceleration tracking.

7. An electronic device, characterized in that: It includes one or more processors, a memory and one or more programs stored in the memory, and the one or more programs include instructions for executing the human-in-the-loop autonomous driving vehicle interactive learning control method as described in any one of claims 1-6.