Multi-level human intelligence enhanced automatic driving vehicle decision control method and system

Abstract

The present application relates to a kind of multi-level human intelligence enhanced automatic driving vehicle decision control method and system, wherein the method comprises: collecting human driver driving data, reward function network and pre-training optimal strategy model are obtained by using inverse reinforcement learning algorithm training;DDPG control model is initialized, and agent uses the state information of current time of the automatic driving car collected to carry out autonomous exploration, and according to state information, decision instruction and reward, training data set is constructed to train the model;If human driver control instruction is received, it is used as decision instruction, otherwise, the control instruction obtained by DDPG control model is used as decision instruction;Based on decision instruction, using MPC controller carries out the lateral and longitudinal planning tracking of automatic driving car, obtains the vehicle control amount of next time, controls vehicle.Compared with prior art, the present application combines the robustness and adaptability advantage of human driving, effectively improves driving safety and comfort.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving behavior decision-making and control, and in particular to a multi-level human intelligence-enhanced autonomous vehicle decision-making and control method and system. Background Technology

[0002] In recent years, with the continuous development and progress of artificial intelligence technology, autonomous vehicle technology has also undergone rapid upgrades and iterations, gradually evolving from rule-based and optimization-based algorithms to imitation-based and self-evolving algorithms. Reinforcement learning, as a typical representative, aims to solve sequential decision-making problems. By constructing a reasonable reward function, it enables autonomous vehicles to autonomously search for feasible control measures and optimize decision-making strategies in their interaction with the driving environment, and is widely used in end-to-end autonomous driving strategies. However, reinforcement learning methods still face problems such as slow learning speed, limited learning capacity, and low reliability in dynamic and real-world traffic environments, and differ significantly from human control behavior. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-level human intelligence-enhanced decision-making and control method and system for autonomous vehicles. It introduces human experience into the training process of autonomous vehicles to overcome the shortcomings of reinforcement learning training, and combines interactive reinforcement learning methods to achieve full-process, multi-level human intelligence-enhanced decision-making and control of autonomous vehicles.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] One aspect of the present invention provides a multi-level human intelligence-enhanced decision-making and control method for autonomous vehicles, the method comprising the following steps:

[0006] S1. Collect driving data from human drivers to form a human demonstration trajectory dataset, and use inverse reinforcement learning algorithm to train a reward function network and a pre-trained optimal policy model.

[0007] S2. The reward function network and pre-trained optimal policy model trained based on the inverse reinforcement learning algorithm are used to initialize the experience replay pool and network parameters of the DDPG control model. The agent uses the collected current state information of the autonomous vehicle to explore autonomously, and updates the experience replay pool according to the current state information, decision instruction information, reward information and the state information of the next moment to build a training dataset and train the DDPG control model.

[0008] S3. Determine whether a human driver's control command has been received. If a human driver's control command has been received, use the human driver's control command as a decision command and use it in the experience replay pool of the training dataset in step S2. Otherwise, use the control command obtained by the DDPG control model based on the current state information as a decision command and use it in the experience replay pool of the training dataset in step S2.

[0009] S4. Based on the decision instructions obtained in step S3, the MPC controller is used to perform lateral and longitudinal planning and tracking of the autonomous vehicle to obtain the vehicle control quantity at the next moment and control the vehicle.

[0010] In step S1, collecting human driver driving data to form a human demonstration trajectory dataset specifically involves: using an onboard platform or sensors to collect autonomous vehicle state perception information and human driver decision-making action information at each moment to form a human demonstration trajectory dataset.

[0011] The human driver's decision-making information includes the acceleration and steering wheel angle information at the next moment.

[0012] Step S2 specifically involves:

[0013] Initialize the experience replay pool, with a data capacity of N;

[0014] Initialize the network parameters in the DDPG control model, including the parameters φ of the value network Q and the target value network. parameters The parameters θ of the policy network π and the target policy network parameters

[0015] The system acquires state perception information of the autonomous vehicle within a preset time interval, control command information obtained by the agent's autonomous exploration, reward information obtained using a reward function network, and state information at the next moment, forming a data pair of state-action-reward value (s). t ,a t ,r t ,s y+1 ), and store them in the experience replay pool. In the process of autonomous exploration by the intelligent agent, if it obtains the control instructions of the human driver, it replaces the control instructions obtained by the current autonomous exploration of the intelligent agent and stores the data in the experience replay pool.

[0016] We sample from an experience replay pool containing data from autonomous exploration and human guidance, construct a training dataset, and update the parameters φ of the value network Q and the parameters θ of the policy network π.

[0017] In this technical solution, online human intelligent decision-making is introduced into the training of the DDPG control model to update the experience replay pool, which serves as the training dataset, and is used to guide the model training update in the next round.

[0018] Step S4 specifically involves:

[0019] Using decision commands as input to the MPC controller, the planning task of the MPC controller is divided into longitudinal and lateral tasks for motion planning. At the same time, safety and comfort constraints are introduced during the planning process to obtain the expected acceleration and steering wheel angle of the autonomous vehicle at the next moment. The PID controller is then used for acceleration tracking to achieve vehicle control.

[0020] Another aspect of the present invention provides a multi-level human intelligence-enhanced autonomous vehicle decision control system, comprising:

[0021] The pre-training module is used to collect driving data from human drivers to form a human demonstration trajectory dataset, and to train the reward function network and the pre-trained optimal policy model using an inverse reinforcement learning algorithm.

[0022] The DDPG control model training module is used to initialize the DDPG control model experience replay pool and network parameters based on the reward function network and pre-trained optimal policy model trained by the inverse reinforcement learning algorithm. The agent uses the collected current state information of the autonomous vehicle to explore autonomously, and updates the experience replay pool according to the current state information, decision instruction information, reward information and the state information of the next moment to build a training dataset and train the DDPG control model.

[0023] The decision instruction determination module is used to perform the following steps: determine whether a human driver's control instruction has been received; if a human driver's control instruction has been received, then the human driver's control instruction is used as the decision instruction and used in the experience replay pool of the training dataset in the DDPG control model training module; otherwise, the control instruction obtained by the DDPG control model based on the input current state information is used as the decision instruction and used in the experience replay pool of the training dataset in the DDPG control model training module.

[0024] The control module is used to determine the decision instructions obtained by the decision instruction determination module, use the MPC controller to perform lateral and longitudinal planning and tracking of the autonomous vehicle, obtain the vehicle control quantity at the next moment, and control the vehicle.

[0025] In the pre-training module, the collection of human driver driving data to form a human demonstration trajectory dataset specifically involves: using an onboard platform or sensors to collect autonomous vehicle state perception information and human driver decision-making action information at each moment to form a human demonstration trajectory dataset.

[0026] The human driver's decision-making information includes the acceleration and steering wheel angle information at the next moment.

[0027] The DDPG control model training module specifically performs the following steps:

[0028] Initialize the experience replay pool, with a data capacity of N;

[0029] Initialize the network parameters in the DDPG control model, including the parameters φ of the value network Q and the target value network. parameters The parameters θ of the policy network π and the target policy network parameters

[0030] The system acquires state perception information of the autonomous vehicle within a preset time interval, control command information obtained by the agent's autonomous exploration, reward information obtained using a reward function network, and state information at the next moment, forming a data pair of state-action-reward value (s). t ,a t ,r t ,s y+1 ), and store them in the experience replay pool. In the process of autonomous exploration by the intelligent agent, if it obtains the control instructions of the human driver, it replaces the control instructions obtained by the current autonomous exploration of the intelligent agent and stores the data in the experience replay pool.

[0031] We sample from an experience replay pool containing data from autonomous exploration and human guidance, construct a training dataset, and update the parameters φ of the value network Q and the parameters θ of the policy network π.

[0032] The control module specifically performs the following steps:

[0033] Using decision commands as input to the MPC controller, the planning task of the MPC controller is divided into longitudinal and lateral tasks for motion planning. At the same time, safety and comfort constraints are introduced during the planning process to obtain the expected acceleration and steering wheel angle of the autonomous vehicle at the next moment. The PID controller is then used for acceleration tracking to achieve vehicle control.

[0034] This invention introduces the influence of human drivers into the closed-loop control of autonomous vehicles throughout the entire process. By leveraging the robustness and adaptability of humans in dealing with complex and uncertain problems, autonomous vehicles can make appropriate decisions based on different road environments and scenario information, and gradually approach the control level of human drivers. At the same time, this invention also deploys an MPC controller in the lower-level control to avoid unnecessary decision-making errors, thereby ensuring the safety of the vehicle during operation.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention realizes fully automated driving control with enhanced human intelligence at multiple levels, divided into offline and online phases. In the offline phase, decision-making information of human drivers during driving is collected to form demonstration data. The reward function network is obtained using the inverse reinforcement learning (IRL) algorithm. Compared with directly obtaining the policy network, it has a certain degree of interpretability for human driving behavior and is subsequently used for training the reinforcement learning network.

[0037] 2. In the online phase, this invention incorporates the driver's necessary decision-making into the value network training process of the Deep Deterministic Policy Gradient (DDPG) algorithm. By leveraging the robustness and adaptability of humans in dealing with complex and uncertain problems, it improves the learning and generalization abilities of the autonomous vehicle's intelligent agent, promoting the rationality and human-likeness of autonomous vehicle decision-making. At the same time, the introduction of DDPG enables the model to handle continuous action problems, which is more in line with the operating logic of autonomous vehicles in the real world.

[0038] 3. In this invention, the model predictive control (MPC) method is used to implement lower-level control on the lower-level controller, and constraints are used to ensure safety and comfort during driving. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0041] Example 1

[0042] This embodiment provides a multi-level human intelligence-enhanced decision-making and control method for autonomous vehicles, such as... Figure 1 As shown, it includes the following steps:

[0043] S1. Collect state perception information s of the autonomous vehicle and decision-making information a of the human driver at each moment using onboard platforms or sensors. H A dataset of human demonstration trajectories (including acceleration and steering wheel angle information at the next moment) was constructed, and a reward function network R was trained using the inverse reinforcement learning (IRL) algorithm. net and pre-trained optimal policy model

[0044] In this embodiment, the inverse reinforcement learning algorithm is used to reversely calculate the reward function (or cost function) R. net The optimal policy model process, and the method used to train the value network Q and policy network π in the reinforcement learning DDPG, demonstrate offline human intelligence enhancement.

[0045] Demonstration trajectory data is constructed by collecting driving data from human drivers using onboard platforms or sensors. Data collection should maintain consistency as much as possible, utilizing the same driver or drivers with similar driving styles. Specific trajectory data information includes a set τ = {s1, a1, ..., s...}, consisting of state perception information and driving decision-making action information over a time series. t ,a t ,…a T The trajectory information is stored in the trajectory dataset D.

[0046] The reward function (cost function) and network parameters are denoted as ψ. The goal of network training is to continuously update the network parameters so that the trajectory of a human expert is within the optimal parameters ψ. * The network that maximizes the log-likelihood value outputs the optimal parameterized reward function.

[0047] In one preferred embodiment, in solving the reward function, key features can be manually selected, and a neural network can be used to find the mapping relationship between the reward function and the features. Assume f(τ) i This indicates feature extraction for a given trajectory. Let the weight vector in the mapping relationship be represented as:

[0048]

[0049] The network then outputs the optimal weight vector.

[0050] Based on the optimal reward function, IRL can obtain an optimal policy network that mimics human experts. In short, starting from an initial random policy, reward function updates and policy model updates alternate, always assuming the expert policy is optimal, ultimately training a reward function network. and the optimal policy model network

[0051] Since skilled human drivers typically follow certain patterns in their driving optimization goals and methods, with little variation, the reward function obtained using IRL has generalization and transfer characteristics. This allows it to be used in the construction of subsequent reinforcement learning reward function networks. Meanwhile, the policy model network trained during the process can also serve as a pre-training model for subsequent policy networks, accelerating model convergence.

[0052] S2. Initialize the DDPG control model's experience replay pool and network parameters based on the reward function network trained by the inverse reinforcement learning algorithm and the pre-trained optimal policy model. The agent uses the collected current state information of the autonomous vehicle to explore autonomously, and updates the experience replay pool according to the current state information, decision instruction information (control instruction information for autonomous network exploration / human driver operation instruction information (if any)), reward information and the state information of the next moment, constructs a training dataset, and trains the DDPG control model.

[0053] In this embodiment, the reinforcement learning layer uses the DDPG algorithm, a typical "actor-critic" type algorithm, which includes a Q-value network and a π-policy network. Its training process includes:

[0054] 1) Import the reward function network R obtained from the inverse reinforcement learning layer. net And the optimal policy model under this reward function. Initialize the experience playback pool with a data capacity of N; initialize the network parameters in the DDPG control model, including the parameters φ of the value network Q and the target value network. parameters The parameters θ of the policy network π and the target policy network parameters The optimal policy model obtained by the policy network using the IRL is:

[0055]

[0056] 2) Utilize onboard platforms or sensors to acquire state perception information of the autonomous vehicle within a preset time interval, control command information obtained through autonomous exploration by the intelligent agent, reward information obtained using a reward function network, and state information at the next moment, forming a state-action-reward data pair (s). t ,a t ,r t ,s t+1 The data is stored in the experience replay pool. During the agent's autonomous exploration, if a human driver's control command is obtained, it replaces the control command obtained during the agent's autonomous exploration, and the data is stored in the experience replay pool. The agent's actions are obtained through a policy network, with appropriate exploration data added.

[0057]

[0058] in This represents a stochastic process, used to encourage appropriate exploration by intelligent agents and prevent the driving performance of autonomous vehicles from being limited by human drivers.

[0059] 3) Sampling from an experience replay pool containing data from autonomous exploration and human guidance (s i ,a i ,r i ,s i++ ), construct a training dataset based on the previously updated target action value The network estimates the target value y, calculates the loss function, performs gradient descent, and updates the parameters θ. The autonomous vehicle updates the Q-network model in the DDPG every M steps, sampling data from the previous M steps for training each time. Every C steps, the Q-network parameters are assigned to the target value network. The estimated target value y is expressed as:

[0060]

[0061] After updating the Q-network, stochastic gradient descent is used to update the policy network parameters, thereby optimizing the policy.

[0062] In this embodiment, online human intelligent decision-making is introduced into the training of the DDPG control model to update the experience replay pool, which serves as the training dataset, and is used to guide the model training update in the next round.

[0063] Introducing online human intelligent decision-making methods brings human influence into the traditional field of artificial intelligence, endowing AI with human-like abilities to handle dynamic and uncertain tasks in the real world. Interactive reinforcement learning, as a typical example of artificial intelligence, uses mechanisms such as reward shaping and exploration guidance to enable autonomous vehicles to learn how to correctly perform autonomous driving tasks from human evaluation and feedback. This can accelerate the learning speed, enhance the learning credibility, and better realize the vision of human-like driving.

[0064] S3. Determine whether a human driver's control command has been received. If a human driver's control command has been received, use the human driver's control command as a decision command and use it in step S2 to update the experience replay pool of the training dataset. Otherwise, use the control command obtained by the policy model of the DDPG control model based on the input current state information as a decision command and use it in step S2 to update the experience replay pool of the training dataset.

[0065] When training reinforcement learning models on autonomous vehicle systems, problems such as slow learning speed, random exploration direction, and difficulty in ensuring safety exist, limiting the speed of training. Furthermore, the presence of unexplored environments in the demonstration dataset leads to certain deficiencies in the reward function network. Therefore, the robustness and adaptability advantages of human drivers in handling complex problems are introduced. In this embodiment, the driver needs to observe the road environment, the driving status of surrounding vehicles, and the vehicle's behavioral decisions in real time during autonomous vehicle operation. When the learning speed is low or the learning state is unreasonable or even erroneous, manual intervention is performed to guide the agent's learning. Specifically, if the vehicle makes an unreasonable or even erroneous acceleration decision at time t, the model will receive reasonable instructions from the driver. The original command is overridden to control the vehicle (i.e., the choice of decision command is different), and the corresponding state-action-reward value information (s) is also overridden. t ,a t ,r t ,s t+1 This information will be stored in the experience replay pool for subsequent network updates. The aforementioned control command switching can employ an agent-based exploratory action. With driver instructions The flexible online switching mechanism is as follows:

[0066]

[0067] When there is human guidance, O(s) t If ) = 1, then during the training process of the intelligent agent, when the human driver deems it necessary to intervene in driving, the driving control will be completely transferred to the driver, and unreasonable or even wrong decisions during the training process will be corrected in a timely manner, so that the strategy can be iterated quickly and efficiently, thereby improving the reliability and safety of the entire driving system.

[0068] In step S3, the human driver's control commands are generated through the interactive device. The driver needs to observe the autonomous vehicle's operation and training in a timely manner and use the interactive device for action guidance.

[0069] Through steps S2 and S3, the two-layer control model combining DDPG and MPC can be updated based on online human intelligence. The human driver monitors the vehicle status in real time and takes over. When receiving instructions from the human driver, the instructions obtained by the current network model are replaced with the human driver's control instructions, which are used as the final decision instructions for vehicle control. The experience replay pool, which serves as the training dataset, is updated for the next round of model training and updates. When training and updating network parameters, the network model is trained using a mixture of autonomous exploration and human guidance data. In the two-layer control model combining DDPG and MPC, the value network model is updated every M steps by the autonomous vehicle. By utilizing human guidance actions, the strong robustness and high adaptability of humans in complex driving scenarios are fully utilized. At the same time, it does not completely detach from autonomous exploration data, ensuring that the model's performance is not hindered from further evolution due to driver capability limitations.

[0070] S4. Based on the decision instructions obtained in step S3, the MPC controller is used to perform lateral and longitudinal planning and tracking of the autonomous vehicle to obtain the vehicle control quantity at the next moment and control the vehicle.

[0071] Specifically, the control commands obtained from the upper-level DDPG model, or the manipulation commands obtained under human guidance, are used as decision commands. These decision commands are then used as inputs to the MPC controller. The planning task of the MPC controller is divided into longitudinal and lateral tasks for motion planning. At the same time, in order to ensure the safety and comfort of vehicle control, safety and comfort constraints are introduced during the planning process to obtain the expected acceleration and steering wheel angle of the autonomous vehicle at the next moment. The PID controller is then used for acceleration tracking to achieve vehicle control.

[0072] In a preferred embodiment, the MPC controller is mainly divided into vertical and horizontal tasks, and the specific structural design is as follows:

[0073] The longitudinal kinematic model is as follows:

[0074]

[0075]

[0076] Where X, v X a X Representing longitudinal displacement, longitudinal velocity, and longitudinal acceleration respectively; state variable x lon Output variable y lon and control variable u lon as follows:

[0077] x lon =[Xv X ] T ,y lon=X,u lon =a X

[0078] The MPC longitudinal motion planning problem can be defined as follows, with the objective function being:

[0079]

[0080] The constraints are set as follows:

[0081]

[0082] The lateral dynamics model is as follows:

[0083]

[0084]

[0085]

[0086]

[0087] Where r represents the yaw rate of the vehicle at its center of gravity, and l f and l r F represents the distance from the center of gravity to the front and rear axles. yf and F yr The expression is as follows:

[0088]

[0089]

[0090] State variable x lat Output variable y lat and control variable u lat as follows:

[0091]

[0092] The objective function of the MPC problem is (introducing a relaxation factor to minimize the possibility that the controller cannot find the optimal solution within a specified time, thus preventing vehicle loss of control):

[0093]

[0094] The constraints are set as follows:

[0095]

[0096] Example 2

[0097] This embodiment provides a multi-level human intelligence-enhanced autonomous vehicle decision-making control system, including:

[0098] The pre-training module is used to collect driving data from human drivers to form a human demonstration trajectory dataset, and to train the reward function network and the pre-trained optimal policy model using an inverse reinforcement learning algorithm.

[0099] The DDPG control model training module is used to initialize the DDPG control model experience replay pool and network parameters based on the reward function network and pre-trained optimal policy model trained by the inverse reinforcement learning algorithm. The agent uses the collected current state information of the autonomous vehicle to explore autonomously, and updates the experience replay pool according to the current state information, decision instruction information, reward information and the state information of the next moment to build a training dataset and train the DDPG control model.

[0100] The decision instruction determination module is used to perform the following steps: determine whether a human driver's control instruction has been received; if a human driver's control instruction has been received, then the human driver's control instruction is used as the decision instruction and used in the experience replay pool of the training dataset in the DDPG control model training module; otherwise, the control instruction obtained by the DDPG control model based on the input current state information is used as the decision instruction and used in the experience replay pool of the training dataset in the DDPG control model training module.

[0101] The control module is used to determine the decision instructions obtained by the decision instruction determination module, use the MPC controller to perform lateral and longitudinal planning and tracking of the autonomous vehicle, obtain the vehicle control quantity at the next moment, and control the vehicle.

[0102] The specific operation mode and execution steps of each module in this embodiment can be found in Embodiment 1, and will not be repeated here.

[0103] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multi-level human intelligence-enhanced decision-making and control method for autonomous vehicles, characterized in that, The method includes the following steps: S1. Collect driving data from human drivers to form a human demonstration trajectory dataset, and use inverse reinforcement learning algorithm to train a reward function network and a pre-trained optimal policy model. S2. The reward function network and pre-trained optimal policy model trained based on the inverse reinforcement learning algorithm are used to initialize the experience replay pool and network parameters of the DDPG control model. The agent uses the collected current state information of the autonomous vehicle to explore autonomously, and updates the experience replay pool according to the current state information, decision instruction information, reward information and the state information of the next moment to build a training dataset and train the DDPG control model. S3. Determine whether a human driver's control command has been received. If a human driver's control command has been received, use the human driver's control command as a decision command and use it in the experience replay pool of the training dataset in step S2. Otherwise, use the control command obtained by the DDPG control model based on the current state information as a decision command and use it in the experience replay pool of the training dataset in step S2. S4. Based on the decision instructions obtained in step S3, the MPC controller is used to perform lateral and longitudinal planning and tracking of the autonomous vehicle to obtain the vehicle control quantity at the next moment and control the vehicle.

2. The multi-level human intelligence-enhanced autonomous vehicle decision-making and control method according to claim 1, characterized in that, In step S1, collecting human driver driving data to form a human demonstration trajectory dataset specifically involves: using an onboard platform or sensors to collect autonomous vehicle state perception information and human driver decision-making action information at each moment to form a human demonstration trajectory dataset.

3. The multi-level human intelligence-enhanced autonomous vehicle decision-making and control method according to claim 2, characterized in that, The human driver's decision-making information includes the acceleration and steering wheel angle information at the next moment.

4. The multi-level human intelligence-enhanced autonomous vehicle decision-making and control method according to claim 1, characterized in that, Step S2 specifically involves: Initialize the experience replay pool, with a data capacity of N; Initialize the network parameters in the DDPG control model, including the parameters φ of the value network Q and the target value network. parameters The parameters θ of the policy network π and the target policy network parameters The system acquires state perception information of the autonomous vehicle within a preset time interval, control command information obtained by the agent's autonomous exploration, reward information obtained using a reward function network, and state information at the next moment, forming a data pair of state-action-reward value (s). t ,a t ,r t ,s t+1 ), and store them in the experience replay pool. In the process of autonomous exploration by the intelligent agent, if it obtains the control instructions of the human driver, it replaces the control instructions obtained by the current autonomous exploration of the intelligent agent and stores the data in the experience replay pool. We sample from an experience replay pool containing data from autonomous exploration and human guidance, construct a training dataset, and update the parameters φ of the value network Q and the parameters θ of the policy network π.

5. The multi-level human intelligence-enhanced autonomous vehicle decision-making and control method according to claim 1, characterized in that, Step S4 specifically involves: Using decision commands as input to the MPC controller, the planning task of the MPC controller is divided into longitudinal and lateral tasks for motion planning. At the same time, safety and comfort constraints are introduced during the planning process to obtain the expected acceleration and steering wheel angle of the autonomous vehicle at the next moment. The PID controller is then used for acceleration tracking to achieve vehicle control.

6. A multi-level human intelligence-enhanced decision-making control system for autonomous vehicles, characterized in that, include: The pre-training module is used to collect driving data from human drivers to form a human demonstration trajectory dataset, and to train the reward function network and the pre-trained optimal policy model using an inverse reinforcement learning algorithm. The DDPG control model training module is used to initialize the DDPG control model experience replay pool and network parameters based on the reward function network and pre-trained optimal policy model trained by the inverse reinforcement learning algorithm. The agent uses the collected current state information of the autonomous vehicle to explore autonomously, and updates the experience replay pool according to the current state information, decision instruction information, reward information and the state information of the next moment to build a training dataset and train the DDPG control model. The decision instruction determination module is used to perform the following steps: determine whether a human driver's control instruction has been received; if a human driver's control instruction has been received, then the human driver's control instruction is used as the decision instruction and used in the experience replay pool of the training dataset in the DDPG control model training module; otherwise, the control instruction obtained by the DDPG control model based on the input current state information is used as the decision instruction and used in the experience replay pool of the training dataset in the DDPG control model training module. The control module is used to determine the decision instructions obtained by the decision instruction determination module, use the MPC controller to perform lateral and longitudinal planning and tracking of the autonomous vehicle, obtain the vehicle control quantity at the next moment, and control the vehicle.

7. A multi-level human intelligence-enhanced autonomous vehicle decision control system according to claim 6, characterized in that, In the pre-training module, the collection of human driver driving data to form a human demonstration trajectory dataset specifically involves: using an onboard platform or sensors to collect autonomous vehicle state perception information and human driver decision-making action information at each moment to form a human demonstration trajectory dataset.

8. A multi-level human intelligence-enhanced autonomous vehicle decision control system according to claim 7, characterized in that, The human driver's decision-making information includes the acceleration and steering wheel angle information at the next moment.

9. A multi-level human intelligence-enhanced autonomous vehicle decision control system according to claim 6, characterized in that, The DDPG control model training module specifically performs the following steps: Initialize the experience replay pool, with a data capacity of N; Initialize the network parameters in the DDPG control model, including the parameters φ of the value network Q and the target value network. parameters The parameters θ of the policy network π and the target policy network parameters The system acquires state perception information of the autonomous vehicle within a preset time interval, control command information obtained by the agent's autonomous exploration, reward information obtained using a reward function network, and state information at the next moment, forming a data pair of state-action-reward value (s). t ,a t ,r t ,s y+1 ), and store them in the experience replay pool. In the process of autonomous exploration by the intelligent agent, if it obtains the control instructions of the human driver, it replaces the control instructions obtained by the current autonomous exploration of the intelligent agent and stores the data in the experience replay pool. We sample from an experience replay pool containing data from autonomous exploration and human guidance, construct a training dataset, and update the parameters φ of the value network Q and the parameters θ of the policy network π.

10. A multi-level human intelligence-enhanced autonomous vehicle decision control system according to claim 6, characterized in that, The control module specifically performs the following steps: Using decision commands as input to the MPC controller, the planning task of the MPC controller is divided into longitudinal and lateral tasks for motion planning. At the same time, safety and comfort constraints are introduced during the planning process to obtain the expected acceleration and steering wheel angle of the autonomous vehicle at the next moment. The PID controller is then used for acceleration tracking to achieve vehicle control.

Images