Parallel learning-based intelligent decision control method for autonomous driving and domain control system

Abstract

The application relates to an automatic driving intelligent decision control method and a domain control system based on parallel learning, which comprises the following steps: dividing an automatic driving decision control problem into sub-problems and determining expert models; based on a deep reinforcement learning algorithm, establishing a state and action space of automatic driving decision control, a decision control rule and a reward function; taking multiple expert models as a bottom layer, taking multiple worker models as a middle layer and taking one leader model as a top layer, establishing a parallel learning framework and a parallel simulation environment, and training the three models; testing the performance of the leader model in a dynamically changing driving environment; if the test requirements are met, observing the state by the leader model, combining the action output by the worker model to obtain current decision information, and executing the decision information to complete vehicle control. Compared with the prior art, the application can effectively solve the problems of low sample data sampling efficiency and poor generalization ability, and can adapt to a dynamic driving environment and changing ride demand.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving decision control technology, and in particular to an intelligent decision control method and domain control system for autonomous driving based on parallel learning. Background Technology

[0002] In recent years, with the rapid development of technologies such as autonomous driving, holographic perception, and intelligent decision-making, autonomous driving decision-making and control methods based on deep reinforcement learning have emerged. These methods utilize interaction data between the agent and the driving environment to train a decision-making and control model in a simulation environment. Then, in a real driving environment, based on the perceived road traffic conditions, the model outputs a decision-making and control scheme and completes autonomous vehicle control. In short, deep reinforcement learning-based autonomous driving decision-making and control methods fully leverage the advantages of neural networks. Compared to traditional rule-based and optimal control methods, these methods offer high efficiency, rapid adaptation, and strong generalization, making them applicable to various driving scenarios.

[0003] Traditional single-agent deep reinforcement learning is typically designed for a specific decision-making and control problem, with long training cycles and limited application scenarios. While neural networks possess some generalization ability, the collection of training samples struggles to cover all possible situations, making it difficult for trained models to achieve ideal driving performance in all driving scenarios. Furthermore, the environment is constantly changing during autonomous driving, and different passengers have varying demands for vehicle decision-making and control. Therefore, to improve the adaptability of trained models to dynamically changing traffic environments and passenger demands, it is necessary to rapidly adjust model parameters by incorporating interaction data between autonomous vehicles and the real environment. In this context, the collection of high-quality training samples and efficient model parameter updates are crucial, but this poses a significant challenge for traditional single-agent deep reinforcement learning. Traditional deep reinforcement learning can only aimlessly explore the action space or spend a long time waiting for an ideal model to be trained.

[0004] In existing research, besides training new deep reinforcement learning models, existing rule-based models, model predictive control, fully trained imitation learning models, and fully trained deep reinforcement learning models can all provide decision-making references for autonomous driving decision control. However, most existing models are designed for different decision control problems, and there is no guarantee that the strategy output by the existing model will be optimal under the current scenario and problem. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an intelligent decision-making and control method and domain control system for autonomous driving based on parallel learning, so as to realize the decision-making and control of autonomous vehicles in complex driving scenarios and solve the problems of low sample collection efficiency and limited generalization encountered by traditional single-agent deep reinforcement learning in solving complex decision-making and control problems.

[0006] The objective of this invention can be achieved through the following technical solution: an intelligent decision-making and control method for autonomous driving based on parallel learning, comprising the following steps:

[0007] S1. Based on the autonomous driving task, the autonomous driving decision control problem is divided into sub-problems, and the existing sub-problem-oriented decision control model is used as the autonomous driving expert model.

[0008] S2. Based on deep reinforcement learning algorithms, establish the state and action space for autonomous driving decision control, as well as safety-oriented decision control rules, and construct a reward function in combination with the diverse travel needs of passengers;

[0009] S3. A parallel learning framework and parallel simulation environment are established, with multiple expert models as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer.

[0010] S4. During the model training process, the worker model and the leader model combine their own experience and strategies, and take the decision control scheme output by the expert model as a reference to select the better decision to execute, or use the mean of the decision as the basis for action selection to formulate the final decision.

[0011] S5. Establish an evaluation index system for autonomous driving decision control. Test the performance of the trained leader model in a dynamically changing driving environment. If the leader model test meets the set test requirements, the leader model observes the state in the current actual environment, combines the current actions output by the worker model, outputs the current decision information, and executes the decision information to complete vehicle control.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] S11. Divide the decision control sub-problems

[0014] With safety, efficiency, and comfort as the decision-making and control objectives for autonomous driving, the weights of passengers’ different riding needs are obtained. Based on the autonomous driving task, the decision-making and control problem is divided into several simpler sub-problems, and the sub-problems contain solutions for different scenarios in the same driving task.

[0015] S12. Determine the expert model

[0016] For each sub-problem, models for different scenarios are selected from existing models for autonomous driving decision control as expert models. Existing models for autonomous driving decision control include rule-based models, model predictive control, trained imitation learning models, trained deep reinforcement learning models, and trained deep inverse reinforcement learning models.

[0017] Furthermore, step S2 specifically includes the following steps:

[0018] S21. Establish state and action space

[0019] Based on the expert model, and considering the current autonomous driving decision-making and control problem, a state space that can provide sufficient information for problem solving and an action space that meets the vehicle control requirements in the real environment are established. The action of autonomous driving decision-making and control is set as a, where the action of longitudinal decision-making and control is longitudinal acceleration, and the action of lateral decision-making and control is lateral acceleration.

[0020] The state space for autonomous driving decision control includes the acceleration of the previous time step and the velocity of the current time step, the speed difference between the vehicles in front and behind, the distance between the vehicles in front and behind, the road surface quality, and the motion information of surrounding vehicles.

[0021] S22. Establish action selection rules for driving safety.

[0022] The action selection rules for driving safety are as follows: if the distance between the autonomous vehicle and the vehicle in front is less than the longitudinal safety distance, the acceleration is selected as the maximum deceleration; otherwise, the longitudinal decision control action output by deep reinforcement learning is adopted. Based on the result of the autonomous vehicle lane change decision, the distance between the autonomous vehicle and the vehicles in front and behind in the target lane after the lane change is predicted. If the longitudinal safety distance requirement cannot be met, the autonomous vehicle continues to drive in the current lane; otherwise, the lateral decision control action output by deep reinforcement learning is adopted.

[0023] S23. Constructing the reward function

[0024] By combining the reward function settings of the expert model for solving the decision-making and control sub-problem of autonomous driving, a reward function that can meet the diverse needs of passengers is designed, including evaluation indicators related to safety, efficiency and comfort, including but not limited to collision time, safe distance, driving efficiency and ride comfort.

[0025] Furthermore, in step S23, the evaluation indicators of driving efficiency include the probability that the current headway is in the headway distribution obtained from the measured road traffic data, and the difference between the current vehicle speed and the maximum road speed.

[0026] The evaluation indicators for ride comfort include longitudinal acceleration rate of change, longitudinal acceleration, lateral acceleration rate of change, lateral acceleration, the difference between the vehicle's current speed and the maximum comfort speed for the current road segment, where the maximum comfort speed is obtained based on vehicle vibration simulation. An autonomous driving simulation environment is established, and road surface elevation and vehicle parameters are input into the simulation environment. By simulating vibrations at different speeds on various road segments, and combining the root mean square value of acceleration and the annoyance rate, the relationship between objective comfort evaluation indicators, passenger subjective feelings, and objective comfort evaluation indicators is established, thus determining the maximum speed that can meet passenger comfort requirements.

[0027] Furthermore, step S3 specifically includes the following steps:

[0028] S31. Establish a parallel learning framework

[0029] The system consists of multiple driving expert models for specific sub-problems as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer. The expert models and worker models interact with the parallel environment, while the leader model interacts with the real environment.

[0030] S32. Set the information transfer mode between parallel models.

[0031] The leader model observes the state in the real environment and passes it down to the worker model and the expert model. Then, the expert model extracts the state information based on its own state settings and passes the expert model's actions up to the worker model and the leader model. The worker model then combines the obtained state information and refers to the expert model's actions and passes the worker model's actions up to the leader model. Finally, the leader model makes the final decision based on the observed state and the worker model's actions.

[0032] S33, Learning Parallel Environment State Transition Functions

[0033] The state transition function of the parallel environment is obtained by training the leader model with the interaction data of the real environment. Then, the worker model and the leader model are trained in parallel by passing state and action information in the parallel environment.

[0034] Furthermore, the expert model training process in step S4 is as follows: based on the state settings of the selected expert model, extract the expert model state information from the state information transmitted by the leadership model, and solve the expert model's decision control scheme.

[0035] Furthermore, the training process of the worker model in step S4 is as follows: in the early stage of training, the decision control scheme of the expert model is executed; in the middle stage of training, the better decision control scheme output by the expert model and the worker model is selected and executed; and in the later stage of training, the decision control scheme is formulated according to its own strategy.

[0036] Furthermore, the training process of the leadership model in step S4 is as follows: in the early stage of training, a decision control scheme is formulated according to its own strategy; in the middle stage of training, the better decision control scheme between the worker model and the leadership model is selected for execution; and in the later stage of training, the average value of the decision control schemes output by the worker model and the leadership model is selected for execution with reference.

[0037] Furthermore, the specific process of testing the performance of the trained leadership model in step S5 includes:

[0038] S51. Establish a decision-making control evaluation index system

[0039] Based on the reward function design of the leadership model, collision time, speed, acceleration, rate of change of acceleration, and annoyance rate, which involve safety, efficiency, and comfort, are used as evaluation indicators of the decision control model to determine whether the autonomous driving performance meets the passenger's riding needs.

[0040] S52. Establish test scenarios for autonomous driving decision control.

[0041] The autonomous driving decision-making and control test scenario consists of road surfaces with different smoothness, traffic flow with different density, and road sections with different alignments, and is used to test the generalization ability of the trained model.

[0042] S53, Testing the Autonomous Driving Decision Control Model

[0043] For a given test scenario, relevant environmental data are randomly selected from the actual vehicle-road database. Several experiments are conducted on the test scenario in a simulation environment. If the decision control scheme output by the leader model can outperform the driving performance of the expert model in different driving scenarios, then the training of the autonomous driving decision control model based on parallel learning is complete.

[0044] An autonomous driving domain control system based on parallel learning includes an intelligent driving domain controller and a chassis domain controller. The intelligent driving domain controller includes a fusion perception module, a navigation and positioning module, an online training module, and a decision generation module. The autonomous vehicle acquires motion data and decision control models of other vehicles beyond line of sight, road surface elevation data, and safe driving rules for the current road via vehicle-to-infrastructure communication. The fusion perception module uses multi-source onboard sensors to perceive traffic conditions and road surface data within the sensor range of the surrounding sensors, matching them according to detection time and location to generate real-time fusion data. The online training module, based on an onboard deep reinforcement learning model, combines safe driving rules and decision control models of other vehicles in the current driving environment to rapidly improve the performance of the decision control model based on interaction data with the environment within a short time. The decision generation module outputs longitudinal, lateral, and vertical motion decisions for the autonomous vehicle based on the environmental perception data output by the fusion perception module and the parallel reinforcement learning model.

[0045] The chassis domain controller obtains the driving decision control scheme output by the intelligent driving domain controller through the gateway device, estimates the lateral, longitudinal, and vertical motion characteristic parameters of the vehicle using on-board sensor signals, predicts the motion characteristic parameters within a set time step in the future, and generates lateral, longitudinal, and vertical motion control signals based on the vehicle state estimation to complete the control of the vehicle trajectory, speed, and suspension.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] This invention first divides the autonomous driving decision-making and control problem into sub-problems based on the autonomous driving task, and uses existing sub-problem-oriented decision-making and control models as autonomous driving expert models. Then, based on deep reinforcement learning algorithms, it establishes the state and action space for autonomous driving decision-making and control, as well as safety-oriented decision-making and control rules, and constructs a reward function in conjunction with diverse passenger travel needs. Next, it establishes a parallel learning framework and parallel simulation environment, using multiple expert models as the bottom layer, multiple deep reinforcement learning-based worker models as the middle layer, and a deep reinforcement learning-based leader model as the top layer. During model training, the worker and leader models combine their own experience and strategies, and refer to the decision-making and control schemes output by the expert models, to select the better decision to execute, or use the average of the decisions as the basis for action selection to formulate the final decision, and execute the corresponding decision to complete vehicle control. Finally, it establishes an autonomous driving decision-making and control evaluation index system, and tests the performance of the trained leader model in a dynamically changing driving environment. This invention, on the one hand, explores existing decision-making and control models, effectively improving the exploration efficiency of agents in unknown action spaces; on the other hand, it employs parallel learning, introducing multiple parallel training environments and agents to the traditional single-agent training, allowing multiple agents to collaboratively explore and develop based on existing models. This enables the collection of a large amount of high-quality training data in a short time, contributing to improved model training efficiency and generalization ability. Thus, it effectively solves the problems of low sampling efficiency and poor generalization ability of learning-based complex autonomous driving decision-making and control models, enabling them to adapt well to dynamic driving environments and changing passenger demands, and achieve autonomous vehicle decision-making and control in complex driving scenarios. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0049] Figure 2 This is a schematic diagram of the domain control system structure of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] like Figure 1 As shown, an intelligent decision-making and control method for autonomous driving based on parallel learning includes the following steps:

[0053] S1. Based on the autonomous driving task, the autonomous driving decision control problem is divided into sub-problems, and the existing sub-problem-oriented decision control model is used as the autonomous driving expert model.

[0054] S2. Based on deep reinforcement learning algorithms, establish the state and action space of autonomous driving decision control and safety-oriented decision control rules, and construct a reward function in combination with the diverse riding needs of passengers;

[0055] S3. A parallel learning framework and parallel simulation environment are established, consisting of multiple expert models as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer.

[0056] S4. During the model training process, the worker model and the leader model combine their own experience and strategies, and take the decision control scheme output by the expert model as a reference to select the better decision to execute, or use the mean of the decision as the basis for action selection to formulate the final decision.

[0057] S5. Establish an evaluation index system for autonomous driving decision control. Test the performance of the trained leader model in a dynamically changing driving environment. If the leader model test meets the set test requirements, the leader model observes the state in the current actual environment, combines the current actions output by the worker model, outputs the current decision information, and executes the decision information to complete vehicle control.

[0058] Step S1 specifically includes the following steps:

[0059] S11. Divide the decision control sub-problems

[0060] With safety, efficiency, and comfort as the decision-making and control objectives for autonomous driving, the weights of passengers’ different riding needs are obtained. Based on the autonomous driving task, the decision-making and control problem is divided into several simpler sub-problems, and the sub-problems contain solutions for different scenarios in the same driving task.

[0061] S12. Determine the expert model

[0062] For each sub-problem, we select expert models from existing models for autonomous driving decision control, such as rule-based models, model predictive control, trained imitation learning models, trained deep reinforcement learning models, and trained deep inverse reinforcement learning models, to suit different scenarios. Specific driving scenarios include different surrounding vehicle movements, road surface quality, road alignment, etc. Training completion means that the model has been trained to convergence and can meet the requirements of safety, efficiency, comfort, etc., and the model has been tested in a simulation or real driving environment.

[0063] Step S2 specifically includes the following steps:

[0064] S21. Establish state and action space

[0065] Based on the expert model in step S12, and according to the current autonomous driving decision-making and control problem, a comprehensive state space that can provide sufficient information for problem solving and an action space that meets the vehicle control requirements in the real environment are established. The action for autonomous driving decision-making and control is denoted as 'a', where the action for longitudinal decision-making control is longitudinal acceleration, and the action for lateral decision-making control is lateral acceleration. For longitudinal decision-making control, the state space includes, but is not limited to, the following information: the longitudinal acceleration of the autonomous vehicle in the previous time step and its velocity in the current time step, the speed difference between the vehicles in front and behind, the distance between the vehicles in front and behind, road surface quality, and the motion information of surrounding vehicles.

[0066] For a given lane, the state space for autonomous driving decision-making and control includes the longitudinal acceleration of the previous time step, the speed of the current time step, the speed difference between the vehicles in front and behind, the distance between the vehicles in front and behind, and the maximum comfortable speed information. The state space can be specifically represented as follows:

[0067] s t =[a t-1 V t ,ΔV t ,ΔS t V p,t ]

[0068] Where: s l V represents the state of lane l; t represents the current time step; t-1 represents the previous time step; V t The speed of the autonomous vehicle; ΔV t Let ΔV be the speed difference between the vehicle in front of the autonomous vehicle and the autonomous vehicle itself. t =V l.t -V t ;ΔS t This represents the distance difference between the vehicle in front of the autonomous vehicle and the autonomous vehicle itself.

[0069] S22. Establish action selection rules for driving safety.

[0070] The action selection rule for driving safety is as follows: if the distance between the autonomous vehicle and the vehicle in front is less than the safe distance, the acceleration is selected as the maximum deceleration (in this embodiment, the longitudinal acceleration range is [-3, 3] m / s²). 2 Otherwise, the action output by deep reinforcement learning is used:

[0071]

[0072]

[0073] Where: a p The action is the output of the parallel learning-based autonomous driving decision-making and control model; x is the position of the autonomous vehicle; xl Let l represent the position of the vehicle ahead of the autonomous vehicle, where l indicates the vehicle ahead of the autonomous vehicle currently making the decision-making and control, including both manually driven vehicles and autonomous vehicles, hereinafter referred to as the vehicle ahead. d is the average length of the vehicle. t For safe distance; t r For reaction time; a min This represents the maximum deceleration of autonomous driving, which is the minimum acceleration.

[0074] For example, based on the results of autonomous driving lane-changing decisions, the distance between the vehicle and the vehicles in front and behind in the target lane after the lane change is predicted. If the longitudinal safety distance requirement cannot be met, the autonomous vehicle continues to drive in the current lane; otherwise, the lateral decision control action output by deep reinforcement learning is adopted.

[0075] S23. Constructing the reward function

[0076] By combining the reward function settings of the expert model for solving the decision-making and control sub-problem of autonomous driving, a reward function that can meet the diverse needs of passengers is designed, including evaluation indicators related to safety, efficiency and comfort, mainly collision time, safe distance, driving efficiency and ride comfort. Among them, the evaluation indicators of driving efficiency include the probability of the current headway in the headway distribution obtained from the measured road traffic data, and the difference between the current vehicle speed and the maximum road speed.

[0077] The evaluation indicators for comfort include the rate of change of longitudinal acceleration, longitudinal acceleration, the difference between the current vehicle speed and the maximum comfort speed of the current road segment, and the maximum comfort speed of the road segment is obtained based on vehicle vibration simulation experiments. In this process, an autonomous driving simulation environment is established, and the road surface elevation and vehicle parameters are input into the autonomous driving simulation environment. By simulating vibration at different speeds on various road segments, and combining objective comfort evaluation indicators such as the root mean square value of acceleration and the annoyance rate, as well as the relationship between passenger subjective feelings and objective comfort evaluation indicators, the maximum speed that can meet the comfort requirements of passengers is determined.

[0078] Specifically, regarding safety, this embodiment uses Time-to-Collision (TTC) as the evaluation metric. If the speed of the autonomous vehicle is lower than the speed of the vehicle in front, a rear-end collision is almost impossible, and the TTC value is infinite. If the speed of the autonomous vehicle is:

[0079]

[0080] Based on the TTC metric, an appropriate TTC threshold is selected, and the agent is penalized when the TTC value falls below this threshold.

[0081]

[0082] Among them: TTC thr This is the TTC threshold.

[0083] Given that the TTC metric is calculated solely based on the relative speed and distance between the autonomous vehicle and the vehicle ahead, to ensure driving safety as much as possible, the agent is penalized when the distance is less than the safe distance, and otherwise, no intervention is provided.

[0084]

[0085] Regarding efficiency, if the current headway of the autonomous vehicle is similar to that maintained by a human driver, then the autonomous vehicle is considered to have learned a certain level of human driving style and can guarantee driving efficiency. Therefore, statistical analysis of actual road traffic data is performed, and the agent is rewarded based on the probability that a human driver maintains a corresponding headway.

[0086]

[0087] Where: h is the headway; μ eh and σ eh The parameters for fitting the headway using a logarithmic distribution are given.

[0088] Since headway only represents the relative relationship between the autonomous vehicle and the vehicle in front, a smaller headway indicates that the autonomous vehicle can closely follow the vehicle ahead. However, the speed of the autonomous vehicle also represents its driving efficiency. Therefore, comparing the speed of the autonomous vehicle with the maximum speed limit on the road, the closer the speed is to the maximum speed, the higher the efficiency. Exceeding the maximum speed limit indicates that the driving behavior is more dangerous, and the intelligent agent will be penalized. However, if the speed is below a certain limit, vehicles behind may easily reduce their speed, thus affecting the overall traffic efficiency. At the same time, excessively low speeds may cause abnormal vibrations in the vehicle body. Therefore, when the autonomous vehicle is driving at low speeds, a larger penalty is imposed on the intelligent agent to avoid negatively impacting traffic safety and comfort due to low speeds.

[0089]

[0090] Where: V max Where V is the maximum speed of the road, ρ is the penalty coefficient greater than 1, and V is the maximum speed of the road. min The minimum speed to avoid affecting road traffic efficiency and personal passenger comfort.

[0091] For ride comfort, real-time longitudinal acceleration change rate and longitudinal acceleration are mainly used for evaluation.

[0092]

[0093]

[0094] Where: j is the rate of change of longitudinal acceleration, j max The maximum rate of change of longitudinal acceleration is given by α, which is a parameter of longitudinal acceleration.

[0095] Considering that vehicle vibration varies with different road surfaces and speeds, the maximum comfort speed for each road segment is determined based on vehicle vibration simulation experiments. This maximum comfort speed is obtained through vehicle vibration simulation experiments. An autonomous driving simulation environment is established, and road surface elevation and vehicle parameters are input into the simulation. By simulating vibration at different speeds on various road segments, and combining objective comfort evaluation indicators such as root mean square acceleration and annoyance rate, as well as the relationship between passenger subjective feelings and objective comfort evaluation indicators, the maximum speed that meets passenger comfort requirements is determined.

[0096] If the speed of the autonomous vehicle exceeds the maximum comfort speed for the current road segment, the current acceleration selection will cause discomfort to the passengers, and the agent will be penalized by the difference between this speed and the maximum speed. Conversely, if the current acceleration selection is below the maximum comfort speed, it is considered that the current acceleration selection will not negatively affect subjective comfort, and no additional penalty or reward will be imposed on the agent.

[0097]

[0098] in: This is the maximum comfortable speed for the current road section.

[0099] The established reward function is a weighted sum of the above features:

[0100] r = w1R st +w2R sd +w3R eh +w4R es +w5R lj +w6R la +w7R v

[0101] Step S3 specifically includes the following steps:

[0102] S31. Establish a parallel learning framework

[0103] The system consists of multiple driving expert models for specific sub-problems as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer. The expert models and worker models interact with the parallel environment, while the leader model interacts with the real environment.

[0104] Define the states of the leader model, worker model, and expert model at time step t as follows: The actions are respectively The rewards are as follows Define the experience pools for the leadership and worker models as R. c and R w , where R w Based on specific worker models, they can be divided into... n represents the total number of worker models, and m represents one of them. To maximize the effect of policy integration, the worker model and the leader model are designed as identical agents, meaning that their state space, action space, and reward function are the same.

[0105] S32. Set the information transfer mode between parallel models.

[0106] The leader model observes the state in the real environment and passes it down to the worker and expert models. The expert model then extracts state information based on its own state settings and passes its actions up to the worker and leader models. The worker model then combines the obtained state information with reference to the expert model's actions and passes its actions up to the leader model. Finally, the leader model decides its final action based on the observed state and the worker model's actions. This information transmission pattern can be expressed as follows:

[0107]

[0108]

[0109] in: For the leadership model after execution The observed states, ← and → indicate that they are affected or determined.

[0110] S33, Learning Parallel Environment State Transition Functions

[0111] The state transition function of the parallel environment is obtained by training the leader model with the interaction data of the real environment. Then, the worker model and the leader model are trained in parallel by passing state and action information in the parallel environment.

[0112] Step S4 specifically includes the following steps:

[0113] S41, Expert Model Decision Control Scheme

[0114] Based on the state settings of the selected expert model, extract the expert model state information from the state information transmitted by the leadership model, and solve the expert model's decision control scheme.

[0115] Specifically, for autonomous driving decision-making and control, expert models include, but are not limited to, Intelligent Driver Model (IDM), Adaptive Cruise Control (ACC), Model Predictive Control (MPC), trained imitation learning models, trained deep reinforcement learning models, and trained deep inverse reinforcement learning models. Since each expert model corresponds to a specific longitudinal motion scenario for autonomous driving, the goal of autonomous driving decision-making and control is to select a suitable existing model as the expert model.

[0116] S42, Worker Model Decision Control Scheme

[0117] The corresponding worker model is set up based on the number of expert models.

[0118] In the initial training phase, to enable the worker model to acquire better decision-making capabilities, the longitudinal decision-making control scheme of the expert model will be implemented:

[0119]

[0120] in: For the decision control scheme of the m-th worker model, Let w be the decision control scheme for the m-th worker model, e be the expert model, and t be the number of workers. pi This indicates the strategy imitation phase in the early stages of training.

[0121] During the middle of training, the worker model selects the better decision control scheme output by both the expert model and the worker model for execution:

[0122]

[0123] in: Let N be the network of critics for the m-th worker model, and t be the noise level. pe This indicates the policy integration phase during the middle of training.

[0124] At this point, during training, the performer network not only needs to maximize the cumulative reward that the worker model can obtain, but also needs to ensure that the suggestions it provides can lead the leader model to choose a better decision, that is, to maximize the cumulative reward of the leader model. At the same time, it is also necessary to ensure that the worker model's strategies are different, so that the worker model can provide references for the leader model's training from different perspectives. Therefore, the training objective of the worker model performer network is designed as follows:

[0125]

[0126] Where λ1 and λ2 are weight parameters. and μ c The strategies are the worker model and the leader model, respectively. and The critic network parameters for the worker model and the leader model are Q, respectively. c Let I be the critic network of the leadership model, D be the function that evaluates the distance between two strategies, and I be the set of worker models other than worker model m, which can be represented as I = [1, m) ∪ (m, n).

[0127] In the later stages of training, the worker model formulates decision-making and control schemes according to its own strategies:

[0128]

[0129] Where: t ps This is the stage where the intelligent agent trains itself based on its own experience.

[0130] Meanwhile, the training objective of the worker model performer network is designed as follows:

[0131]

[0132] in: and ...

[0133] S43, Leadership Model Decision Control Scheme

[0134] In the early stages of training, the leadership model based on deep reinforcement learning formulates decision-making and control schemes according to its own strategies.

[0135] During the middle of training, the leader model selects the better decision-making control scheme from both the worker model and the leader model for implementation:

[0136]

[0137] Among them: Q c A network of critics for leadership models.

[0138] In the later stages of training, the leadership model is selected based on the average of the decision control schemes output by the worker model and the leadership model.

[0139]

[0140] The goal of updating the performer network parameters in the leadership model is to maximize the cumulative reward.

[0141]

[0142] When testing the performance of the leadership model in step S5, the specific steps include:

[0143] S51. Establish a longitudinal motion evaluation index system

[0144] Based on the reward function design of the leadership model, collision time, speed, acceleration, rate of change of acceleration, and annoyance rate, which involve safety, efficiency, and comfort, are used as evaluation indicators of the decision control model to determine whether the autonomous driving performance meets the passenger's riding needs.

[0145] S52. Establish test scenarios for longitudinal control of autonomous driving.

[0146] The autonomous driving decision-making and control test scenario consists of road surfaces with different smoothness, traffic flow with different density, and road sections with different alignments, and is used to test the generalization ability of the trained model.

[0147] S53, Testing the Autonomous Driving Decision Control Model

[0148] For a given test scenario, relevant environmental data are randomly selected from the actual vehicle-road database. Several experiments are conducted on the test scenario in a simulation environment. If the decision control scheme output by the leader model can outperform the driving performance of the expert model in different driving scenarios, then the training of the autonomous driving decision control model based on parallel learning is complete.

[0149] This embodiment applies the above technical solution, taking longitudinal control of autonomous driving as an example of decision control to illustrate the implementation of the present invention. The steps of longitudinal control include:

[0150] I. Divide the longitudinal motion control problem of autonomous driving into sub-problems of following and free driving, and identify existing expert models for longitudinal motion control of autonomous driving oriented towards sub-problems;

[0151] First, the longitudinal control sub-problem is divided: Safety, efficiency, and comfort are the longitudinal control objectives for autonomous driving. It is assumed that passengers assign equal weights to different driving objectives, meaning that different objectives are considered equally important. Based on whether there is a vehicle in front of the vehicle, the sub-problem is further divided into following-vehicle and free-roaming sub-problems. Taking the vehicle traveling on a straight road as an example, and considering road surface quality and existing longitudinal control models, the sub-problem is further divided into following-vehicle sub-problem on a smooth road surface and free-roaming sub-problem on a rough road surface.

[0152] Next, the expert models were determined: for each sub-problem, the intelligent driving model, the pre-trained deep reinforcement learning-based autonomous driving following model, and the pre-trained deep reinforcement learning-based autonomous driving model were selected as expert models.

[0153] II. Based on deep reinforcement learning algorithms, establish the state and action space of longitudinal control for autonomous driving and safety-oriented longitudinal control rules, and construct a reward function in combination with the diverse riding needs of passengers;

[0154] First, establish the state and action space.

[0155] Based on the established expert model, and considering the current longitudinal control problem in autonomous driving, a comprehensive state space and action space are established to provide sufficient information for problem solving and meet the vehicle control requirements in a real environment. The action for longitudinal control in autonomous driving is defined as longitudinal acceleration *a*, and the action space is set to [-3, 3] m / s². 2 The state space includes the longitudinal acceleration of the previous time step, the velocity of the current time step, the speed difference between the vehicles in front and behind, the distance between the vehicles in front and behind, and the maximum comfortable speed information. The state space can be specifically represented as follows:

[0156] s t =[a t-1 V t ,ΔV t ,ΔS t V p,t ]

[0157] Where: t is the current time step; t-1 is the previous time step; V t The speed of the autonomous vehicle; ΔV t Let ΔV be the speed difference between the vehicle in front of the autonomous vehicle and the autonomous vehicle itself. t =V l.t -V t ;ΔS t This represents the distance difference between the vehicle in front of the autonomous vehicle and the autonomous vehicle itself.

[0158] Then, establish action selection rules oriented towards driving safety.

[0159] The action selection rule for driving safety is that if the distance between the autonomous vehicle and the vehicle in front is less than the safe distance, the acceleration is selected as -3m / s². 2 Otherwise, the action output by deep reinforcement learning is used;

[0160]

[0161]

[0162] Where: a p The action is the output of the parallel learning-based autonomous driving decision-making and control model; x is the position of the autonomous vehicle; x lLet l represent the position of the vehicle ahead of the autonomous vehicle, where l indicates the vehicle ahead of the autonomous vehicle currently making the decision-making and control, including both manually driven vehicles and autonomous vehicles, hereinafter referred to as the vehicle ahead. d is the average length of the vehicle. t For safe distance; t r For reaction time; a min This represents the maximum deceleration for autonomous driving.

[0163] Reconstruct the reward function.

[0164] Based on the reward function settings of the expert model for solving the longitudinal control sub-problem of autonomous driving, a reward function capable of meeting diverse passenger needs is designed, including evaluation indicators related to safety, efficiency, and comfort, primarily collision time, safe distance, driving efficiency, and ride comfort. For safety, collision time (TTC) is used as the evaluation indicator. If the speed of the autonomous vehicle is lower than the speed of the vehicle in front, a rear-end collision is almost impossible, and the TTC value is infinite. If the speed of the autonomous vehicle is:

[0165]

[0166] Based on the TTC metric, an appropriate TTC threshold is selected, and the agent is penalized when the TTC value falls below this threshold.

[0167]

[0168] Among them: TTC thr This is the TTC threshold.

[0169] Given that the TTC metric is calculated solely based on the relative speed and distance between the autonomous vehicle and the vehicle ahead, to ensure driving safety as much as possible, the agent is penalized when the distance is less than the safe distance, and otherwise, no intervention is taken:

[0170]

[0171] Regarding efficiency, if the current headway of the autonomous vehicle is similar to that maintained by a human driver, then the autonomous vehicle is considered to have learned a certain level of human driving style and can guarantee driving efficiency. Therefore, statistical analysis of actual road traffic data is performed, and the agent is rewarded based on the probability that a human driver maintains a corresponding headway.

[0172]

[0173] Where: h is the headway; μ eh and σ ehTo fit the headway parameters using a logarithmic distribution, specifically, a human driving dataset from Shanghai was used to fit the headway distribution, yielding μ. eh =0.49 and σ eh =0.42.

[0174] Since headway only represents the relative relationship between the autonomous vehicle and the vehicle in front, a smaller headway indicates that the autonomous vehicle can closely follow the vehicle ahead. However, the speed of the autonomous vehicle also represents its driving efficiency. Therefore, comparing the speed of the autonomous vehicle with the maximum speed limit on the road, the closer the speed is to the maximum speed, the higher the efficiency. Exceeding the maximum speed limit indicates that the driving behavior is more dangerous, and the intelligent agent will be penalized. However, if the speed is below a certain limit, vehicles behind may easily reduce their speed, thus affecting the overall traffic efficiency. At the same time, excessively low speeds may cause abnormal vibrations in the vehicle body. Therefore, when the autonomous vehicle is driving at low speeds, a larger penalty is imposed on the intelligent agent to avoid negatively impacting traffic safety and comfort due to low speeds.

[0175]

[0176] Where: V max The maximum road speed is set to 16.67 m / s (60 km / h) in this embodiment, ρ is a penalty coefficient greater than 1, and is set to 30 in this embodiment. min To avoid affecting road traffic efficiency and personal riding comfort, the minimum speed is set to 3m / s in this embodiment.

[0177] For ride comfort, real-time longitudinal acceleration change rate and longitudinal acceleration are mainly used for evaluation.

[0178]

[0179]

[0180] Where: j is the rate of change of longitudinal acceleration, j max The maximum rate of change of longitudinal acceleration is given by α, which is a parameter of longitudinal acceleration.

[0181] Considering that vehicle vibration varies with different road surfaces and speeds, the maximum comfort speed for each road segment is determined based on vehicle vibration simulation experiments. This maximum comfort speed is obtained through vehicle vibration simulation experiments. An autonomous driving simulation environment is established, and road surface elevation and vehicle parameters are input into the simulation. By simulating vibration at different speeds on various road segments, and combining objective comfort evaluation indicators such as root mean square acceleration and annoyance rate, as well as the relationship between passenger subjective feelings and objective comfort evaluation indicators, the maximum speed that meets passenger comfort requirements is determined.

[0182] If the speed of the autonomous vehicle exceeds the maximum comfort speed for the current road segment, the current acceleration selection will cause discomfort to the passengers, and the agent will be penalized by the difference between this speed and the maximum speed. Conversely, if the current acceleration selection is below the maximum comfort speed, it is considered that the current acceleration selection will not negatively affect subjective comfort, and no additional penalty or reward will be imposed on the agent.

[0183]

[0184] The established reward function is a weighted sum of the above features:

[0185] r = w1R st +w2R sd +w3R eh +w4R es +w5R lj +w6R la +w7R v

[0186] Where: w1=1, w2=1, w3=2, w4=1, w5=1, w6=0.1, w7=5. In particular, the weights are set to ensure that the values ​​of each feature are on the same order of magnitude.

[0187] Third, a parallel learning framework and parallel simulation environment are established, with multiple expert models as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer.

[0188] First, establish a parallel learning framework.

[0189] The system consists of multiple driving expert models for specific sub-problems as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer. The expert models and worker models interact with the parallel environment, while the leader model interacts with the real environment.

[0190] Define the states of the leader model, worker model, and expert model at time step t as follows: The actions are respectively The rewards are as follows Define the experience pools for the leadership and worker models as R. c and R w , where R w Based on specific worker models, they can be divided into... n represents the total number of worker models, and m represents one of them. To maximize the effect of policy integration, the worker model and the leader model are designed as identical agents, meaning that their state space, action space, and reward function are the same.

[0191] Next, we set the information transfer mode between parallel models.

[0192] The leader model observes the state in the real environment and passes it down to the worker and expert models. The expert model then extracts state information based on its own state settings and passes its actions up to the worker and leader models. The worker model then combines the obtained state information with reference to the expert model's actions and passes its actions up to the leader model. Finally, the leader model decides its final action based on the observed state and the worker model's actions. This information transmission pattern can be expressed as follows:

[0193]

[0194]

[0195] in: For the leadership model after execution The observed states are indicated by ← and →, which represent the direction of information transmission.

[0196] Next, we will study the state transition function of a parallel environment.

[0197] The state transition function of the parallel environment is obtained by training the leader model with the interaction data of the real environment. Then, the worker model and the leader model are trained in parallel by passing state and action information in the parallel environment.

[0198] For longitudinal control in autonomous driving, the state transitions in real-world and parallel environments can be described using a relatively simple vehicle kinematics model:

[0199] V t+1 =V t +a t ΔT

[0200]

[0201] Where: ΔT is the interval between two adjacent time steps.

[0202] Specifically, during the training of the deep reinforcement learning model, after the corresponding longitudinal acceleration is performed in both the real and parallel environments, the actions of the environment at the next time step are recorded in the experience pool corresponding to each environment. Then, after the leader model in the real environment observes a new state, it transmits the state information of the real environment to the worker and expert models in the parallel environment. At this point, the transition state observed by the worker model is consistent with that of the leader model, while the expert model extracts the state information used for decision-making from the received state information according to its own settings.

[0203] Fourth, during the model training process, the worker model and the leader model combine their own experience and strategies, and take the vertical control scheme output by the expert model as a reference to select the better decision to execute, or use the mean of the vertical control actions as the basis for action selection to make decisions.

[0204] For expert model longitudinal control scheme

[0205] Based on the selected expert model's state settings, extract the expert model's state information from the state information transmitted by the leadership model, and solve the expert model's vertical control scheme.

[0206] For intelligent driving models, the main basis for decision-making is the longitudinal acceleration of autonomous vehicles:

[0207]

[0208]

[0209] Where: a max V0 represents the maximum longitudinal acceleration, and V0 represents the most desired driving speed, which can be set to the maximum road speed V. max s0 is the minimum distance, and T is the safe headway.

[0210] For a car-following model based on deep reinforcement learning, following existing research, the longitudinal acceleration information of the autonomous vehicle at the previous time step is incorporated into the state, and the expert model is trained in a simulation environment. The state of this expert model is...

[0211]

[0212] The reward function of this expert model only considers the collision time, headway, and longitudinal acceleration change rate.

[0213]

[0214]

[0215]

[0216]

[0217] The model is trained to convergence using a deep deterministic gradient strategy network, resulting in a trained carousel model based on deep reinforcement learning.

[0218] For the autonomous driving model based on deep reinforcement learning, the impact of road surface quality on driving comfort is considered. The "maximum comfort speed" within a 60m range of the road ahead is used to represent the maximum speed that can meet the comfort requirements of passengers. At the same time, the acceleration and current speed of the autonomous vehicle in the previous time step are added to the state.

[0219]

[0220] The reward function of this free-driving expert model takes driving efficiency into account. Vertical comfort longitudinal comfort and Energy consumption

[0221]

[0222]

[0223]

[0224]

[0225]

[0226]

[0227] P t =V t (1.1a t +9.81g t +0.132)+0.000302V t 3

[0228] Where: g is the road slope.

[0229] Vertical control scheme for worker model

[0230] The corresponding worker model is set up based on the number of expert models.

[0231] For both the worker and leader models, noise N is added to the action selection during training. t Specifically, noise N t As training progresses, the value decreases continuously when the time step exceeds the experience pool capacity, and becomes negligible once the model converges.

[0232] N t =ND(a t ,v t )

[0233]

[0234]

[0235] Where: ND is the generator function of normally distributed random numbers, τ is a reduction factor less than 1, which is set to 0.9995 in this embodiment, and M is the capacity of the experience pool.

[0236] In the initial training phase, to enable the worker model to acquire better decision-making capabilities, the longitudinal control scheme of the expert model will be implemented:

[0237]

[0238] in: For the longitudinal control scheme of the m-th worker model, Let w be the longitudinal control scheme for the m-th worker model, e be the expert model, and t be the control scheme for the m-th worker model. pi This indicates the strategy imitation phase in the early stages of training.

[0239] During the middle of training, the worker model selects the better longitudinal control scheme output from both the expert model and the worker model for execution:

[0240]

[0241] in: Let N be the network of critics for the m-th worker model, and t be the noise level. pe This indicates the policy integration phase during the middle of training.

[0242] At this point, during training, the performer network not only needs to maximize the cumulative reward that the worker model can obtain, but also needs to ensure that the suggestions it provides can lead the leader model to choose a better decision, that is, to maximize the cumulative reward of the leader model. At the same time, it is also necessary to ensure that the worker model's strategies are different, so that the worker model can provide references for the leader model's training from different perspectives. Therefore, the training objective of the worker model performer network is designed as follows:

[0243]

[0244] Where λ1 and λ2 are weight parameters. and μ c The strategies are the worker model and the leader model, respectively. and The critic network parameters for the worker model and the leader model are Q, respectively. c Let I be the critic network of the leadership model, D be the function that evaluates the distance between two strategies, and I be the set of worker models other than worker model m, which can be represented as I = [1, m) ∪ (m, n).

[0245] In the later stages of training, the worker model formulates a vertical control scheme according to its own strategy:

[0246]

[0247] Where: t ps This is the stage where the intelligent agent trains itself based on its own experience.

[0248] Meanwhile, the training objective of the worker model performer network is designed as follows:

[0249]

[0250] in: and ...

[0251] Vertical control scheme for leadership model

[0252] In the early stages of training, the leadership model based on deep reinforcement learning formulates a vertical control scheme according to its own strategy.

[0253] During the middle of training, the leader model selects the superior vertical control scheme from both the worker and leader models for implementation:

[0254]

[0255] Among them: Q c A network of critics for leadership models.

[0256] In the later stages of training, the leader model is selected based on the mean of the longitudinal control schemes output by the worker model and the leader model:

[0257]

[0258] The goal of updating the performer network parameters in the leadership model is to maximize the cumulative reward.

[0259]

[0260] V. Establish an evaluation index system for longitudinal control of autonomous driving, and test the performance of the trained leadership model in a dynamically changing driving environment;

[0261] First, establish a vertical control evaluation index system.

[0262] Based on the reward function design of the leadership model, collision time, speed, longitudinal acceleration, rate of change of longitudinal acceleration, and annoyance rate, which involve safety, efficiency, and comfort, are used as evaluation indicators of the longitudinal control model to determine whether the autonomous driving performance meets the passenger's riding needs.

[0263] Specifically, for speed and annoyance rate, the mean of all time steps on the test set is used to represent the overall efficiency and vertical comfort of the model. For longitudinal acceleration and rate of change of longitudinal acceleration, their absolute values ​​are used, and then the mean of all time steps on the test set is used to represent the overall longitudinal comfort of the model. For collision time, a threshold of 4 seconds is used. The number of time steps with collision time less than 4 seconds represents the number of dangerous driving events, and the number of time steps with collision time of infinity represents the number of safe driving events, thus evaluating the safety of the model.

[0264] Then, establish a test scenario for longitudinal control of autonomous driving.

[0265] The test scenario for longitudinal control of autonomous driving consists of road surfaces with different smoothness, traffic flow with different density, and road sections with different alignments, and is used to test the generalization ability of the trained model.

[0266] Test scenarios include, but are not limited to:

[0267] Scenario 1: Autonomous vehicles following each other on a smooth road surface;

[0268] Scenario 2: Autonomous vehicles freely driving on a smooth road surface;

[0269] Scenario 3: Autonomous vehicles following each other on rough roads;

[0270] Scenario 4: Autonomous vehicles freely driving on rough roads;

[0271] Scenario 5: Autonomous vehicles following each other on roads with alternating smooth and rough surfaces;

[0272] Scenario 6: Autonomous vehicles freely driving on roads with alternating smooth and rough surfaces;

[0273] Scenario 7: Autonomous vehicles are driving freely on a road surface with alternating smooth and rough surfaces. Suddenly, a vehicle in the lane next to the autonomous vehicle changes lanes and becomes the vehicle in front of the autonomous vehicle.

[0274] Finally, the autonomous driving longitudinal control model was tested.

[0275] For a certain test scenario, relevant environmental data are randomly selected from the actual vehicle-road database, and several experiments are carried out on the test scenario in the simulation environment. If the longitudinal control scheme output by the leader model can outperform the driving performance of the expert model in different driving scenarios, then the training of the autonomous driving longitudinal control model based on parallel learning is completed.

[0276] Example 2

[0277] As another embodiment of the present invention, this embodiment provides a domain control system for intelligent decision-making control of autonomous driving based on parallel learning, the structure of which is as follows: Figure 2 As shown, it includes: intelligent driving domain controller and chassis domain controller.

[0278] The intelligent driving domain controller includes a fusion perception module, a navigation and positioning module, an online training module, and a decision generation module. The autonomous vehicle uses vehicle-to-infrastructure communication to acquire motion data and decision control models of other vehicles beyond line-of-sight, road surface elevation data, and safe driving rules for the current road. The fusion perception module uses multi-source onboard sensors to perceive traffic conditions and road surface data within the sensor range of the surrounding area, matching the detection time and location to generate real-time fusion data. The online training module, based on the onboard deep reinforcement learning model, combines safe driving rules and decision control models of other vehicles in the current driving environment. Based on interaction data with the environment, it rapidly improves the performance of the decision control model within a short time, aiming for safety, efficiency, and comfort. The decision generation module outputs longitudinal, lateral, and vertical motion decisions for the autonomous vehicle based on the environmental perception data output by the fusion perception module and a parallel reinforcement learning model.

[0279] The chassis domain controller obtains the driving decision control scheme output from the intelligent driving domain controller through the gateway device, estimates the lateral, longitudinal, and vertical motion characteristic parameters of the vehicle using on-board sensor signals, and predicts the motion characteristic parameters within a certain time step in the future. Based on the vehicle state estimation, it generates lateral, longitudinal, and vertical motion control signals of the vehicle to complete the control of the vehicle trajectory, speed, and suspension.

[0280] The main data flow includes: First, vehicle-mounted sensors provide the autonomous vehicle's domain control system with vehicle-road data within its perception range and send the vehicle's current positioning data to the roadside intelligent unit. Next, the roadside intelligent unit sends vehicle-road data within a certain distance around and in front of the vehicle, along with current road safety rules (such as speed limits, safe headway, and safe distance between vehicles) to the autonomous vehicle. Then, the autonomous vehicle's intelligent driving domain control system combines the acquired vehicle-road data with the vehicle motion state data sent by the chassis domain controller, uses a parallel learning framework to update the decision control model online, and sends the driving decisions to the chassis domain controller via a gateway device. Finally, the chassis domain controller calculates control signals for vehicle trajectory, wheel speed, and suspension damping based on the driving decisions, thereby controlling the vehicle's lateral, longitudinal, and vertical movements.

[0281] The domain control system proposed in this solution can be applied to intelligent transportation systems, which include:

[0282] Intelligent connected vehicles, which are autonomous vehicles with network capabilities, can acquire real-time vehicle-road perception data and autonomous driving decision-making and control models on the vehicle side, and transmit the data and models to intelligent connected roadside units via the network. At the same time, the vehicle's electronic control unit can load the aforementioned parallel learning framework, and quickly adapt to the dynamically changing driving environment in real driving environment by using existing decision-making and control models.

[0283] The intelligent connected roadside unit integrates real-time vehicle-road perception data on a road segment basis. It selects the best driving performance data from the decision control models uploaded by vehicles, stores it, and uploads it to the cloud database. At the same time, it sends the integrated road segment vehicle-road perception data to intelligent connected vehicles entering the current road segment.

[0284] A cloud-based database stores real-time sensing data and decision-making control models.

[0285] The data processing engine processes data from cloud databases, including comparing driving performance with decision control models and releasing better decision control models to various intelligent connected vehicles.

[0286] In practical applications, firstly, intelligent connected vehicle groups transmit real-time perception data and decision-making control models via the network, including but not limited to perceived vehicle-to-infrastructure (V2I) data and onboard or trained decision-making control models. This data and model can be transmitted via the network to intelligent connected roadside units, then uploaded to the cloud for integration. Subsequently, after a vehicle enters a corresponding road segment, the network transmits V2I data and a superior decision-making control model to the autonomous vehicle. The data processing engine processes the data in the cloud database, including integrating V2I data from different time periods on the same road segment, and comparing the driving performance of different decision-making control models. The decision generation module of the intelligent driving domain control system of the autonomous vehicle is considered an intelligent agent, on which the aforementioned parallel learning framework, such as a deep reinforcement learning model based on parallel learning, can be loaded. This allows the vehicle to quickly adapt to dynamic driving environments and changing passenger needs based on existing decision-making control models from real-world driving environments.

[0287] In summary, this technical solution, based on existing autonomous driving decision-making and control models, fully utilizes these models and develops deep reinforcement learning algorithms to improve the efficiency of acquiring high-quality samples in complex scenarios and enhance the generalization ability of the deep reinforcement learning-based autonomous driving decision-making and control model for dynamic traffic environments. For a multi-agent collaborative training model, this solution employs parallel learning, using the policies output by existing autonomous driving decision-making and control models to guide agent learning. It also utilizes policy imitation and policy ensemble techniques to provide an efficient and reliable computational framework for autonomous driving decision-making and control. This technical solution effectively addresses the problems of low sample collection efficiency and limited generalization encountered by traditional single-agent deep reinforcement learning in solving complex decision-making and control problems, thereby reliably achieving autonomous vehicle decision-making and control in complex driving scenarios.

Claims

1. A method for intelligent decision-making and control of autonomous driving based on parallel learning, characterized in that, Includes the following steps: S1. Based on the autonomous driving task, the autonomous driving decision control problem is divided into sub-problems, and the existing sub-problem-oriented decision control model is used as the autonomous driving expert model. S2. Based on deep reinforcement learning algorithms, establish the state and action space for autonomous driving decision control, as well as safety-oriented decision control rules, and construct a reward function in combination with the diverse travel needs of passengers; S3. A parallel learning framework and parallel simulation environment are established, with multiple expert models as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer. S4. During the model training process, the worker model and the leader model combine their own experience and strategies, and take the decision control scheme output by the expert model as a reference to select the better decision to execute, or use the mean of the decision as the basis for action selection to formulate the final decision. S5. Establish an evaluation index system for autonomous driving decision control. Test the performance of the trained leader model in a dynamically changing driving environment. If the leader model test meets the set test requirements, the leader model observes the state in the current actual environment, combines the current actions output by the worker model, outputs the current decision information, and executes the decision information to complete vehicle control.

2. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Divide the decision control sub-problems With safety, efficiency, and comfort as the decision-making and control objectives for autonomous driving, the weights of passengers’ different riding needs are obtained. Based on the autonomous driving task, the decision-making and control problem is divided into several simpler sub-problems, and the sub-problems contain solutions for different scenarios in the same driving task. S12. Determine the expert model For each sub-problem, models for different scenarios are selected from existing models for autonomous driving decision control as expert models. Existing models for autonomous driving decision control include rule-based models, model predictive control, trained imitation learning models, trained deep reinforcement learning models, and trained deep inverse reinforcement learning models.

3. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Establish state and action space Based on the expert model, and considering the current autonomous driving decision-making and control problem, a state space that can provide sufficient information for problem solving and an action space that meets the vehicle control requirements in the real environment are established. The action of autonomous driving decision-making and control is set as a, where the action of longitudinal decision-making and control is longitudinal acceleration, and the action of lateral decision-making and control is lateral acceleration. The state space for autonomous driving decision control includes the acceleration of the previous time step and the velocity of the current time step, the speed difference between the vehicles in front and behind, the distance between the vehicles in front and behind, the road surface quality, and the motion information of surrounding vehicles. S22. Establish action selection rules for driving safety. The action selection rules for driving safety are as follows: if the distance between the autonomous vehicle and the vehicle in front is less than the longitudinal safety distance, the acceleration is selected as the maximum deceleration; otherwise, the longitudinal decision control action output by deep reinforcement learning is adopted. Based on the result of the autonomous vehicle lane change decision, the distance between the autonomous vehicle and the vehicles in front and behind in the target lane after the lane change is predicted. If the longitudinal safety distance requirement cannot be met, the autonomous vehicle continues to drive in the current lane; otherwise, the lateral decision control action output by deep reinforcement learning is adopted. S23. Constructing the reward function By combining the reward function settings of the expert model for solving the decision-making and control sub-problem of autonomous driving, a reward function that can meet the diverse needs of passengers is designed, including evaluation indicators related to safety, efficiency and comfort, including but not limited to collision time, safe distance, driving efficiency and ride comfort.

4. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 3, characterized in that, In step S23, the evaluation indicators of driving efficiency include the probability that the current headway is in the headway distribution obtained from the measured road traffic data, and the difference between the current vehicle speed and the maximum road speed. The evaluation indicators for ride comfort include longitudinal acceleration rate of change, longitudinal acceleration, lateral acceleration rate of change, lateral acceleration, the difference between the vehicle's current speed and the maximum comfort speed for the current road segment, where the maximum comfort speed is obtained based on vehicle vibration simulation. An autonomous driving simulation environment is established, and road surface elevation and vehicle parameters are input into the simulation environment. By simulating vibrations at different speeds on various road segments, and combining the root mean square value of acceleration and the annoyance rate, the relationship between objective comfort evaluation indicators, passenger subjective feelings, and objective comfort evaluation indicators is established, thus determining the maximum speed that can meet passenger comfort requirements.

5. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 4, characterized in that, Step S3 specifically includes the following steps: S31. Establish a parallel learning framework The system consists of multiple driving expert models for specific sub-problems as the bottom layer, multiple worker models based on deep reinforcement learning as the middle layer, and a leader model based on deep reinforcement learning as the top layer. The expert models and worker models interact with the parallel environment, while the leader model interacts with the real environment. S32. Set the information transfer mode between parallel models. The leader model observes the state in the real environment and passes it down to the worker model and the expert model. Then, the expert model extracts the state information based on its own state settings and passes the expert model's actions up to the worker model and the leader model. The worker model then combines the obtained state information and refers to the expert model's actions and passes the worker model's actions up to the leader model. Finally, the leader model makes the final decision based on the observed state and the worker model's actions. S33, Learning Parallel Environment State Transition Functions The state transition function of the parallel environment is obtained by training the leader model with the interaction data of the real environment. Then, the worker model and the leader model are trained in parallel by passing state and action information in the parallel environment.

6. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 5, characterized in that, The training process of the expert model in step S4 is as follows: based on the state settings of the selected expert model, extract the state information of the expert model from the state information transmitted by the leadership model, and solve the decision control scheme of the expert model.

7. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 6, characterized in that, The training process of the worker model in step S4 is as follows: in the early stage of training, the decision control scheme of the expert model is executed; in the middle stage of training, the better decision control scheme output by the expert model and the worker model is selected and executed; and in the later stage of training, the decision control scheme is formulated according to its own strategy.

8. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 7, characterized in that, The training process of the leadership model in step S4 is as follows: in the early stage of training, a decision control scheme is formulated according to its own strategy; in the middle stage of training, the better decision control scheme between the worker model and the leadership model is selected for execution; and in the later stage of training, the average value of the decision control schemes output by the worker model and the leadership model is selected for execution with reference.

9. The intelligent decision-making and control method for autonomous driving based on parallel learning according to claim 8, characterized in that, The specific process of testing the performance of the trained leadership model in step S5 includes: S51. Establish a decision-making control evaluation index system Based on the reward function design of the leadership model, collision time, speed, acceleration, rate of change of acceleration, and annoyance rate, which involve safety, efficiency, and comfort, are used as evaluation indicators of the decision control model to determine whether the autonomous driving performance meets the passenger's riding needs. S52. Establish test scenarios for autonomous driving decision control. The autonomous driving decision-making and control test scenario consists of road surfaces with different smoothness, traffic flow with different density, and road sections with different alignments, and is used to test the generalization ability of the trained model. S53, Testing the Autonomous Driving Decision Control Model For a given test scenario, relevant environmental data are randomly selected from the actual vehicle-road database. Several experiments are conducted on the test scenario in a simulation environment. If the decision control scheme output by the leader model can outperform the driving performance of the expert model in different driving scenarios, then the training of the autonomous driving decision control model based on parallel learning is complete.

10. An autonomous driving domain control system applying the parallel learning-based intelligent decision-making control method for autonomous driving as described in any one of claims 1 to 9, characterized in that, The system includes an intelligent driving domain controller and a chassis domain controller. The intelligent driving domain controller comprises a fusion perception module, a navigation and positioning module, an online training module, and a decision generation module. The autonomous vehicle uses vehicle-to-infrastructure communication to acquire motion data and decision control models of other vehicles beyond line-of-sight, road surface elevation data for each road, and safe driving rules for the current road. The fusion perception module uses multi-source onboard sensors to perceive traffic conditions and road surface data within the sensor's perception range around the vehicle, matching them based on detection time and location to generate real-time fusion data. The online training module, based on an onboard deep reinforcement learning model, combines safe driving rules and decision control models of other vehicles in the current driving environment to rapidly improve the performance of the decision control model within a short time based on interaction data with the environment. The decision generation module outputs longitudinal, lateral, and vertical motion decisions for the autonomous vehicle based on the environmental perception data output by the fusion perception module and a parallel reinforcement learning model. The chassis domain controller obtains the driving decision control scheme output by the intelligent driving domain controller through the gateway device, estimates the lateral, longitudinal, and vertical motion characteristic parameters of the vehicle using on-board sensor signals, predicts the motion characteristic parameters within a set time step in the future, and generates lateral, longitudinal, and vertical motion control signals based on the vehicle state estimation to complete the control of the vehicle trajectory, speed, and suspension.

Images