Human-machine cooperation based reinforcement learning automatic driving method and system

Abstract

The application discloses a kind of based on man-machine cooperation's reinforcement learning automatic driving method and system, belong to the field of automatic driving technology. Including: the current state of the automatic driving vehicle is obtained;The current state of the automatic driving vehicle is input into the trained automatic driving network and is processed to obtain the decision action of the automatic driving vehicle current time;Introduce reward-free policy learning method, with human takeover cost minimization, action distribution entropy and maximum stability as the goal, the automatic driving network is trained by explicit intervention mechanism and implicit intervention mechanism. The value function of explicit and implicit intervention is designed, with the minimum human takeover cost and comfort maximization as the goal, to train agent to drive in invisible traffic scene;Solve the strategy optimization problem of automatic driving car in complex dynamic traffic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and particularly relates to a reinforcement learning automatic driving method and system based on human-machine cooperation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In recent years, automated vehicles (AVs) have received extensive attention as a disruptive technology, as they are expected to greatly improve travel experience by improving road safety, optimizing traffic flow, and improving fuel economy. However, the realization of automated driving is full of challenges, the most critical of which is how to develop appropriate driving strategies to optimize traffic efficiency while ensuring safety.

[0004] Automated vehicles usually rely on sensors to perceive the environment and generate online driving strategies after a large amount of offline training. These strategies cover operational level control (such as acceleration, braking, and steering signals), tactical level decision-making (such as lane changing and lane keeping), and strategic level route planning. However, due to the high dynamicity and unpredictability of the road environment, especially the complex interaction between automated vehicles and human-driven vehicles (HVs), it is extremely challenging to develop a universally applicable and reliable automated driving strategy.

[0005] Most traditional automated driving systems use reinforcement learning (RL) methods, and RL agents obtain feedback from the environment through trial-and-error exploration to optimize driving strategies, but such autonomous exploration inevitably exposes agents to high-risk situations, limiting their application in safety-critical tasks. In addition, it is also extremely challenging to design a reward function that can cover all possible driving behaviors.

[0006] To overcome these problems, in recent years, the "human-in-the-loop learning" (HL) method has been proposed, which improves the safety and sampling efficiency of the system by injecting human intelligence into RL or other learning paradigms. Under the HL framework, humans can directly participate in the training process through intervention, demonstration or evaluation, thereby helping the agent to learn a suitable strategy more quickly and safely. HL methods have achieved remarkable success in the fields of robot control and large language model development, however, the application of HL in the field of autonomous driving still faces some challenges: existing researches mostly rely on passive human feedback, such as suggested actions or evaluated trajectories, which may not be sufficient to ensure safety in complex scenarios of autonomous driving. At the same time, frequent human intervention also increases the cognitive burden of human experts and affects the training efficiency.

[0007] In addition, existing researches often only focus on the safety of individual autonomous vehicles, such as collision avoidance strategies, while ignoring the impact of these safe maneuvers on the overall traffic flow efficiency. However, researches in the field of transportation have shown that the behavior of autonomous vehicles can significantly affect the overall traffic flow, including traffic flow disturbance, congestion and emission problems. Although some researches have begun to combine data-driven methods with rule-based traffic models, they mainly focus on the optimization of specific scenarios, such as intersection management and ramp merging, and have not been able to comprehensively solve the problem of strategy optimization for autonomous vehicles in complex dynamic traffic environments. SUMMARY

[0008] To solve the problems of the prior art, the present application provides a human-machine collaboration based reinforcement learning autonomous driving method, system, electronic device, computer readable storage medium and computer program product, which optimizes for the goals of maximum smoothness, maximum action distribution entropy, minimum explicit intervention and minimum implicit intervention, ensures the safety of a single autonomous vehicle in an uncertain environment, and guarantees the smoothness and comfort during vehicle travel.

[0009] In a first aspect, the present application provides a human-machine collaboration based reinforcement learning autonomous driving method;

[0010] A human-machine collaboration based reinforcement learning autonomous driving method, comprising:

[0011] Obtaining the current state of the autonomous vehicle;

[0012] Inputting the current state of the autonomous vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous vehicle at the current time;

[0013] The reward-free policy learning method is introduced, human takeover cost minimization, action distribution entropy maximization and stationarity maximization are taken as objectives, and the autonomous driving network is trained through an explicit intervention mechanism and an implicit intervention mechanism.

[0014] In some embodiments, the reward-free policy learning method specifically comprises: updating the Q value of the current time according to the discount factor and the maximum Q value corresponding to the generated policy of the next time; and optimizing the driving policy of the agent by taking the Q value maximization as an objective.

[0015] In some embodiments, the training of the autonomous driving network through the explicit intervention mechanism for the purpose of minimizing the human takeover cost specifically comprises:

[0016] Based on the preset explicit intervention value function, the difference between the agent Q value of the autonomous driving vehicle automatic action and the agent Q value of the human action is minimized as a constraint for optimization.

[0017] In some embodiments, the training of the autonomous driving network through the implicit intervention mechanism for the purpose of maximizing the action distribution entropy and stationarity specifically comprises:

[0018] Based on the preset implicit intervention value function, the difference between the agent Q value of the autonomous driving vehicle automatic action and the agent Q value of the human action is minimized as a constraint for optimization.

[0019] Based on the preset action distribution entropy function, the difference between the reward-free agent value and the entropy regularization term is minimized as a constraint for optimization.

[0020] In some embodiments, the training of the autonomous driving network further comprises: obtaining state information, image information and exploration data of the autonomous driving vehicle and expert intervention data and storing them into an experience replay pool.

[0021] In some embodiments, the autonomous driving network is an improved off-line policy-based actor-critic algorithm.

[0022] In a second aspect, the present application provides a human-machine cooperation based reinforcement learning autonomous driving system.

[0023] A human-machine cooperation based reinforcement learning autonomous driving system comprises:

[0024] The acquisition module is configured to acquire the current state of the autonomous driving vehicle.

[0025] The decision module is configured to input the current state of the autonomous driving vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current time.

[0026] Among them, the reward-free policy learning method is introduced, the smoothness maximization and the human takeover cost minimization are taken as the goals, and the automatic driving network is trained through an explicit intervention mechanism and an implicit intervention mechanism.

[0027] In a third aspect, the present application provides an electronic device;

[0028] An electronic device includes a memory, a processor, and a computer program stored on the memory, the processor executing the computer program to implement the steps of the above-mentioned human-machine collaboration-based reinforcement learning automatic driving method.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium;

[0030] A computer-readable storage medium has a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned human-machine collaboration-based reinforcement learning automatic driving method.

[0031] In a fifth aspect, the present application provides a computer program product;

[0032] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned human-machine collaboration-based reinforcement learning automatic driving method.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. The technical solution provided by the present application is a learning mode inspired by human learning process, which effectively injects human intelligence into artificial intelligence learning, allows human experts to act as tutors of artificial intelligence, and performs real-time supervision, intervention and training in the learning process, and improves the convergence speed of training AI through explicit and implicit intervention mechanisms.

[0035] 2. The technical solution provided by the present application proposes a deep reinforcement learning framework, which uses the data collected by free exploration and partial human demonstration as a training source, not only improves the ability of the autonomous vehicle in complex traffic scenarios in safety, but also optimizes the performance of the vehicle in comfort, ensuring that the driving experience in dynamic traffic environment is more stable and safe.

[0036] 3. The technical solution provided by the present application introduces a reward-free policy learning method, in the training process, the autonomous vehicle learns which state-action pairs are unsafe or inappropriate through the intervention signal of human takeover behavior; this explicit intervention signal not only conveys human intention in training, but also accelerates the convergence of the policy.

[0037] 4、The technical scheme provided by the present application introduces an entropy maximization strategy to encourage the agent to explore in a safe state; in the absence of takeover signal intervention, the action distribution entropy of the agent is maximized, thereby ensuring sufficient exploration within the allowed state-action space.

[0038] 5、The technical scheme provided by the present application reduces the traffic flow interference on the subsequent human-driven vehicle by minimizing the interference cost caused by the agent from the current state and action; the interference cost is estimated by indicators such as vehicle state and vehicle smoothness, and is used as a constraint condition in training to reduce the negative impact on the surrounding traffic flow.

[0039] 6、The technical scheme provided by the present application designs a mechanism to minimize takeover cost; this mechanism aims to reduce the agent's dependence on human takeover signals, thereby gradually improving its autonomy, and ultimately enabling it to independently deal with various uncertainty scenarios in complex traffic environments. BRIEF DESCRIPTION OF DRAWINGS

[0040] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application.

[0041] Figure 1 The flowchart of the human-machine cooperation-based reinforcement learning automatic driving method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0043] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0044] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0045] Embodiment one

[0046] The driving strategy optimization of the existing automatic driving vehicle cannot balance the efficiency of the traffic flow and the system-level performance under the premise of ensuring the safety of the automatic driving vehicle, therefore, the application provides a human-machine cooperation based reinforcement learning automatic driving method to optimize with the minimum human takeover cost and the maximum comfort.

[0047] Next, the human-machine cooperation based reinforcement learning automatic driving method disclosed in the embodiment will be described in detail. Figure 1 The human-machine cooperation based reinforcement learning automatic driving method comprises the following steps:

[0048] S1, obtaining the state information and image information of the automatic driving vehicle and storing them into an experience replay pool.

[0049] Meanwhile, the exploration data is stored in the experience replay pool, and the experience replay pool is divided into an expert experience replay pool and an agent experience replay pool, the exploration data of the automatic driving vehicle after the intervention of the expert is stored in the expert experience replay pool, and the exploration data of the automatic driving vehicle is stored in the agent experience replay pool.

[0050] S2, inputting the state information and image information into the trained automatic driving network for processing to obtain the decision action of the automatic driving vehicle at the current time.

[0051] In the embodiment, the automatic driving network is an off-policy actor-critic algorithm, which is composed of an actor module and a critic module, the actor module is used to generate a policy, that is, to determine the optimal action taken under a given state, and the critic module is responsible for evaluating the value of the action generated by the actor module under the current state, thereby guiding the actor module to improve the policy.

[0052] The off-policy actor-critic algorithm has the characteristic of off-policy, that is, the learning of the policy can use the experience samples generated by different policies. This learning mechanism can significantly improve the sample efficiency, and fully utilize the historical data in the process of policy improvement, thereby having higher robustness and stability in complex and variable driving scenarios.

[0053] A key feature of the off-policy actor-critic algorithm is that it stores and utilizes offline data through the experience replay pool (Replay Buffer), thereby being able to optimize the policy without relying on the current policy. Specifically, the samples in the experience replay pool include a state-action-reward-next state four tuple (s t ,α t ,r t ,s t+1). At each training time, a certain amount of random sampled data is extracted from the expert experience replay pool and the agent experience replay pool for updating the target functions of the Critic module and the Actor module respectively, so as to break the time correlation, reduce the correlation between samples, and improve the convergence speed of the model, and at the same time, to solve the problem that the expert experience is gradually diluted as the data volume increases.

[0054] In the embodiment, a reward-free policy learning method is introduced in the Off-Policy Actor-Critic algorithm. The goal of the driving policy learning task is to reflect human preferences into the behavior of the agent, and the takeover operation of the human expert can be regarded as dissatisfaction (such as insecurity or poor performance) with the current behavior of the agent, and the non-takeover indicates that the behavior meets the human expectation. Based on this, in the embodiment, an agent value function is used to represent the human preference instead of relying on the traditional reward function.

[0055] In the specific implementation, the Critic module evaluates the policy π generated by the Actor by learning a state-action value function Q π (s t ,a t ), which is defined as follows:

[0056]

[0057] Where E represents the expected value, r t is the reward value at the current time t, γ is a discount factor for balancing current and future rewards, and p(·|s t ,a t ) represents the state transition probability. The Actor module updates the policy by maximizing the expected value of Q π (s t ,a t ), and the objective function is defined as:

[0058]

[0059] Where θ and θ respectively represent the neural network parameters of the Actor module and the Critic module. In order to stabilize the policy update process, the objective function of the Critic is defined as minimizing the estimation error of the state-action value function Q:

[0060]

[0061] That is, the Q value update is changed from to to avoid the limitations of manually designing the reward function.

[0062] Based on the learned agent value function, the policy π θ:

[0063]

[0064] The standard RL task is converted into a reward-free setting, enabling the agent to learn from human participation and avoiding complex reward design. An ideal driving strategy should adapt to various traffic scenarios, such as overtaking, avoiding, emergency stopping, and interaction with other vehicles.

[0065] At the same time, the maximum smoothness, the maximum action distribution entropy, the minimum explicit intervention, and the minimum implicit intervention are used as the comprehensive objectives of the autonomous driving network, and the existing human data is used to optimize the strategy of the autonomous vehicle in mixed traffic. The comprehensive objectives are as follows:

[0066]

[0067] In the formula, E represents the expected value, π represents the autonomous driving strategy, V EX (s,a AV ) represents the explicit intervention value function, and Q IM (s,a AV ) represents the implicit intervention value function, s represents the state, a AV represents the action of the autonomous vehicle.

[0068] The comprehensive objectives take into account the smoothness and the burden of experts, and improve the overall performance of the vehicle fleet. By maximizing the smoothness proxy value function to reflect the expectations of humans for driving behavior and ensure smooth operation. In the state that meets the expectations of humans, the action distribution entropy H(π) is maximized to promote the agent to actively explore the state-action space within the allowed range. By minimizing the explicit intervention value function Q EX (s,a AV ), the frequency of tutor takeover is reduced. By minimizing the implicit intervention value function Q IM (s,a AV ), the influence of the action on the smoothness of the vehicle fleet is reduced.

[0069] As an embodiment, the process of training the autonomous driving network specifically includes:

[0070] Step 1, obtaining observation space data and action information.

[0071] Here, the observation space data includes numerical observation space and visual observation space, the numerical observation space includes the current state of the vehicle, navigation information and laser radar surrounding environment information, and the visual observation space includes the image input of the camera. The strategy network generates corresponding actions based on the observation space data, and the actions represent the control instructions taken by the autonomous vehicle in the current state, such as steering, accelerating or decelerating, etc.

[0072] The action information includes the actions in the exploration process of the autonomous vehicle and the actions after the takeover of the human experts, which are respectively stored in the vehicle experience replay pool and the expert experience replay pool.

[0073] Step 2, initialize the parameters and data of the autonomous driving network, set the network parameters θ of the Actor module, the network parameters of the Critic module, and the experience replay pool B for storing the state, action and intervention data collected during the training process.

[0074] Step 3, based on the set comprehensive target, perform an iterative training process through the training set, and continue training until the stop condition is met (for example, reaching the set training times or performance indicators), wherein each iteration contains multiple training episodes.

[0075] State update and action selection within a single episode: in each episode state s t , the agent selects and executes action a AV according to the strategy π AV ; after the action a AV is executed, the environment returns a new state s t+1 , and the agent updates the decision according to the state.

[0076] Human expert intervention judgment: the human expert judges whether intervention is needed under the current state s t , and records the intervention identifier I(s t , a AV ); if I(s t , a AV ) = True, it means that the human expert thinks that the current action a AV of the agent is not appropriate and needs to be taken over; otherwise, the action a AV of the agent is continued to be executed.

[0077] Execution of human expert takeover action: if the intervention identifier is true, the human expert executes action a human , and replaces the action a AV of the agent with the action a human , so as to ensure that the vehicle can take reasonable behavior in complex or dangerous states; if the intervention identifier is false, the action a AV of the agent is continued to be executed.

[0078] Compute intervention cost: When the human expert does not intervene in the last state but needs to intervene in the current state, compute explicit intervention cost C EX (s t ,a AV ), which measures the deviation of action a AV from human action a human in the current state; when the acceleration acc AV of the agent exceeds the threshold λ and did not exceed it in the previous time, compute implicit intervention cost C IM (s t ,a AV ), which evaluates the impact of action a AV on vehicle stability in the current state according to the vehicle stability model.

[0079] Data recording and experience storage: Store the current state s t , action a AV , human action a human , intervention identifier I(s t ,a AV ), and next state s t+1 to the experience replay pool B to provide data support for subsequent training.

[0080] Policy and value function update: Update the parameters of the agent value function explicit intervention value function Q EX (s,a AV ), implicit intervention value function Q IM (s,a AV ), and action distribution entropy H(π) according to the optimization objectives established earlier.

[0081] Exemplary, the specific learning process is as follows:

[0082] (1) Learn explicit intervention

[0083] Efforts should be made to make the behavior of the agent close to the demonstration of the human expert, while avoiding the execution of actions that have been intervened. The optimal strategy will always choose the action with the highest Q value. Therefore, if the human expert intervenes in some states, the Q value of human action a human ~ π human in that state should be higher than that of other actions, while the Q value of agent action a AV ~ π AV should be lower than that of other actions.

[0084] When the agent frequently relies on the human expert to take over, it may lead to over-reliance, causing abnormal performance without human supervision and leading to a decrease in automation level. To reduce the cognitive load of the expert instructor, introduce takeover cost C EX (s,aAV ), to measure the similarity between the agent's action and the human's action. When the agent's behavior deviates too much from the human's intention, a higher takeover cost will be generated, thereby inhibiting the behavior that does not meet the human's expectation. The takeover cost is represented as:

[0085]

[0086] where a AV and a human represent the agent's and human's action in state s respectively. The takeover cost C EX (s,a AV ) is larger, indicating that the similarity between the agent's and human's action is lower, and thus the agent is punished more.

[0087] An explicit intervention value function Q EX (s,a AV ) is introduced to represent the expected cumulative cost of a state-action pair leading to human takeover:

[0088]

[0089] By minimizing the takeover cost and the explicit intervention value function, the agent can improve the automation level while reducing unnecessary human intervention, thereby effectively reducing the cognitive load of the human instructor and accelerating the policy learning process.

[0090] The hybrid behavior policy π mix (a|s) is used for environment exploration, and part of the transition sequence collected during takeover is regarded as part of the demonstration I(s t ,a AV ,s t+1 ,…), and stored in the experience replay pool B. The reward and cost of the environment in the experience replay pool B do not need to be recorded.

[0091] Since only part of the demonstration can be learned from B, this leads to a distribution shift problem. To this end, the CQL method is used to train with human-generated offline data. The data s t ,a AV ,a human is sampled from the experience replay pool B, and the agent Q-value of the human action is labeled as The agent Q-value of the agent action is

[0092] The optimization problem of the agent value function is then formulated as:

[0093]

[0094] By minimizing the difference between the agent's proxy Q-value and the human expert's, the agent's actions can be guided towards the high-value state-action subspace preferred by the human expert.

[0095] (2) Learning implicit intervention

[0096] In implicit intervention learning, the goal is to optimize the agent's behavior to be as close as possible to the human expert's driving style while avoiding executing actions that the human has intervened on to ensure the vehicle's smoothness. Specifically, the implicit intervention value function is defined as to evaluate the impact of the agent's actions on the vehicle's smoothness.

[0097] The formula for calculating the implicit intervention value function is:

[0098]

[0099] The optimization goal is to make the agent's behavior consistent with the human expert's demonstration while avoiding taking action in states where the expert has intervened. To this end, the objective optimization problem can be represented as:

[0100]

[0101] During training, the state s t , action a AV and the human expert's action a human are stored in the experience replay pool B for updating the proxy value function.

[0102] The optimization problem can be represented as:

[0103]

[0104] By minimizing the difference between the agent's actions and the human expert's behavior, the agent's decision-making safety and reliability are enhanced while improving the vehicle's smoothness. This module aims to ensure that the agent always follows the human's driving intentions during driving, improving the smoothness of the driving experience.

[0105] (3) Learning implicit intervention with entropy regularization

[0106] In the reward-free framework, entropy regularization is introduced to maximize the entropy H(π(s)) of the policy to encourage the agent to explore widely in the state-action space, thereby improving learning efficiency. Traditional methods rely on external reward signals, while the entropy regularization guides policy updates without explicit rewards.

[0107] The entropy of the policy is defined as:

[0108] H(π(s))=-∑ a π(a|s)logπ(a|s);

[0109] To balance exploration and exploitation, an entropy regularization term is added to the policy update target, and the optimization objective of the agent value function is as follows:

[0110]

[0111] wherein, is the reward-free agent value function, and α is a hyperparameter that adjusts the intensity of entropy. When the difference between the two decreases, the policy tends to be uniformly distributed, avoiding falling into local optimal solutions, thereby improving the robustness and adaptability of the policy.

[0112] Finally, through this optimization objective, the agent can learn a wide range of state-action distributions in a reward-free environment, effectively improving the exploration and behavior diversity of the policy, and thus better adapting to complex environments.

[0113] Repeat the training until it converges, and repeatedly repeat the above process until the preset training target is reached, such as model convergence, performance reaching the standard, or training rounds being exhausted.

[0114] Next, in order to verify the effect of the human-machine collaboration-based reinforcement learning automatic driving method described in the embodiment, specific experimental verification is carried out.

[0115] In order to fully include dangerous and long-tail scenarios, MetaDrive is used to generate various driving scenarios, and each training set includes 50 different scenarios. The road scenarios are composed of various typical street types, such as straight roads, slopes, intersections, roundabouts, curves, t-intersections, and crossroads. The surrounding vehicles are controlled using two different models: the IDM model for longitudinal dynamics, and the lane-changing model for lateral motion causing minimum overall braking. In addition, traffic flows of different densities are introduced to interact with the AV. To increase the complexity of the driving task, each scenario contains randomly placed obstacles, such as moving traffic vehicles, stationary traffic cones, and triangular warning signs.

[0116] The generated driving scenarios are divided into two groups: training sets and test sets, each with 50 different scenarios. The AV agent is specifically trained on the training set and then evaluated on the reserved test set. At the beginning of each iteration, a scene is randomly selected from the training set or the test set. After each training iteration, the learned agent is deployed to the test environment without any human supervision, and the safety violations, training deployment, and success rate given by the environment are recorded and presented in Table 1. The results show that the success rate is high, the safety violation situation is less, and the training time is greatly shortened.

[0117] Table 1

[0118]

[0119] Embodiment Two

[0120] The embodiment discloses a human-computer cooperation-based reinforcement learning automatic driving system, comprising:

[0121] The acquisition module is configured to acquire a current state of the automatic driving vehicle.

[0122] The decision module is configured to input the current state of the automatic driving vehicle into the trained automatic driving network for processing to obtain a decision action of the automatic driving vehicle at the current time.

[0123] The reward-free policy learning method is introduced to train the automatic driving network by taking the maximum smoothness and the minimum human takeover cost as the target through an explicit intervention mechanism and an implicit intervention mechanism.

[0124] It should be noted that the acquisition module and the decision module correspond to the steps in Embodiment One, and the above modules and the corresponding steps have the same examples and application scenarios, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0125] Embodiment Three

[0126] The embodiment three of the present application provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are executed by the processor, the steps of the above-mentioned human-computer cooperation-based reinforcement learning automatic driving method are completed.

[0127] Embodiment Four

[0128] The embodiment four of the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the steps of the above-mentioned human-computer cooperation-based reinforcement learning automatic driving method are completed.

[0129] Embodiment Five

[0130] The embodiment five of the present application provides a computer program product, comprising computer programs / instructions, when the computer programs / instructions are executed by the processor, the steps of the above-mentioned human-computer cooperation-based reinforcement learning automatic driving method are implemented.

[0131] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0132] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0133] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0134] The above description of the various embodiments can have emphasized certain aspects of each embodiment, which have not been described in detail in other embodiments. For those embodiments not described in detail, reference is made to the relevant description of the other embodiments.

[0135] The above description is only preferred embodiments of the present application and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of the present application.

Claims

1. A human-robot collaboration based reinforcement learning automatic driving method, characterized in that, The method comprises the following steps: obtaining a current state of an autonomous vehicle; inputting the current state of the autonomous vehicle into a trained autonomous driving network for processing to obtain a decision action of the autonomous vehicle at the current time; wherein a reward-free policy learning method is introduced, and the reward-free policy learning method specifically comprises: updating a Q value at the current time according to a discount factor and a maximum Q value corresponding to a policy generated at a next time; and optimizing a driving policy of an agent with a maximum Q value as a target; training the autonomous driving network through an explicit intervention mechanism with a minimum human takeover cost as a target, and the training specifically comprises: optimizing, based on a preset explicit intervention value function, with minimization of a difference between a proxy Q value of an automatic action of the autonomous vehicle and a proxy Q value of a human action as a constraint; training the autonomous driving network through an implicit intervention mechanism with maximum action distribution entropy and maximum stationarity as targets, and the training specifically comprises: optimizing, based on a preset implicit intervention value function, with minimization of a difference between the proxy Q value of the automatic action of the autonomous vehicle and the proxy Q value of the human action as a constraint; and optimizing, based on a preset action distribution entropy function, with minimization of a difference between a reward-free proxy value and an entropy regularization term as a constraint; taking maximum stationarity, maximum action distribution entropy, minimum explicit intervention and minimum implicit intervention as comprehensive targets of the autonomous driving network, and optimizing a policy of the autonomous vehicle in mixed traffic by using existing human data, and the comprehensive targets are represented as follows: ； wherein, represents an expected value, represents an autonomous driving policy, represents a stationarity agent value function, represents an action distribution entropy function, represents an explicit intervention value function, represents an implicit intervention value function, represents a state, represents an autonomous vehicle action. 2.The human-machine collaboration based reinforcement learning autonomous driving method of claim 1, wherein, The training of the autonomous driving network further comprises: obtaining state information, image information and exploration data of the autonomous vehicle and exploration data after intervention of an expert and storing the data into an experience replay pool. 3.The human-machine collaboration based reinforcement learning autonomous driving method of claim 1, wherein, The autonomous driving network is an improved actor-critic algorithm based on an offline policy.

4. A human-robot collaboration based reinforcement learning autonomous driving system, characterized in that, The method comprises the following steps: an obtaining module configured to obtain a current state of an autonomous vehicle; a decision module configured to input the current state of the autonomous vehicle into a trained autonomous driving network for processing to obtain a decision action of the autonomous vehicle at the current time; wherein a reward-free policy learning method is introduced, and the reward-free policy learning method specifically comprises: updating a Q value at the current time according to a discount factor and a maximum Q value corresponding to a policy generated at a next time; and optimizing a driving policy of an agent with a maximum Q value as a target; training the autonomous driving network through an explicit intervention mechanism with a minimum human takeover cost as a target, and the training specifically comprises: optimizing, based on a preset explicit intervention value function, with minimization of a difference between a proxy Q value of an automatic action of the autonomous vehicle and a proxy Q value of a human action as a constraint; training the autonomous driving network through an implicit intervention mechanism with maximum action distribution entropy and maximum stationarity as targets, and the training specifically comprises: optimizing, based on a preset implicit intervention value function, with minimization of a difference between the proxy Q value of the automatic action of the autonomous vehicle and the proxy Q value of the human action as a constraint; and optimizing, based on a preset action distribution entropy function, with minimization of a difference between a reward-free proxy value and an entropy regularization term as a constraint; With the maximization of stationarity, the maximization of action distribution entropy, the minimization of explicit intervention and the minimization of implicit intervention as the comprehensive goals of the automatic driving network, the existing human data is used to optimize the strategy of the automatic driving vehicle in mixed traffic, and the comprehensive goals are represented as follows: ； wherein denotes an expected value, denotes an autonomous driving policy, denotes a stationarity proxy value function, denotes an action distribution entropy function, denotes an explicit intervention value function, denotes an implicit intervention value function, denotes a state, denotes an autonomous vehicle action.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 4. The processor executes the computer program to implement the steps of the human-machine collaboration based reinforcement learning automatic driving method according to any one of claims 1-3.

6. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the human-machine collaboration based reinforcement learning automatic driving method according to any one of claims 1-3.

7. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the human-machine collaboration based reinforcement learning automatic driving method according to any one of claims 1-3. The computer program / instruction is executed by the processor to implement the steps of the human-machine collaboration based reinforcement learning automatic driving method according to any one of claims 1-3.

Images