Reinforcement learning automatic driving method and system based on man-machine cooperation

Abstract

The invention discloses a reinforcement learning automatic driving method and system based on man-machine cooperation, and belongs to the technical field of automatic driving. The method includes: acquiring a current state of the autonomous vehicle; inputting the current state of the automatic driving vehicle into the trained automatic driving network for processing so as to obtain a decision-making action of the automatic driving vehicle at the current moment; a reward-free strategy learning method is introduced, and an automatic driving network is trained through an explicit intervention mechanism and an implicit intervention mechanism by taking human takeover cost minimization and action distribution entropy and stability maximization as targets. An explicit and implicit intervention value function is designed, and an intelligent agent is trained to drive in an invisible traffic scene with minimum human takeover cost and comfort maximization as targets; the strategy optimization problem of an automatic driving automobile in a complex dynamic traffic environment is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a reinforcement learning autonomous driving method and system based on human-machine collaboration. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, automated vehicles (AVs) have attracted widespread attention as a disruptive technology, as they are expected to greatly improve the travel experience by improving road safety, optimizing traffic flow, and improving fuel economy. However, the realization of autonomous driving is full of challenges. The most critical thing is how to formulate appropriate driving strategies to optimize traffic efficiency while ensuring safety.

[0004] Autonomous vehicles usually rely on sensors to perceive the environment and generate online driving strategies after a lot of offline training. These strategies cover operational level control (such as acceleration, braking, turn signals), tactical level decision-making (such as lane changing, lane keeping), and strategic level route planning. However, due to the high dynamics and unpredictability of the road environment, especially the complex interaction between autonomous vehicles and human-driven vehicles (HVs), it is extremely challenging to develop a universally applicable and reliable autonomous driving strategy.

[0005] Most traditional autonomous driving systems use reinforcement learning (RL) methods. RL agents obtain feedback from the environment through trial-and-error exploration to optimize driving strategies, but this autonomous exploration inevitably exposes the agent to high-risk situations, limiting its application in safety-critical tasks. In addition, it is also extremely challenging to design a reward function that can cover all possible driving behaviors.

[0006] To overcome these problems, the "Human-in-the-Loop Learning" (HL) method has been proposed in recent years. This method improves the safety and sampling efficiency of the system by injecting human intelligence into RL or other learning paradigms. Under the HL framework, humans can directly participate in the training process through intervention, demonstration or evaluation, thereby helping the intelligent agent to learn appropriate strategies more quickly and safely. The HL method has achieved remarkable success in fields such as robot control and large-scale language model development. However, the application of HL in the field of autonomous driving still faces some challenges: existing research mostly relies on passive human feedback, such as suggesting actions or evaluating trajectories, which may not be sufficient to ensure safety in complex scenarios of autonomous driving. At the same time, frequent manual intervention will also increase the cognitive burden of human experts and affect training efficiency.

[0007] In addition, existing research often focuses only on the safety of individual autonomous vehicles, such as collision avoidance strategies, while ignoring the impact of these safety maneuvers on the overall traffic flow efficiency. However, research in the field of transportation has shown that the behavior of autonomous vehicles can significantly affect the overall traffic flow, including problems such as traffic flow interference, congestion, and emissions. Although some studies have begun to combine data-driven methods with rule-based traffic models, they mainly focus on the optimization of specific scenarios, such as intersection management and ramp merging, and fail to fully address the problem of strategy optimization for autonomous vehicles in complex and dynamic traffic environments. Summary of the invention

[0008] In order to address the deficiencies of the prior art, the present invention provides a reinforcement learning autonomous driving method, system, electronic device, computer-readable storage medium and computer program product based on human-machine collaboration, which are optimized with the goals of maximizing smoothness, maximizing action distribution entropy, minimizing explicit intervention and minimizing implicit intervention, ensuring the safety of a single autonomous driving vehicle under environmental uncertainty, while ensuring the smoothness and comfort of the vehicle during driving.

[0009] In a first aspect, the present invention provides a reinforcement learning autonomous driving method based on human-machine collaboration;

[0010] A reinforcement learning autonomous driving method based on human-machine collaboration, comprising:

[0011] Get the current state of the autonomous vehicle;

[0012] Input the current state of the autonomous driving vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current moment;

[0013] Among them, a reward-free strategy learning method is introduced, with the goal of minimizing the human takeover cost and maximizing the action distribution entropy and smoothness, and the autonomous driving network is trained through explicit intervention mechanisms and implicit intervention mechanisms.

[0014] In some embodiments, the non-reward strategy learning method is specifically as follows: updating the Q value at the current moment according to the discount factor and the maximum Q value corresponding to the strategy generated at the next moment; optimizing the driving strategy of the intelligent agent with the goal of maximizing the Q value.

[0015] In some implementations, the autonomous driving network is trained by an explicit intervention mechanism with the goal of minimizing the human takeover cost, specifically:

[0016] Based on a preset explicit intervention value function, optimization is performed with the constraint of minimizing the difference between the proxy Q-value of the autonomous vehicle's automatic actions and the proxy Q-value of human actions when human expert intervention is present.

[0017] In some implementations, the training of the autonomous driving network through an implicit intervention mechanism with the goal of maximizing the action distribution entropy and smoothness specifically includes:

[0018] Based on the preset implicit intervention value function, optimization is performed with the constraint of minimizing the difference between the proxy Q-value of the autonomous vehicle's automatic action and the proxy Q-value of the human action;

[0019] Based on the preset action distribution entropy function, optimization is performed with the constraint of minimizing the difference between the unrewarded proxy value and the entropy regularization term.

[0020] In some embodiments, training the autonomous driving network further includes: acquiring state information, image information, and exploration data of the autonomous driving vehicle and exploration data after expert intervention and storing them in an experience replay pool.

[0021] In some implementations, the autonomous driving network is an improved offline strategy-based actor-critic algorithm.

[0022] In a second aspect, the present invention provides a reinforcement learning autonomous driving system based on human-machine collaboration;

[0023] A reinforcement learning autonomous driving system based on human-machine collaboration, comprising:

[0024] The acquisition module is configured to: acquire the current state of the autonomous driving vehicle;

[0025] The decision module is configured to: input the current state of the autonomous driving vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current moment;

[0026] Among them, a reward-free strategy learning method is introduced, with the goals of maximizing smoothness and minimizing human takeover costs, and the autonomous driving network is trained through explicit intervention mechanisms and implicit intervention mechanisms.

[0027] In a third aspect, the present invention provides an electronic device;

[0028] An electronic device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned reinforcement learning autonomous driving method based on human-machine collaboration.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0030] A computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning autonomous driving method based on human-machine collaboration.

[0031] In a fifth aspect, the present invention provides a computer program product;

[0032] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning autonomous driving method based on human-machine collaboration.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The technical solution provided by the present invention designs a learning model inspired by the human learning process, which effectively injects human intelligence into the learning of artificial intelligence, allows human experts to act as mentors of artificial intelligence, conduct real-time supervision, intervention and training in the learning process, and improves the convergence speed of training AI through explicit and implicit intervention mechanisms.

[0035] 2. The technical solution provided by the present invention proposes a deep reinforcement learning framework, which not only improves the safety of autonomous driving vehicles in complex traffic scenarios, but also optimizes the vehicle's performance in terms of comfort, ensuring a smoother and safer driving experience in dynamic traffic environments by utilizing data collected from free exploration and partial human demonstrations as a training source.

[0036] 3. The technical solution provided by the present invention introduces a reward-free strategy learning method. During the training process, the autonomous driving vehicle clearly learns which state-action pairs are unsafe or inappropriate through intervention signals of human takeover behavior; this explicit intervention signal can not only convey human intentions during training, but also accelerate the convergence of the strategy.

[0037] 4. The technical solution provided by the present invention introduces an entropy maximization strategy to encourage the intelligent agent to explore in a safe state; in the absence of takeover signal intervention, the action distribution entropy of the intelligent agent is maximized to ensure full exploration within the allowed state-action space.

[0038] 5. The technical solution provided by the present invention reduces the interference to the traffic flow of subsequent human-driven vehicles by minimizing the interference cost caused by the current state and action of the intelligent agent; the interference cost is estimated through indicators such as vehicle state and vehicle stability, and is used as a constraint in training to reduce the negative impact on the surrounding traffic flow.

[0039] 6. The technical solution provided by the present invention designs a mechanism to minimize takeover costs; this mechanism aims to reduce the agent's dependence on human takeover signals, thereby gradually improving its autonomy, and ultimately enabling it to independently cope with various uncertain scenarios in complex traffic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0041] Figure 1 A schematic diagram of a process flow of a reinforcement learning autonomous driving method based on human-machine collaboration provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0045] Embodiment 1

[0046] The existing driving strategy optimization of autonomous driving vehicles cannot take into account both traffic flow efficiency and system-level performance while ensuring the safety of autonomous driving vehicles; therefore, the present invention provides a reinforcement learning autonomous driving method based on human-machine collaboration, which is optimized with the goal of minimizing human takeover costs and maximizing comfort.

[0047] Next, combine Figure 1 , a reinforcement learning autonomous driving method based on human-machine collaboration disclosed in this embodiment is described in detail. The reinforcement learning autonomous driving method based on human-machine collaboration includes the following steps:

[0048] S1. Obtain the state information and image information of the autonomous driving vehicle and store them in the experience replay pool.

[0049] At the same time, exploration data is stored in the experience replay pool. The experience replay pool is divided into an expert experience replay pool and an agent experience replay pool. The expert experience replay pool stores the exploration data of the autonomous driving vehicle after expert intervention, and the agent experience replay pool stores the exploration data of the autonomous driving vehicle.

[0050] S2. Input the state information and image information into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current moment.

[0051] In this embodiment, the autonomous driving network is an Off-Policy Actor-Critic algorithm based on an offline policy, which consists of an Actor module and a Critic module. The Actor module is used to generate a policy, that is, to determine the optimal action to take under a given state; the Critic module is responsible for evaluating the value of the action generated by the Actor module under the current state, thereby guiding the Actor module to improve its policy.

[0052] The Off-PolicyActor-Critic algorithm has the characteristics of Off-Policy, that is, the learning of the strategy can use the experience samples generated by different strategies. This learning mechanism can significantly improve the sample efficiency and make full use of historical data in the process of strategy improvement, so as to have higher robustness and stability in complex and changing driving scenarios.

[0053] A key feature of the offline-policy-based actor-critic algorithm is that it stores and utilizes offline data through the experience replay pool (ReplayBuffer), so that it can optimize the strategy without relying on the current strategy. Specifically, the samples in the experience replay pool include the state-action-reward-next-state quadruple (s t ,α t ,r t ,s t+1). During each training, a certain amount of random sampling data is extracted from the expert experience replay pool and the agent experience replay pool to update the objective functions of the Critic module and the Actor module, thereby breaking the time correlation, reducing the correlation between samples, and improving the convergence speed of the model. At the same time, it is used to solve the problem of gradually diluting expert experience as the amount of data increases.

[0054] In this embodiment, a non-reward policy learning method is introduced in the Off-PolicyActor-Critic algorithm. The goal of the driving policy learning task is to reflect human preferences in the behavior of the agent. The takeover operation of the human expert can be regarded as dissatisfaction with the current behavior of the agent (such as unsafe or poor performance), while no takeover means that the behavior is in line with human expectations. Based on this, in this embodiment, a proxy value function is used to represent human preferences instead of relying on the traditional reward function.

[0055] In the specific implementation, the Critic module learns the state-action value function Q π (s t ,a t ) to evaluate the strategy π generated by the Actor, which is defined as follows:

[0056]

[0057] Where E represents the expected value, r t is the reward value at the current time t, γ is the discount factor used to weigh current and future rewards, p(·|s t ,a t ) represents the state transition probability. The Actor module maximizes Q π (s t ,a t ) to update the strategy, and its objective function is defined as:

[0058]

[0059] in, and θ represent the neural network parameters of the Actor module and the Critic module respectively. In order to stabilize the policy update process, the Critic objective function is defined as minimizing the estimation error of the state-action value function Q:

[0060]

[0061] The Q value will be updated by Transformed into Avoid the limitations of manually designed reward functions.

[0062] Based on the learned proxy value function, the strategy π can be optimized by the following objectives θ:

[0063]

[0064] Converting standard RL tasks into reward-free settings enables the agent to learn from human participation and avoids complex reward design. The ideal driving policy should adapt to a variety of traffic scenarios, such as overtaking, avoidance, emergency stops, and interactions with other vehicles.

[0065] At the same time, the comprehensive goal of the autonomous driving network is to maximize the smoothness, maximize the action distribution entropy, minimize the explicit intervention, and minimize the implicit intervention, and use the existing human data to optimize the autonomous driving vehicle strategy in mixed traffic. The comprehensive goal is expressed as follows:

[0066]

[0067] In the formula, E represents the expected value, π represents the autonomous driving strategy, represents the stability proxy value function, H(π) represents the action distribution entropy function, Q EX (s,a AV ) represents the explicit intervention value function, Q IM (s,a AV ) represents the implicit intervention value function, s represents the state, a AV Represents an autonomous vehicle action.

[0068] This comprehensive goal takes into account both stability and expert burden and improves the overall performance of the fleet. To reflect human expectations of driving behavior and ensure smooth operation. In a state that meets human expectations, maximize the action distribution entropy H(π) to encourage the agent to actively explore the state-action space within the allowed range. Reduce human tutor intervention by minimizing the explicit intervention value function Q EX (s,a AV ) to reduce the frequency of mentor takeover. Reduce the impact of actions on the stability of the fleet and minimize the implicit intervention value function Q IM (s,a AV ) to optimize fleet stability.

[0069] As an implementation method, the process of training an autonomous driving network specifically includes:

[0070] Step 1: Obtain observation space data and action information.

[0071] Here, the observation space data includes numerical observation space and visual observation space. The numerical observation space includes the current state of the vehicle, navigation information and lidar surrounding environment information, and the visual observation space includes the image input of the camera. The policy network will generate corresponding actions based on the observation space data. These actions represent the control instructions taken by the autonomous driving vehicle in the current state, such as steering, acceleration or deceleration.

[0072] The action information includes the actions taken by the autonomous vehicle during exploration and the actions taken after the human expert takes over, which are stored in the vehicle experience replay pool and the expert experience replay pool respectively.

[0073] Step 2: Initialize the parameters and data of the autonomous driving network, set the Actor module network parameters θ and the Critic module network parameters And the experience replay pool B is used to store the state, action and intervention data collected during the training process.

[0074] Step 3: Based on the set comprehensive goal, an iterative training process is performed through the training set, and training is continued until the stopping condition is met (for example, the set number of training times or performance indicators are reached), where each iteration contains multiple training rounds (Episode).

[0075] State update and action selection in a single round: In each round, the state s t Next, the agent follows the strategy π AV Select and perform an action AV ; Action a AV After execution, the environment returns to the new state s t+1 , the agent updates its decision based on this state.

[0076] Human expert intervention judgment: Human experts in the current state s t Determine whether intervention is needed and record the intervention indicator I(s t ,a AV ); if I(s t ,a AV )=True, indicating that the human expert believes that the current action a of the agent AV Not suitable, need to take over; otherwise, continue to execute the agent's action a AV .

[0077] Execute human expert takeover action: if some pre-marks are true, the human expert performs action a human , and replace the agent's action a with this action AV To ensure that the vehicle can take reasonable actions in complex or dangerous situations; if some pre-marks are false, the agent's action a will continue to be executed. AV .

[0078] Calculate intervention cost: When the human expert does not intervene in the previous state but needs to intervene in the current state, calculate the explicit intervention cost C EX (s t ,a AV ), used to measure action a in the current state AV With human action human The deviation of the agent’s acceleration acc AV When the threshold λ is exceeded and it was not exceeded at the previous moment, the implicit intervention cost C is calculated. IM (s t ,a AV ), used to evaluate the action a in the current state under the vehicle stability model AV Impact on vehicle stability.

[0079] Data recording and experience storage: the current state s t 、Action a AV 、Human Action human , intervention indicator I(s t ,a AV ) and the next state s t+1 Stored in experience replay pool B to provide data support for subsequent training.

[0080] Strategy and value function update: According to the previously established agent value function Explicit intervention value function Q EX (s,a AV ), implicit intervention value function Q IM (s,a AV ) and the optimization objectives of the action distribution entropy H(π), and update their respective parameters.

[0081] For example, the specific learning process is as follows:

[0082] (1) Learning explicit intervention

[0083] We should strive to make the agent's behavior close to the demonstration of human experts, while avoiding the execution of intervention actions. The optimal strategy will always choose the action with the highest Q value. Therefore, if the human expert intervenes in certain states, the human action a in that state human ~π human The Q value of the agent action a should be higher than that of other actions. AV ~π AV The Q value of should be lower than other actions.

[0084] When the agent frequently relies on human experts to take over, it may lead to over-reliance, causing abnormal performance without supervision, resulting in a decrease in the level of automation. In order to reduce the cognitive load of the expert tutor, the takeover cost C is introduced. EX (s,aAV ) to measure the similarity between the agent's actions and human actions. When the agent's behavior deviates too much from human intentions, a higher takeover cost will be generated, thereby suppressing behaviors that do not meet human expectations. The takeover cost is expressed as:

[0085]

[0086] Among them, a AV and a human Represent the actions of the agent and the human in state s respectively. Takeover cost C EX (s,a AV ) value, the lower the similarity between the agent and human actions, and the greater the penalty.

[0087] Introducing the explicit intervention value function Q EX (s,a AV ), which is used to express the expected cumulative cost of a state-action pair leading to human takeover:

[0088]

[0089] By minimizing the takeover cost and explicit intervention value function, the agent can reduce unnecessary human intervention while improving the level of automation, thereby effectively reducing the cognitive load of human tutors and accelerating the strategy learning process.

[0090] Adopting a mixed behavior strategy π mix (a|s) to explore the environment, and some transition sequences collected during the takeover are regarded as partial demonstrations I(s t ,a AV ,s t+1 ,…), and stored in the experience replay pool B, which does not need to record the rewards and costs of the environment.

[0091] Since only part of the demonstration can be learned from B, this leads to the distribution shift problem. To this end, this embodiment adopts the CQL method to use human-generated offline data for training. Sample data s from the experience replay pool B t 、a AV 、a human , and mark the proxy Q value of human action as The proxy Q value of the agent's action is

[0092] Then the optimization problem of the proxy value function is expressed as:

[0093]

[0094] By minimizing the difference between the proxy Q-values ​​of human experts and the agent, the agent's actions can be guided to move closer to the high-value state-action subspace preferred by human experts.

[0095] (2) Learning implicit intervention

[0096] In implicit intervention learning, the goal is to optimize the behavior of the agent to make it as close as possible to the driving style of human experts, while avoiding actions that humans have intervened to ensure the stability of the vehicle. Specifically, define the implicit intervention value function Used to evaluate the impact of agent actions on vehicle stability.

[0097] The calculation formula of the implicit intervention value function is:

[0098]

[0099] The optimization goal is to make the agent's behavior consistent with the demonstration of human experts, while avoiding taking actions in states where experts have intervened. To this end, the objective optimization problem can be expressed as:

[0100]

[0101] During training, the state s is stored in the experience replay pool B. t 、Action a AV and the human expert's action a human , to be used to update the proxy value function.

[0102] The optimization problem can be expressed as:

[0103]

[0104] By minimizing the difference between the agent's actions and human expert behavior, the agent is encouraged to improve the stability of the vehicle while enhancing the safety and reliability of decision-making. This module aims to ensure that the agent always follows the human driving intention during the driving process and improve the smoothness of the driving experience.

[0105] (3) Learning to introduce implicit intervention of entropy regularization

[0106] In the reward-free framework, entropy regularization is introduced to encourage the agent to conduct extensive exploration in the state-action space by maximizing the entropy of the policy H(π(s)), thereby improving learning efficiency. Traditional methods rely on external reward signals, while entropy regularization guides policy updates without explicit rewards.

[0107] The entropy of a policy is defined as:

[0108] H(π(s))=-∑ a π(a|s)logπ(a|s);

[0109] In order to balance exploration and development, the entropy regularization term is added to the strategy update objective, and the optimization objective of the proxy value function is as follows:

[0110]

[0111] in, is the value function of the non-reward agent, and α is a hyperparameter that adjusts the entropy intensity. When the difference decreases, the strategy tends to be evenly distributed, avoiding falling into the local optimal solution, thereby improving the robustness and adaptability of the strategy.

[0112] Ultimately, through this optimization objective, the agent can learn a wide range of state-action distributions in a reward-free environment, effectively improving the exploration and behavior diversity of the strategy, and thus better adapting to complex environments.

[0113] Repeat the training until convergence, and repeat the above process until the preset training goal is achieved, such as model convergence, performance reaches the standard, or the number of training rounds is exhausted.

[0114] Next, in order to verify the effect of the reinforcement learning autonomous driving method based on human-machine collaboration described in this embodiment, a specific experimental verification was carried out.

[0115] In order to fully include dangerous and long-tail scenarios, MetaDrive is used to generate various driving scenarios, and each training set includes 50 different scenarios. Road scenarios consisting of various typical block types are included, such as straight roads, ramps, forks, roundabouts, curves, T-junctions, and intersections. The surrounding vehicles are controlled using two different models: the IDM model for longitudinal dynamics, and the minimum overall braking caused by the lane change model for lateral movement. In addition, traffic flows of different densities are introduced to interact with the AV. In order to increase the complexity of the driving task, each scene contains randomly placed obstacles, such as moving traffic vehicles, fixed traffic cones, and triangular warning signs.

[0116] The generated driving scenarios are divided into two sets: training set and test set, each with 50 different scenarios. The AV agent is trained exclusively on the training set and subsequently evaluated on the reserved test set. At the beginning of each iteration, a scene is randomly selected from the training set or the test set. After each training iteration, we deploy the learned agent to the test environment without any human supervision, record the safety violations, training deployments, and the success rate given by the environment, and present them in Table 1. The results show a high success rate and fewer safety violations, while significantly reducing the training time.

[0117] Table 1

[0118]

[0119] Embodiment 2

[0120] This embodiment discloses a reinforcement learning autonomous driving system based on human-machine collaboration, including:

[0121] The acquisition module is configured to: acquire the current state of the autonomous driving vehicle;

[0122] The decision module is configured to: input the current state of the autonomous driving vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current moment;

[0123] Among them, a reward-free strategy learning method is introduced, with the goals of maximizing smoothness and minimizing human takeover costs, and the autonomous driving network is trained through explicit intervention mechanisms and implicit intervention mechanisms.

[0124] It should be noted that the acquisition module and decision module described above correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0125] Embodiment 3

[0126] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned reinforcement learning autonomous driving method based on human-computer collaboration are completed.

[0127] Embodiment 4

[0128] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned reinforcement learning autonomous driving method based on human-computer collaboration are completed.

[0129] Embodiment 5

[0130] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning autonomous driving method based on human-machine collaboration.

[0131] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0134] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A reinforcement learning autonomous driving method based on human-machine collaboration, characterized in that: include: Get the current state of the autonomous vehicle; Input the current state of the autonomous driving vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current moment; Among them, a reward-free strategy learning method is introduced, with the goal of minimizing the human takeover cost and maximizing the action distribution entropy and smoothness, and the autonomous driving network is trained through explicit intervention mechanisms and implicit intervention mechanisms.

2. The method for autonomous driving based on human-machine collaboration through reinforcement learning according to claim 1, characterized in that: The non-reward strategy learning method is specifically as follows: updating the Q value at the current moment according to the discount factor and the maximum Q value corresponding to the strategy generated at the next moment; optimizing the driving strategy of the intelligent body with the goal of maximizing the Q value.

3. The method for autonomous driving based on human-machine collaboration through reinforcement learning according to claim 1, characterized in that: With the goal of minimizing the cost of human takeover, the autonomous driving network is trained through an explicit intervention mechanism as follows: Based on a preset explicit intervention value function, optimization is performed with the constraint of minimizing the difference between the proxy Q-value of the autonomous vehicle's automatic actions and the proxy Q-value of human actions when human expert intervention is present.

4. The method for autonomous driving based on human-machine collaboration through reinforcement learning as claimed in claim 1, characterized in that: With the goal of maximizing the action distribution entropy and smoothness, the autonomous driving network is trained through an implicit intervention mechanism, including: Based on the preset implicit intervention value function, optimization is performed with the constraint of minimizing the difference between the proxy Q-value of the autonomous vehicle's automatic action and the proxy Q-value of the human action; Based on the preset action distribution entropy function, optimization is performed with the constraint of minimizing the difference between the unrewarded proxy value and the entropy regularization term.

5. The method for autonomous driving based on human-machine collaboration through reinforcement learning as claimed in claim 1, characterized in that: Training the autonomous driving network also includes: obtaining state information, image information, and exploration data of the autonomous driving vehicle and exploration data after expert intervention and storing them in an experience replay pool.

6. The method for autonomous driving based on human-machine collaboration through reinforcement learning as claimed in claim 1, characterized in that: The autonomous driving network is an improved actor-critic algorithm based on offline strategy.

7. A reinforcement learning autonomous driving system based on human-machine collaboration, characterized in that: include: The acquisition module is configured to: acquire the current state of the autonomous driving vehicle; The decision module is configured to: input the current state of the autonomous driving vehicle into the trained autonomous driving network for processing to obtain the decision action of the autonomous driving vehicle at the current moment; Among them, a reward-free strategy learning method is introduced, with the goals of maximizing smoothness and minimizing human takeover costs, and the autonomous driving network is trained through explicit intervention mechanisms and implicit intervention mechanisms.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the reinforcement learning autonomous driving method based on human-machine collaboration as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the reinforcement learning autonomous driving method based on human-machine collaboration as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the reinforcement learning autonomous driving method based on human-machine collaboration as described in any one of claims 1 to 6 are implemented.

Images