An automatic driving decision-making method and system based on multi-stage human intervention feedback

Abstract

The application relates to the technical field of automatic driving, in particular to an automatic driving decision method and system based on multi-stage human intervention feedback, which comprises the following steps: constructing a reinforcement learning basic framework without a reward function; training an agent; in the process of automatic decision of the agent, when it is judged that the current driving behavior does not meet the expectation, executing manual takeover or rule expert intervention on the vehicle through an intervention switching mechanism; storing state-action sequences generated by autonomous exploration of the agent and demonstration sequences generated by human intervention in corresponding experience replay pools respectively, and constructing a split-trajectory experience replay structure; and updating a policy and a value function based on the split-trajectory experience replay structure. In the automatic driving algorithm training process, multi-stage human intervention behavior is introduced to express driving preferences, and a reward-free human-in-the-loop reinforcement learning mechanism is combined to realize learning and optimization of the vehicle decision process, and the application has the characteristics of safety and reliability, real controllability and human driving preference.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method and system based on multi-stage human intervention feedback. Background Technology

[0002] In autonomous driving decision-making, it is crucial to align algorithmic behavior with human driving preferences. However, due to the complexity and diversity of real-world driving intentions, existing methods often struggle to accurately express and learn human preferences using fixed reward functions or single demonstration data, resulting in insufficient policy stability and consistency. Therefore, a more effective technological approach is needed to capture and utilize human preference information to overcome this training challenge and enhance the safety, controllability, and human consistency of autonomous driving systems. Summary of the Invention

[0003] This application provides an autonomous driving decision-making method and system based on multi-stage human intervention feedback. This method can combine autonomous exploration with multi-stage human intervention feedback, thereby effectively avoiding the difficulties of artificial reward design, improving the alignment of decision-making strategies with human preferences, and significantly enhancing the controllability and reliability of vehicles in complex scenarios.

[0004] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide an autonomous driving decision-making method based on multi-stage human intervention feedback, comprising the following steps: First, constructing a reinforcement learning framework without a reward function; then, training the agent based on the reinforcement learning framework; during the agent's automatic decision-making process, when it is determined that the current driving behavior does not meet expectations, a manual takeover or rule expert intervention is performed on the vehicle through an intervention switching mechanism, and a state-action sequence and a demonstration sequence are generated; next, the state-action sequence generated by the agent's autonomous exploration and the demonstration sequence generated by human intervention are stored in corresponding experience replay pools respectively, constructing a trajectory-based experience replay structure; finally, the policy and value function are updated based on the trajectory-based experience replay structure.

[0005] In some exemplary embodiments, constructing a reinforcement learning framework without a reward function includes: acquiring vehicle sensor data, including the vehicle's dynamic state, navigation vectors, and LiDAR environmental perception information; using the vehicle sensor data as the observation input of the agent; and employing a two-dimensional continuous control quantity as the agent's action, enabling the agent to freely interact with the environment and explore rewardless policies during the training process.

[0006] In some exemplary embodiments, the intervention switching mechanism includes human expert intervention and IDM-MOBIL-based human-like mechanistic expert intervention. During training, when the agent exhibits trajectory deviation, abnormal speed control, or behavior inconsistent with human driving preferences at certain moments, the system allows the human driver to immediately take over via the steering wheel, accelerator, or brake pedal. IDM-MOBIL-based human-like mechanistic expert intervention is used to generate regularized, interpretable, and human-style automatic intervention actions. In simple traffic scenarios, the human-like mechanistic expert can maintain stable speed following, reasonable vehicle spacing control, and predictable lane-changing behavior, thereby providing high-quality demonstrations without human intervention.

[0007] In some exemplary embodiments, when the environmental state meets preset intervention conditions, the system automatically selects the corresponding expert action based on the intervention type and switches the action via an intervention switching function. The autonomous actions of the intelligent agent are replaced to generate hybrid behavioral outputs; a dual-source intervention mechanism, combining human expert intervention and human-like mechanistic expert intervention, enriches the training data and makes its coverage more comprehensive; among these, The action currently output by the agent. Actions demonstrated by human experts or experts on human-like mechanisms. For switching functions; when This indicates that expert intervention occurred, when This indicates that the intelligent agent is free to explore.

[0008] In some exemplary embodiments, the intervention switching mechanism enables manual takeover or rule expert intervention of the vehicle, including: adopting a course-learning approach, using different levels of intervention strategies according to different training stages, and providing human intervention feedback. Specifically, in the early stage of training, a human-assisted intervention method mainly based on human-like mechanism experts is used to handle low-risk, rule-based, and simple working conditions; in the middle stage of training, when the agent has basic controllable capabilities, timely intervention by human experts is added to handle driving tasks of moderate complexity; in the later stage of training, for high-risk or complex dynamic scenarios, such as emergency obstacle avoidance, oncoming traffic, or changes in traffic structure, a dual intervention method of "safety constraints + human experts" is adopted; the safety constraint module imposes restrictions on throttle and steering operations to ensure that the vehicle does not perform dangerous actions; at the same time, human experts provide the final decision, enabling the strategy to learn real human preferences and high-quality emergency behavior patterns within a safe range.

[0009] In some exemplary embodiments, constructing a trajectory-based experience replay structure includes: constructing a reinforcement learning replay pool and an intervention demonstration replay pool respectively; further including: during training, after each vehicle performs an action, recording the current experience sample according to a preset data format, wherein the data format is a quintuple; storing the quintuples into the reinforcement learning replay pool and the intervention demonstration replay pool respectively according to different intervention labels; the reinforcement learning replay pool is used to store exploration samples; the intervention demonstration replay pool is used to store demonstration samples generated by humans or humanoid mechanism experts; during the model training phase, random sampling is performed from the two replay pools according to a preset ratio to form a training batch containing demonstration data and exploration data; the demonstration data is used to provide credible supervision for the Critic's value alignment term and the Actor's preference alignment term, and the exploration data is used to maintain the algorithm's broad coverage and long-term stability of the environmental state space; by adjusting the sampling ratio of the two types of samples, a dynamic balance can be achieved between "convergence efficiency" and "humanoid behavior consistency".

[0010] In some exemplary embodiments, policy and value function updates are performed based on the trajectory-based experience replay structure, including: improving the original SAC (Soft Actor-Critic) algorithm by removing the environmental reward function to achieve efficient policy training using intervention demonstration data. Specifically, this includes: rewriting the original SAC's temporal difference (TD) objective into a proxy objective without a reward term to update the Critic network; the original SAC's temporal difference (TD) objective is:

[0011] The rewritten agent objective, excluding reward items, is as follows:

[0012] By omitting reward items, we can avoid training instability caused by inaccurate reward definitions, allowing policy learning to be driven entirely by demonstrated behaviors and value alignment items, thus avoiding conflicts with the logic of human intervention.

[0013] In some exemplary embodiments, to fully utilize the high-quality action samples in the intervention demonstration trajectory, an intervention value alignment term is added to the Critic network. Specifically, at the time of intervention, expert actions are assigned a fixed high value, and the following alignment loss is minimized:

[0014] Intervention value alignment terms construct a "value upper bound" for expert actions, forcing the Critic to learn the high-value meaning of expert behavior, enabling the agent to actively approach the expert's demonstration actions during the learning process and improve its ability to align with human preferences; During the Actor network update process, based on the maximum entropy policy optimization framework, the original SAC policy objective is extended to a joint optimization objective that incorporates a human preference alignment regularization term: The original SAC's strategic objective was:

[0015] The extended joint optimization objective is:

[0016] Among them, alignment items By measuring the difference between the agent's actions and those of the expert using cosine distance, we can suppress action choices that deviate from expert behavior and make the policy more similar to human driving style in the global distribution.

[0017] In some exemplary embodiments, in each training cycle, a dual-pool sampling mechanism is used, in which the system samples batch data proportionally from the exploration replay pool and the intervention replay pool, and uses them respectively to update the Critic's agent TD objective, expert value alignment term, and Actor's preference alignment objective. Through the dual-pool sampling mechanism, the training process can obtain convergence guidance from expert demonstrations while maintaining sufficient environmental exploration capabilities, thereby achieving a balance between learning efficiency and behavioral stability.

[0018] Secondly, this application also provides an autonomous driving decision-making system based on multi-stage human intervention feedback. This system is used to implement the autonomous driving decision-making method based on multi-stage human intervention feedback described in any of the above embodiments. The system includes: a basic framework construction module, a training module, a sub-trajectory experience playback structure construction module, and an update module connected sequentially. The basic framework construction module is used to construct a reinforcement learning basic framework without a reward function. The training module is used to train the agent based on the reinforcement learning basic framework. During the agent's automatic decision-making process, when it is determined that the current driving behavior does not meet expectations, an intervention switching mechanism is used to perform manual takeover or rule expert intervention on the vehicle, and to generate a state-action sequence and a demonstration sequence. The sub-trajectory experience playback structure construction module is used to store the state-action sequence generated by the agent's autonomous exploration and the demonstration sequence generated by human intervention into corresponding experience playback pools, respectively, to construct a scored trajectory experience playback structure. The update module updates the policy and value function based on the sub-trajectory experience playback structure.

[0019] The technical solution provided in this application has at least the following advantages: This application provides an autonomous driving decision-making method and system based on multi-stage human intervention feedback. The method includes the following steps: First, constructing a reinforcement learning framework without a reward function; then, training the agent based on the reinforcement learning framework; during the agent's automatic decision-making process, when the current driving behavior is determined to be inconsistent with expectations, a switch-intervention mechanism is used to manually take over the vehicle or allow rule experts to intervene, generating state-action sequences and demonstration sequences; next, the state-action sequences generated by the agent's autonomous exploration and the demonstration sequences generated by human intervention are stored in corresponding experience replay pools to construct a trajectory-based experience replay structure; finally, the policy and value function are updated based on the trajectory-based experience replay structure. The method provided in this application expresses driving preferences by introducing multi-stage human intervention behavior during the training process of the autonomous driving algorithm, and combines a human-in-the-loop reinforcement learning mechanism without reward to achieve learning and optimization of the vehicle's decision-making process. It has the characteristics of being safe, reliable, realistically controllable, and conforming to human driving preferences. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0021] Figure 1 A flowchart of an autonomous driving decision-making method based on multi-stage human intervention feedback provided in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the model architecture provided for an embodiment of this application.

[0023] Figure 3 This is a schematic diagram of an intervention method provided in an embodiment of this application. Detailed Implementation

[0024] As can be seen from the background technology, existing methods often struggle to accurately express and learn human preferences using fixed reward functions or single demonstration data, resulting in insufficient policy stability and consistency.

[0025] To address the aforementioned technical problems, this application provides an autonomous driving decision-making method and system based on multi-stage human intervention feedback. The method includes the following steps: First, constructing a reinforcement learning framework without a reward function; then, training the agent based on the reinforcement learning framework; during the agent's automatic decision-making process, when the current driving behavior is deemed undesirable, a switchover mechanism is used to manually take over the vehicle or allow rule experts to intervene, generating state-action sequences and demonstration sequences; next, the state-action sequences generated by the agent's autonomous exploration and the demonstration sequences generated by human intervention are stored in corresponding experience replay pools to construct a trajectory-based experience replay structure; finally, the strategy and value function are updated based on the trajectory-based experience replay structure. This application provides an autonomous driving decision-making method and system based on multi-stage human intervention feedback. This method combines autonomous exploration with multi-stage human intervention feedback, effectively avoiding the difficulties of designing manual rewards, improving the alignment of decision-making strategies with human preferences, and significantly enhancing the controllability and reliability of the vehicle in complex scenarios.

[0026] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0027] See Figure 1 This application provides an autonomous driving decision-making method based on multi-stage human intervention feedback, including the following steps: Step S1: Construct a basic framework for reinforcement learning without a reward function.

[0028] Step S2: Train the agent based on the reinforcement learning framework; during the agent's automatic decision-making process, when it is determined that the current driving behavior does not meet expectations, the vehicle is manually taken over or a rule expert intervenes through the intervention switching mechanism, and a state-action sequence and demonstration sequence are generated.

[0029] Step S3: Store the state-action sequences generated by the agent's autonomous exploration and the demonstration sequences generated by human intervention into the corresponding experience replay pools to construct a trajectory-based experience replay structure.

[0030] Step S4: Update the strategy and value function based on the trajectory experience replay structure.

[0031] This application realizes an autonomous driving decision-making method that combines autonomous exploration with multi-stage human intervention feedback through the above four steps, thereby effectively avoiding the difficulties of artificial reward design, improving the alignment of decision-making strategies with human preferences, and significantly enhancing the controllability and reliability of vehicles in complex scenarios.

[0032] In some embodiments, the reinforcement learning framework without a reward function in step S1 includes: acquiring vehicle sensor data, including vehicle dynamics state, navigation vector and lidar environmental perception information; using the vehicle sensor data as the observation input of the agent; this application adopts a continuous control action space, that is, using two-dimensional continuous control quantities as agent actions, so that the agent can freely interact with the environment and explore rewardless policies during the training process.

[0033] Specifically, step S1 constructs a basic framework for reinforcement learning without a reward function, which includes: First, the raw sensor data constituting the intelligent agent's observation space is acquired, including vehicle dynamics-related vehicle state information, route guidance vectors output by the navigation module, and two-dimensional point cloud distance sequences acquired by LiDAR. Among them, the vehicle state is read in real time by the on-board controller, including the vehicle's current speed, heading angle, steering angle, and lateral distance to the road boundary. The navigation vector is calculated by the local path planner to determine the relative position information of the vehicle to the next target checkpoint. The LiDAR data is processed by dividing the scanned point cloud within a 50-meter radius around the vehicle into directions and normalizing the distance to form a fixed-dimensional environmental perception vector.

[0034] Next, the three types of information are concatenated into a complete state vector in a preset order and input into the policy network and value network as the observation input for the agent at each time step. This state vector eliminates the need for manual extraction of intermediate semantic features, thereby reducing the model's reliance on manual priors and improving the system's generalization ability in different road scenarios.

[0035] Subsequently, to ensure the continuity and controllability of the action output, a two-dimensional continuous control quantity is used as the agent's action, including throttle control signal and steering angle control signal. The throttle signal has a value range of [-1, 1], with negative values ​​corresponding to braking and positive values ​​corresponding to acceleration; the steering signal is also defined within [-1, 1], with negative values ​​indicating right turns and positive values ​​indicating left turns, thus enabling the strategy to have fine-grained control capabilities in trajectory adjustment and speed management.

[0036] Then, during training, the mechanism of obtaining immediate rewards from the simulation environment is eliminated, and a policy optimization framework under no-reward conditions is constructed. Specifically, after performing an action, the environment only returns the observed state at the next moment, without providing any form of task reward or penalty signal. This design avoids the policy from exhibiting behaviors that conflict with human driving styles due to insufficient expressive power or improper reward shaping of traditional reward functions, and also avoids policy oscillations caused by inconsistencies between environmental rewards and expert demonstrations when human intervention occurs.

[0037] Finally, the agent autonomously samples trajectories in the environment using an exploration mode under no-reward conditions, acquires continuous actions through reparameterization techniques in the policy network, and estimates the value of actions using a value network. This process is not driven by external rewards, thus ensuring that the agent's behavioral adjustments are entirely determined by subsequent human intervention, expert demonstration data, and agent value function constraints, laying the foundation for achieving preference consistency between humans and the agent.

[0038] In some embodiments, step S2 introduces phased human intervention feedback. During the agent's automatic decision-making process, when a human determines that the current driving behavior does not meet expectations, manual takeover or rule expert intervention is performed on the vehicle, and corresponding expert actions are generated; under normal operating conditions, the agent continues to explore autonomously.

[0039] In step S2, an intervention switching mechanism is used to achieve dynamic selection between expert actions and policy actions. The intervention switching mechanism includes human expert intervention and human-like mechanism expert intervention based on IDM-MOBIL. During training, when the agent deviates from its trajectory, exhibits abnormal speed control, or behaves in a way that does not conform to human driving preferences at certain moments, the system allows the human driver to immediately take over through the steering wheel, accelerator, or brake pedal. Human expert intervention has the characteristics of flexible response and the ability to reflect subtle intentions in complex scenarios. It is particularly suitable for highly complex scenarios such as lane changing, obstacle avoidance, and passing other vehicles. Its actions can be regarded as high-value demonstration data.

[0040] Next, to reduce the long-term, high workload of human experts, this invention introduces a human-like mechanistic expert based on the IDM (Intelligent Driver Model) longitudinal control model and the MOBIL (Minimizing Overall Braking Induced by Lanechanges) lateral decision-making model. This expert generates rule-based, interpretable, and human-style automated intervention actions. In simple traffic scenarios, this human-like mechanistic expert can maintain stable speed following, reasonable vehicle spacing control, and predictable lane-changing behavior, thus providing high-quality demonstrations without human intervention.

[0041] In some embodiments, when the environmental conditions meet preset intervention conditions, the system automatically selects the appropriate expert action based on the intervention type and switches the action via an intervention switching function. The autonomous actions of the intelligent agent are replaced to generate hybrid behavioral outputs; a dual-source intervention mechanism, combining human expert intervention and human-like mechanistic expert intervention, enriches the training data and makes its coverage more comprehensive; among these, The action currently output by the agent. Actions demonstrated by human experts or experts on human-like mechanisms. For switching functions; when This indicates that expert intervention occurred, when This indicates that the intelligent agent is free to explore.

[0042] In some embodiments, the intervention switching mechanism performs manual takeover or rule expert intervention on the vehicle, including: adopting the concept of course learning, using different levels of intervention strategies according to different training stages, and providing human intervention feedback. Specifically, in the early stage of training, a human-assisted intervention method mainly based on human-like mechanism experts is used to handle low-risk, rule-based simple working conditions. This stage mainly deals with low-risk, rule-based simple working conditions, such as basic lane keeping and normal following, with continuous intervention provided by the IDM-MOBIL model to quickly establish a safe and stable driving strategy foundation.

[0043] Next, during the middle of training, once the agent has acquired basic controllability, timely intervention from human experts is introduced to handle moderately complex driving tasks, such as low-speed lane changes and navigating static obstacles. At this stage, human experts supplement the agent's learning of detailed driving behaviors through manual intervention, making the strategy more natural and smooth in its local action choices.

[0044] Then, in the later stages of training, for high-risk or complex dynamic scenarios, such as emergency obstacle avoidance, oncoming traffic, or changes in traffic structure, a dual intervention approach of "safety constraints + human experts" is adopted. The safety constraint module imposes restrictions on throttle and steering operations to ensure that the vehicle does not perform dangerous actions; at the same time, human experts provide the final decision, enabling the strategy to learn real human preferences and high-quality emergency behavior patterns within a safe range.

[0045] Through the above three-stage curriculum-based intervention arrangement, the system can gradually increase the complexity of the scenario while ensuring safety, so that the intelligent agent can obtain specific types of high-value demonstration samples at different training stages, thereby achieving more efficient and stable human-like driving strategy learning.

[0046] In some embodiments, the experience replay mechanism in step S3 includes constructing a reinforcement learning replay pool and an intervention demonstration replay pool, respectively, and storing them using a unified data format to achieve balanced utilization of exploratory and intervention data. Specifically, it includes the following: The reinforcement learning replay pool and the intervention demonstration replay pool are constructed separately. Further, during training, after each vehicle performs an action, the current experience sample is recorded according to a preset data format, where the data format is a quintuple. Based on different intervention labels, the quintuples are stored in the reinforcement learning replay pool and the intervention demonstration replay pool respectively. The reinforcement learning replay pool is used to store exploration samples; the intervention demonstration replay pool is used to store demonstration samples generated by human or humanoid mechanism experts. Through this dual-pool structure, high-value demonstration trajectories and large-scale exploration trajectories are managed separately, ensuring that the quality of demonstration data is not diluted while preserving the policy's coverage capability across multiple scenarios.

[0047] During the model training phase, random sampling is performed from two replay pools at a preset ratio to form training batches containing demonstration data and exploration data. Demonstration data is used to provide credible supervision for the Critic's value alignment and the Actor's preference alignment, while exploration data is used to maintain the algorithm's broad coverage of the environmental state space and long-term stability. By adjusting the sampling ratio of the two types of samples, a dynamic balance can be achieved between "convergence efficiency" and "human-like behavior consistency".

[0048] In some embodiments, in step S4, the policy and value function are updated based on the experience replay pool. Data is sampled proportionally from the two types of experience pools, the value network is updated using the no-reward TD objective, and a value alignment term is introduced based on expert intervention. At the same time, the policy update is constrained by the human preference alignment regularization term, so that the final autonomous driving decision model is consistent with the human driving style in terms of safety, stability and behavioral preferences.

[0049] Specifically, in step S4, based on the trajectory-based experience replay structure, the policy and value function are updated, including: Improving the original SAC (Soft Actor-Critic) algorithm by removing the environmental reward function to achieve efficient policy training using intervention demonstration data. First, in a reward-free environment, the system only obtains the next observation state after executing an action, without receiving any form of immediate reward value; to ensure the Critic network can still be updated, the original SAC's temporal difference (TD) objective is rewritten as a proxy objective without a reward term; the original SAC's temporal difference (TD) objective is:

[0050] The rewritten agent objective, excluding reward items, is as follows:

[0051] By omitting reward items, we can avoid training instability caused by inaccurate reward definitions, allowing policy learning to be driven entirely by demonstrated behaviors and value alignment items, thus avoiding conflicts with the logic of human intervention.

[0052] In some embodiments, to fully utilize the high-quality action samples in the intervention demonstration trajectory, an intervention value alignment term is added to the Critic network. That is, at the time of intervention, expert actions are assigned a fixed high value, and the following alignment loss is minimized:

[0053] Intervention value alignment is equivalent to constructing a "value upper bound" for expert actions, forcing the Critic to learn the high-value meaning of expert behavior, enabling the agent to actively approach the expert's demonstration actions during the learning process, and improving its ability to align with human preferences; Then, during the Actor network update process, based on the maximum entropy policy optimization framework, the original SAC policy objective is extended to a joint optimization objective that incorporates a human preference alignment regularization term: The original SAC's strategic objective was:

[0054] The extended joint optimization objective is:

[0055] Among them, alignment items By measuring the difference between the agent's actions and those of the expert using cosine distance, we can suppress action choices that deviate from expert behavior and make the policy more similar to human driving style in the global distribution.

[0056] Subsequently, in each training cycle, a dual-pool sampling mechanism is used, in which the system samples batch data proportionally from the exploration replay pool and the intervention replay pool, and uses them respectively to update the Critic's agent TD objective, expert value alignment term, and Actor's preference alignment objective. Through the dual-pool sampling mechanism, the training process can obtain convergence guidance from expert demonstrations while maintaining sufficient environmental exploration capabilities, thereby achieving a balance between learning efficiency and behavioral stability.

[0057] Finally, by employing a soft update method on the Critic network and gradually reducing the policy entropy coefficient, the policy exhibits less randomness and executes high-value actions more accurately in the later stages of training. This update process ensures stable convergence even without rewards, enabling the autonomous driving policy to simultaneously possess consistency with human preferences, safety, and executability in real-world road environments.

[0058] Furthermore, this application also provides an autonomous driving decision-making system based on multi-stage human intervention feedback. This system is used to implement the autonomous driving decision-making method based on multi-stage human intervention feedback described in any of the above embodiments. The system includes: a basic framework construction module, a training module, a trajectory-specific experience playback structure construction module, and an update module connected sequentially. The basic framework construction module is used to construct a reinforcement learning basic framework without a reward function. The training module is used to train the agent based on the reinforcement learning basic framework. During the agent's automatic decision-making process, when the current driving behavior is determined to be unsatisfactory, an intervention switching mechanism is used to manually take over the vehicle or allow rule experts to intervene, generating a state-action sequence and a demonstration sequence. The trajectory-specific experience playback structure construction module stores the state-action sequence generated by the agent's autonomous exploration and the demonstration sequence generated by human intervention into corresponding experience playback pools, respectively, to construct a scored trajectory experience playback structure. The update module updates the policy and value function based on the trajectory-specific experience playback structure.

[0059] Based on the above technical solutions, this application provides an autonomous driving decision-making method and system based on multi-stage human intervention feedback. The method includes the following steps: First, constructing a reinforcement learning framework without a reward function; then, training the agent based on the reinforcement learning framework; during the agent's automatic decision-making process, when the current driving behavior is determined to be undesirable, a manual takeover or rule expert intervention is performed on the vehicle through an intervention switching mechanism, generating state-action sequences and demonstration sequences; next, the state-action sequences generated by the agent's autonomous exploration and the demonstration sequences generated by human intervention are stored in corresponding experience replay pools to construct a trajectory-based experience replay structure; finally, the policy and value function are updated based on the trajectory-based experience replay structure. The method provided in this application expresses driving preferences by introducing multi-stage human intervention behavior during the training process of the autonomous driving algorithm, and combines it with a human-in-the-loop reinforcement learning mechanism without reward, thereby achieving the learning and optimization of the vehicle's decision-making process. It possesses the characteristics of being safe, reliable, realistically controllable, and conforming to human driving preferences.

[0060] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. An autonomous driving decision-making method based on multi-stage human intervention feedback, characterized in that, Includes the following steps: Construct a basic framework for reinforcement learning without a reward function; The agent is trained based on a reinforcement learning framework. During the agent's automatic decision-making process, when it is determined that the current driving behavior does not meet expectations, the vehicle is manually taken over or a rule expert intervenes through an intervention switching mechanism, and a state-action sequence and demonstration sequence are generated. The state-action sequences generated by the intelligent agent's autonomous exploration and the demonstration sequences generated by human intervention are stored in the corresponding experience replay pools to construct a trajectory-based experience replay structure. Based on the trajectory experience replay structure, the strategy and value function are updated.

2. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 1, characterized in that, The basic framework for reinforcement learning without a reward function is constructed, including: Acquire vehicle sensor data, including vehicle dynamics, navigation vectors, and LiDAR environmental perception information; Use vehicle sensor data as the observation input for the intelligent agent; Two-dimensional continuous control variables are used as agent actions, enabling the agent to freely interact with the environment and explore rewardless strategies during training.

3. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 1, characterized in that, The intervention switching mechanism includes human expert intervention and human-like mechanism expert intervention based on IDM-MOBIL; wherein, During training, if the agent deviates from its trajectory, exhibits abnormal speed control, or behaves in a way that does not conform to human driving preferences at certain moments, the system allows the human driver to immediately take over via the steering wheel, accelerator, or brake pedal. Human-like mechanistic expert intervention based on IDM-MOBIL is used to generate rule-based, interpretable, and human-style automatic intervention actions. In simple traffic scenarios, the human-like mechanistic expert can maintain stable speed following, reasonable vehicle spacing control, and predictable lane-changing behavior, thus providing high-quality demonstrations without human intervention.

4. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 3, characterized in that, When the environmental conditions meet the preset intervention criteria, the system automatically selects the appropriate expert action based on the intervention type and switches between them via the intervention switching function. Replace the autonomous actions of the intelligent agent to generate hybrid behavioral outputs; through a dual-source intervention mechanism of human expert intervention and human-like mechanism expert intervention, the training data is made richer and more comprehensive. in, The action currently output by the agent. Actions demonstrated by human experts or experts on human-like mechanisms. For switching functions; when This indicates that expert intervention occurred, when This indicates that the intelligent agent is free to explore.

5. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 1, characterized in that, Through intervention and switching mechanisms, manual takeover or intervention by rule experts can be implemented for vehicles, including: Employing a curriculum-based learning approach, different levels of intervention strategies are used based on the training phase to provide human intervention feedback, specifically including: In the early stages of training, a human-assisted intervention approach, primarily led by experts in human-like mechanisms, was adopted to handle simple work conditions with low risk and strong rules. During the middle of training, once the agent has acquired basic controllability, timely intervention from human experts is introduced to handle driving tasks of moderate complexity. In the later stages of training, for high-risk or complex dynamic scenarios, such as emergency obstacle avoidance, oncoming traffic, or changes in traffic structure, a dual intervention approach of "safety constraints + human experts" is adopted. The safety constraint module imposes restrictions on throttle and steering operations to ensure that the vehicle does not perform dangerous actions. At the same time, human experts provide the final decision, enabling the strategy to learn real human preferences and high-quality emergency behavior patterns within a safe range.

6. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 1, characterized in that, Constructing a trajectory-based experience playback structure includes: Construct reinforcement learning replay pools and intervention demonstration replay pools respectively, further including: During the training process, after each vehicle performs an action, the current experience sample is recorded according to a preset data format, which is a quintuple. Depending on the intervention label, the quintuples are stored in the reinforcement learning replay pool and the intervention demonstration replay pool, respectively; the reinforcement learning replay pool is used to store exploration samples; the intervention demonstration replay pool is used to store demonstration samples generated by human or humanoid mechanism experts. During the model training phase, random sampling is performed from two replay pools at a preset ratio to form training batches containing demonstration data and exploration data. Demonstration data is used to provide credible supervision for the Critic's value alignment and the Actor's preference alignment, while exploration data is used to maintain the algorithm's broad coverage of the environmental state space and long-term stability. By adjusting the sampling ratio of the two types of samples, a dynamic balance can be achieved between "convergence efficiency" and "human-like behavior consistency".

7. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 1, characterized in that, Based on the trajectory-based experience replay structure, policy and value function updates are performed, including: By removing the environmental reward function, the original SAC (Soft Actor-Critic) algorithm is improved to achieve efficient policy training using intervention demonstration data. Specifically, this includes: The original SAC's time difference (TD) objective is rewritten as a proxy objective without a reward term to update the Critic network; The original SAC time difference (TD) objective was: The rewritten agent objective, excluding reward items, is as follows: By omitting reward items, we can avoid training instability caused by inaccurate reward definitions, allowing policy learning to be driven entirely by demonstrated behaviors and value alignment items, thus avoiding conflicts with the logic of human intervention.

8. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 7, characterized in that, To fully utilize the high-quality action samples in the intervention demonstration trajectory, an intervention value alignment term is added to the Critic network. Specifically, during intervention, expert actions are assigned a fixed high value, and the following alignment loss is minimized: Intervention value alignment terms construct a "value upper bound" for expert actions, forcing the Critic to learn the high-value meaning of expert behavior, enabling the agent to actively approach the expert's demonstration actions during the learning process and improve its ability to align with human preferences; During the Actor network update process, based on the maximum entropy policy optimization framework, the original SAC policy objective is extended to a joint optimization objective that incorporates a human preference alignment regularization term: The original SAC's strategic objective was: The extended joint optimization objective is: Among them, alignment items By measuring the difference between the agent's actions and those of the expert using cosine distance, we can suppress action choices that deviate from expert behavior and make the policy more similar to human driving style in the global distribution.

9. The autonomous driving decision-making method based on multi-stage human intervention feedback according to claim 8, characterized in that, In each training cycle, a dual-pool sampling mechanism is used, in which the system samples batch data proportionally from the exploration replay pool and the intervention replay pool, and uses them respectively to update the Critic's agent TD target, expert value alignment item, and Actor's preference alignment target; Through the dual-pool sampling mechanism, the training process can obtain convergence guidance from expert demonstrations while maintaining sufficient environmental exploration capabilities, thereby achieving a balance between learning efficiency and behavioral stability.

10. An autonomous driving decision-making system based on multi-stage human intervention feedback, the system being used to implement the autonomous driving decision-making method based on multi-stage human intervention feedback as described in any one of claims 1 to 9, characterized in that, The system includes: a basic framework construction module, a training module, a trajectory-specific experience playback structure construction module, and an update module, connected sequentially; among them, The basic framework building module is used to build a basic framework for reinforcement learning without a reward function; The training module is used to train the agent based on the reinforcement learning framework. During the agent's automatic decision-making process, when it is determined that the current driving behavior does not meet expectations, the system will use an intervention switching mechanism to manually take over the vehicle or allow rule experts to intervene, and generate state-action sequences and demonstration sequences. The sub-trajectory experience replay structure construction module is used to store the state-action sequence generated by the intelligent agent's autonomous exploration and the demonstration sequence generated by human intervention into the corresponding experience replay pools to construct the score trajectory experience replay structure. The update module updates the strategy and value function based on the trajectory experience replay structure.

Images