Learning type automatic driving method based on deep fusion of physical model and human intelligence
By combining a hybrid enhancement guidance strategy that integrates physical models with human intelligence and a gradual attenuation intervention mechanism, the uncertainties and safety risks of human guidance in autonomous driving are resolved, achieving efficient and safe human-like autonomous driving decision-making.
Patent Information
- Application Number
- CN202511400523.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2025-12-26
AI Technical Summary
When dealing with the complexity, randomness, and uncertainty of dynamic traffic conditions, existing autonomous driving technologies are hampered by overly conservative mechanism-driven methods that lack human-like behavioral characteristics, while data-driven methods, with their trial-and-error nature, pose safety risks. Human-guided reinforcement learning suffers from uncertainty and skill fluctuations, resulting in low learning efficiency and insufficient safety.
By combining physical models with human intelligence, a progressive attenuation intervention mechanism is designed through a hybrid enhanced guidance strategy based on risk perception. This dynamically adjusts the weights of human guidance and physical models, integrates physical knowledge models, human driver guidance, and reinforcement learning agents, and optimizes the training process of the decision-making model.
It improves the data quality and learning efficiency of reinforcement learning, enhances the safety and human-likeness of autonomous driving, ensures stability and robustness in uncertain environments, and improves the reliability and autonomy of the learning process.
Smart Images

Figure CN121209265A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of automatic driving, in particular to a learning type automatic driving method deeply integrating physical models and human intelligence. BACKGROUND
[0002] Automatic driving has great potential in intelligent transportation systems, which can significantly improve driving safety, comfort and traffic efficiency, and has become a strategic development direction of the global automotive industry. The decision planning system is the "brain" of the autonomous vehicle, which uses external environmental information, combines driving tasks and control targets, and issues control commands to the execution layer, directly determining the intelligent level of autonomous driving. At present, the decision and motion planning methods of autonomous driving are mainly divided into two categories: mechanism-driven and data-driven. Mechanism-driven methods, such as artificial potential field method, are based on pre-defined models or rules, consistent with human cognitive logic, and have good interpretability. However, these methods are usually only applicable to specific driving scenarios, and face challenges in dealing with the complexity, randomness and uncertainty of dynamic traffic conditions. On the other hand, data-driven methods, especially reinforcement learning, have attracted widespread attention in the field of autonomous driving due to their advantages in handling high-dimensional and nonlinear problems.
[0003] Reinforcement learning enhances the driving ability of autonomous vehicles by exploring and interacting with the environment, and has achieved remarkable success in tasks such as lane changing and highway exit. However, in practical applications, the inherent trial-and-error characteristics of reinforcement learning introduce considerable safety risks. Therefore, safe reinforcement learning has become an important research field, and one of the promising solutions is human-guided reinforcement learning. Humans have rich common sense and reasoning ability, and can think and judge flexibly in various situations, making them an ideal learning object for autonomous driving systems. Human-guided reinforcement learning uses human drivers as instructors for reinforcement learning agents, and when reinforcement learning exploration leads to unsafe driving behavior, human instructors intervene to correct these behaviors, guiding the learning process towards safer and more human-like directions. Integrating human intelligence into the reinforcement learning training process can significantly improve the safety and human-likeness of autonomous driving behavior, and has been successfully applied to various driving scenarios.
[0004] Autonomous driving needs to balance safety, efficiency and human-likeness. Although some progress has been made in research, human-guided autonomous driving still has significant limitations: (1) traditional reinforcement learning has autonomous optimization capability, but its nature of trial and error exploration brings safety risks and reduces learning efficiency; (2) the model based on physical knowledge can guarantee safety, but it is often too conservative and lacks human-like behavior characteristics; (3) existing human-guided reinforcement learning usually assumes that human guidance is always perfect, but in reality, the skills and attention of participants may fluctuate, resulting in a decrease in driving data quality and even dangerous operations, and there is still a lack of effective methods to guarantee system performance under unreliable human guidance and fully utilize excellent human behavior to improve credibility. Therefore, the present application proposes a learning-based autonomous driving method deeply integrating physical models and human intelligence. SUMMARY
[0005] The present application aims to provide a learning-based autonomous driving method deeply integrating physical models and human intelligence, solving the challenges of non-perfect human demonstration uncertainty on reinforcement learning data efficiency and driving safety, improving the data quality, learning efficiency of reinforcement learning, and the driving safety and human-likeness of autonomous driving.
[0006] According to the first aspect of the present application, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a learning-based autonomous driving method deeply integrating physical models and human intelligence, comprising the following steps:
[0007] Constructing a risk perception-based autonomous driving decision model, wherein the autonomous driving decision model comprises a physical knowledge model for representing vehicle longitudinal and lateral safety constraints, a human driver guidance module and a reinforcement learning agent;
[0008] Designing a driving risk field of surrounding vehicles and roads, and calculating the physical knowledge model risk Human driver guidance risk And reinforcement learning agent control risk Wherein the physical knowledge model risk Human driver guidance risk Constitute a hybrid enhanced guidance driving risk
[0009] Based on the hybrid enhanced guidance driving risk Calculate the fusion weight λ t , according to the fusion weight λ t The physical model action generated by the physical knowledge model And the guidance action generated by the human driver guidance module Fusion, get hybrid enhanced guidance action
[0010] Risk of driving based on hybrid reinforcement guidance and reinforcement learning agent control Computing collaborative control weight τ t , according to collaborative control weight τ t mixing reinforcement guidance action and the guidance action output by the reinforcement learning agent fusion, the final autonomous driving decision model outputs the final execution action of the autonomous driving vehicle
[0011] Designing a progressive decay intervention mechanism to iteratively train the autonomous driving decision model, obtaining the trained autonomous driving decision model, and deploying the trained autonomous driving decision model to the car to realize autonomous driving decision.
[0012] Further, the physical knowledge model includes intelligent driver model IDM and MOBIL model, and the intelligent driver model IDM and MOBIL model are specifically constructed as follows:
[0013] (21) Intelligent driver model IDM simulates the interaction between the autonomous driving vehicle and the preceding vehicle, adjusts the acceleration to maintain a safe following distance, and the expression is as follows:
[0014]
[0015] wherein, and are the acceleration and speed of the autonomous driving vehicle, a is the maximum acceleration, is the desired speed, g0 is the minimum safety distance, H is the safety time headway, δ is the speed decay index, g and are the actual distance and relative speed between the autonomous driving vehicle and the preceding vehicle, γ and β are predefined parameters; (22) MOBIL model is used for lateral control, simulates lane changing behavior, and the expression is as follows:
[0016]
[0017] wherein, and represent the desired acceleration, and are the current acceleration of the autonomous driving vehicle, the new following vehicle and the old following vehicle; p c is the aggressive parameter, Δa th is the acceleration threshold for executing lane changing.
[0018] Further, a driving risk field of surrounding vehicles and roads is designed, and the physical knowledge model risk Human driver guidance risk and reinforcement learning agent control risk Specifically as follows:
[0019] (31) Driving risk field of surrounding vehicles The expression is:
[0020]
[0021] wherein, and respectively represent the field strength of the starting point and the vector of any point in the SVCS coordinate system of the i-th surrounding vehicle; v i , a ix and a iy are respectively the speed, longitudinal acceleration and lateral acceleration of the i-th surrounding vehicle; and are the angle between the X-axis and the Y-axis in the SVCS; η represents the lane line attenuation coefficient, when the end point of the i-th surrounding vehicle is located in the lane, η = 1; otherwise, η = 0.1;
[0022] (32) Driving risk field of road As follows:
[0023]
[0024] wherein, y AV , y f , y r , y c1 and y c2 respectively represent the lateral position of the autonomous vehicle, the position of the left and right road boundaries, and the left and right lane markings; and σ j are respectively the field strength, protection distance parameter and scaling factor related to each road feature j;
[0025] The final dynamic artificial potential field O AV is used to evaluate the driving risk of the autonomous vehicle, and the expression is as follows:
[0026]
[0027] wherein, and represent the coordinates of the autonomous vehicle in the i-th SVCS.
[0028] Further, the hybrid enhanced guidance driving risk The fusion weight λ t is calculated, and the autonomous vehicle is controlled according to the fusion weight λ tThe physical model action generated by the physical knowledge model The guidance action generated by the human driver guidance module The final execution action of the autonomous vehicle Specifically as follows:
[0029]
[0030] In the formula, κ1 represents a driving risk scaling factor;
[0031] The action of the hybrid enhanced guidance Is expressed as:
[0032]
[0033] Wherein, The action of the hybrid enhanced guidance, The action of the human driver model guidance, The action of the physical model, λ t Indicates the fusion weight.
[0034] Further, based on the hybrid enhanced guidance driving risk And the reinforcement learning agent control risk Calculate the cooperative control weight τ t According to the cooperative control weight τ t The action of the hybrid enhanced guidance And the guidance action output by the reinforcement learning agent Fusion, the final autonomous driving decision model outputs the final execution action of the autonomous vehicle Specifically as follows:
[0035]
[0036] Wherein, λ t Indicates the fusion weight, τ t Indicates the weight of the hybrid enhanced guidance and the reinforcement learning agent cooperative control, κ2 represents a driving risk scaling factor, Indicates the driving risk of the vehicle under the reinforcement learning control, Indicates the driving risk of the vehicle under the hybrid enhanced guidance.
[0037] The final execution action Can be obtained by the following formula:
[0038]
[0039] Wherein, The final execution action of the autonomous vehicle, The action of the hybrid enhanced guidance, an action representing an output of the reinforcement learning agent.
[0040] Further, the design of the progressive intervention mechanism iteratively trains the autonomous driving decision model, which includes three training stages:
[0041] Suppose that the reinforcement learning agent in the autonomous driving decision model achieves an optimal policy through C iterations of interaction with the environment, and let L represent the intervention set.
[0042] (61) The fully guided intervention stage uses the 0th to A th iterations:
[0043]
[0044] where Lc represents the fully guided intervention, S t represents the state quantity at time T, represents the action quantity of the hybrid enhanced guidance HEG at time t;
[0045] (62) The collaborative control intervention stage uses the A th to B th iterations:
[0046]
[0047] where Ls represents the collaborative guidance intervention, represents the state quantity at time T,
[0048] (63) The autonomous exploration stage uses the B th to C th iterations:
[0049]
[0050] where La is the autonomous exploration guidance, represents the empty set, and respectively represent the final time step of the intervention, and 0 < A < B < C.
[0051] Further, in the learning process of the autonomous driving decision model, an actor-critic reinforcement learning framework is used to handle continuous action space. This problem is modeled as a Markov decision process, in which at the current time step s t , the reinforcement learning agent performs an action a t , moves to the next state s t+1 , and obtains a reward r t ;
[0052] In the time span T, the cumulative discounted return R t is represented as
[0053] The reinforcement learning agent is in st Choose a t The probability is defined by policy π, and the execution of policy π starts from s. t The initial expected cumulative reward is expressed by the state value function V. π (s t ) indicates that from the state-action pair (s) t ,a t The expected cumulative reward of the execution strategy π is expressed using the action-value function (Q). π (s t ,a t )express:
[0054]
[0055] Where ρ represents the transition probability, and the optimal strategy π * Ensure that the agent achieves the maximum state value or action value in any state or state-action pair, thereby continuously optimizing performance and accumulating rewards:
[0056]
[0057] Where D is a playback buffer used to store interactive experiences;
[0058] Key information under collaborative control is stored in a replay buffer to provide experiential feedback to the reinforcement learning agent, thereby accelerating policy optimization.
[0059]
[0060] Here, G is the hybrid enhanced guidance intervention discriminator, and key information includes the current time step s. t Final execution action Reward r t Next state time step s t+1 Fusion weight λ t Hybrid reinforcement guidance and reinforcement learning agent collaborative control weights τ t Hybrid Enhanced Guided Actions Human driver model guidance actions Physics Model Actions
[0061] To avoid local optima, policy entropy is introduced into the objective function of the actor network, and the critic network is updated by minimizing the mean square Bellman error.
[0062]
[0063] Where H represents the entropy of the policy, φ and θ represent the parameters of the actor and critic networks, respectively, α is the temperature parameter, and i is the index of the dual critic network.
[0064] According to a second aspect of the present application, the present application provides a learning type automatic driving system deeply integrating mechanism knowledge and human intelligence, which is used to implement the learning type automatic driving method deeply integrating physical model and human intelligence described in the first aspect, and comprises:
[0065] a model construction model, which is used to construct an automatic driving decision model based on risk perception;
[0066] a risk calculation module, which is used to design a driving risk field of surrounding vehicles and roads, and calculate a physical knowledge model risk, a human driver guidance risk and a reinforcement learning intelligent agent control risk based on the vehicle and road driving risk field respectively, wherein the physical knowledge model risk and the human driver guidance risk constitute a hybrid enhanced guidance driving risk;
[0067] a hybrid enhanced guidance module, which is used to calculate a fusion weight based on the hybrid enhanced guidance driving risk, and fuse a physical model action generated by the physical knowledge model and a guidance action generated by the human driver guidance module according to the fusion weight to obtain a hybrid enhanced guidance action;
[0068] an execution output module, which is used to calculate a collaborative control weight based on the hybrid enhanced guidance driving risk and the reinforcement learning intelligent agent control risk, and fuse the hybrid enhanced guidance action and a guidance action output by the reinforcement learning intelligent agent according to the collaborative control weight to finally output a final execution action of the automatic driving vehicle by the automatic driving decision model;
[0069] an iterative training module, which is used to design a gradual decay intervention mechanism to iteratively train the automatic driving decision model to obtain a trained automatic driving decision model, and deploy the trained automatic driving decision model to a car to implement automatic driving decision.
[0070] According to a third aspect of the present application, the present application provides a terminal device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor loads and executes the computer program, the learning type automatic driving method deeply integrating physical model and human intelligence described in the first aspect is adopted.
[0071] According to a fourth aspect of the present application, the present application provides a storage medium containing computer executable instructions, which are used to execute the learning type automatic driving method deeply integrating physical model and human intelligence described in the first aspect when executed by a computer processor.
[0072] The present application at least has the following beneficial effects:
[0073] 1. The application effectively captures human driving behavior by developing a hybrid enhanced guidance (HEG) strategy, while explicitly considering the potential errors of human guidance and the differences between drivers. HEG combines physical models to ensure that even if there is uncertainty or risk in human input, the reinforcement learning agent can obtain high-quality and reliable guidance, and through a risk perception-based dynamic task delegation mechanism, the application adjusts the weight of human guidance and physical models in real time according to the scene risk, realizes the priority use of human guidance in low-risk scenarios, and relies more on physical models in high-risk scenarios, thereby reducing the data bias caused by human demonstration uncertainty, significantly improving the quality and reliability of learning samples, and ensuring the stability of the learning process.
[0074] 2. The application adopts a dynamic risk perception task delegation strategy to dynamically optimize the fusion ratio of human guidance and physical models according to the scene risk, ensures the stability of decision-making in uncertain environments and sudden situations, combines a progressive fading intervention mechanism (PFIM), and the agent obtains continuous and forward-looking risk perception guidance in the cooperative control stage, and gradually reduces the dependence on external intervention, transitions from imitation learning to autonomous optimization beyond human guidance, and realizes higher decision-making robustness and safety.
[0075] 3. By effectively combining physical constraints and HEG in the reinforcement learning process, the application significantly improves the learning efficiency, PFIM ensures high-quality initial guidance, while providing continuous, proactive and risk-aware task allocation in the cooperative control stage, so that the agent can efficiently learn even in high-risk or complex scenarios, and as the external intervention gradually weakens, the system adaptability is enhanced, and the agent autonomy and human-like driving behavior are fully expressed, laying a foundation for the intelligence and individualization of the automatic driving system.
[0076] 4. Through the progressive intervention strategy of PFIM, the agent in the application can internalize advanced skills while ensuring safety, and ultimately achieve performance beyond HEG guidance in specific driving tasks, which is different from traditional curriculum learning, imitation learning or safe reinforcement learning methods, making the system not only safe and reliable, but also having the ability to improve driving performance and human-machine collaboration efficiency.
[0077] Of course, any product implementing the application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS
[0078] Figure 1 Flowchart of the method described in the application;
[0079] Figure 2 Framework schematic diagram of the method described in the application;
[0080] Figure 3 Road and vehicle risk field schematic diagram for the method of the present application
[0081] Figure 4 Performance comparison graph of the PI-HGRL algorithm of the present application and the baseline algorithm. DETAILED DESCRIPTION
[0082] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present disclosure.
[0083] Embodiment one:
[0084] Please refer to Figures 1-4 The present application provides a technical solution: a learning type automatic driving method with deep fusion of physical models and human intelligence, which comprises:
[0085] S1. Construct a risk perception based automatic driving decision model, wherein the automatic driving decision model comprises a physical knowledge model for representing vehicle longitudinal and lateral safety constraints, a human driver guidance module and a reinforcement learning intelligent agent, specifically as follows:
[0086] S11. The intelligent driver model (IDM) simulates the interaction between the automatic driving vehicle and the preceding vehicle, and adjusts the acceleration to maintain a safe following distance, the expression is as follows:
[0087]
[0088] wherein, and are the acceleration and speed of the automatic driving vehicle, a is the maximum acceleration, is the desired speed, g0 is the minimum safety distance, H is the safety time headway, δ is the speed decay index, g
[0089] and are the actual distance and relative speed between the automatic driving vehicle and the preceding vehicle, γ and β are predefined parameters;
[0090] The MOBIL model is used for lateral control and simulates lane changing behavior, the expression is as follows:
[0091]
[0092] wherein, and represent the desired acceleration, and are the current accelerations of the ego vehicle, the new following vehicle and the old following vehicle respectively; p c is the aggressive parameter, Δa th is the acceleration threshold for performing lane change;
[0093] It should be noted that the new following vehicle represents the following vehicle after the ego vehicle changes from the current lane to the target lane; the old following vehicle represents the following vehicle in the current lane before the ego vehicle changes lane;
[0094] S12. The human driver guidance module is used to take over when the reinforcement learning agent exhibits abnormal or inappropriate behavior, serving as a safety bottom mechanism. The module receives driving environment information in real time through human drivers in the loop and observes the actions of the ego vehicle. When the behavior of the reinforcement learning agent deviates from the safe or reasonable range, the human driver immediately takes over the control of the vehicle. The driving behavior parameters output by the module include steering wheel operation, accelerator pedal depression depth and brake pedal operation, which are used to guide or correct the ego driving decision;
[0095] S13. The driving task requires precise, smooth and real-time control of the vehicle to cope with the complexity of the scene and ensure driving safety. To meet these needs, the actor-critic reinforcement learning framework is used to handle continuous action space in this embodiment. This problem can be modeled as a Markov decision process;
[0096] At the current time step s t , the agent performs action a t , moves to the next state s t+1 and obtains reward r t Over a time horizon T, the cumulative discounted return R t is represented as The probability of the agent choosing a t at s t is defined by the policy π. The expected cumulative reward starting from s t with policy π is represented by the state value function V π (s t ). The expected cumulative reward with policy π from state-action pair (s t , a t ) is represented by the action value function (Q π (s t , a t ):
[0097]
[0098] where ρ represents the transition probability, and the optimal policy π *Ensure that the agent realizes the maximum state value or action value in any state or state-action pair, thereby continuously optimizing performance and accumulating rewards:
[0099]
[0100] Where D is a replay buffer for storing interaction experience, and the key information under the collaborative regulation is stored in the replay buffer, so that the agent can use high-quality experience to accelerate policy optimization:
[0101]
[0102] Where G is a hybrid enhanced guidance intervention discriminator, then in order to enable the actor network to effectively imitate the hybrid enhanced guidance demonstration and maximize the action value function, so as to efficiently learn the data in the replay buffer, introducing the entropy of the policy in the objective function of the actor network can help avoid local optimum, in addition, the critic network is updated by minimizing the mean square Bellman error:
[0103]
[0104] Where H represents the entropy of the policy, φ and θ represent the parameters of the actor and critic networks respectively, α is the temperature parameter, and i is the index of the double critic network;
[0105] S2. Design the driving risk field of surrounding vehicles and roads, and calculate the physical knowledge model risk based on the vehicle and road driving risk field Human driver guidance risk And reinforcement learning agent control risk Where the physical knowledge model risk Human driver guidance risk Constitute a hybrid enhanced guidance driving risk Specifically as follows:
[0106] S21. In the process of obstacle avoidance and lane changing, the trajectory of the autonomous vehicle is affected by the surrounding vehicles, and the driving risk field of the surrounding vehicles is designed Its expression is:
[0107]
[0108] Where, And respectively represent the field strength of the starting point and the vector of any point in the coordinate system (SVCS) of the i-th surrounding vehicle; v i , a ix and a iy are the speed, longitudinal acceleration and lateral acceleration of the i-th surrounding vehicle, respectively and for is the angle between the X-axis and the Y-axis in the SVCS; η represents the lane line attenuation coefficient, when the end point of the lane line is located in the lane of the i-th surrounding vehicle, η = 1; otherwise, η = 0.1;
[0109] S22. In addition, the road boundary and lane marking limit the control space of the autonomous vehicle, and the proximity to the road edge significantly reduces the available space for obstacle avoidance, increasing the risk of collision, while the proximity to the lane marking can cause lane deviation and vehicle interference, posing a threat to driving safety. The artificial potential field designed for road constraints is as follows:
[0110]
[0111] where y AV , y f , y r , y c1 and y c2 represent the lateral position of the autonomous vehicle, the positions of the left and right road boundaries, and the left and right lane markings, respectively; and σ j are the field strength, protection distance parameter and scaling factor associated with each road feature j (i.e. road boundary or lane marking), respectively;
[0112] S23. The final dynamic artificial potential field O AV is used to evaluate the driving risk of the AV, and its expression is as follows:
[0113]
[0114] where, and represent the coordinates of the autonomous vehicle in the i-th SVCS, and the final vehicle and road driving risk field is shown in Figure 3 ; According to the above formula, the driving risks under the physical knowledge model human guidance and reinforcement learning control are calculated respectively;
[0115] According to the technical scheme of the embodiment, based on the 2-DOF vehicle model, the IDM and MOBIL models are combined to predict the trajectory of the autonomous vehicle under three conditions (physical knowledge model, human guidance and reinforcement learning control), and the corresponding driving risk is calculated. In order to reduce the overshoot phenomenon of trajectory prediction and improve the accuracy of safety evaluation, the prediction time domain is set to 0.5 seconds, and the model-based trajectory prediction method has the advantages of low computational burden and high accuracy in short-time prediction range;
[0116] S3. Driving risk based on hybrid enhanced guidance Calculate fusion weight λ t According to the fusion weight λ t The physical model action generated by the physical knowledge model is fused with the guidance action generated by the human driver guidance module to obtain a hybrid enhanced guidance action Specifically as follows:
[0117]
[0118] In the formula, κ1 represents a driving risk scaling factor;
[0119] The action of hybrid enhanced guidance is represented as:
[0120]
[0121] Wherein, The hybrid enhanced guidance action is represented as, The human driver model guidance action is represented as, The physical model action is represented as λ t The fusion weight is represented as;
[0122] S4. Driving risk based on hybrid enhanced guidance And reinforcement learning agent control risk Calculate collaborative control weight τ t According to the collaborative control weight τ t The hybrid enhanced guidance action is fused with the guidance action output by the reinforcement learning agent to obtain the final automatic driving decision model output of the final execution action of the automatic driving vehicle Specifically as follows:
[0123]
[0124] Wherein, λ t The fusion weight is represented as τ t The weight of hybrid enhanced guidance and reinforcement learning agent collaborative control is represented as κ2, which represents a driving risk scaling factor, The driving risk of the vehicle under reinforcement learning control is represented as, The driving risk of the vehicle under hybrid enhanced guidance is represented as;
[0125] The final execution action Can be obtained by the following formula:
[0126]
[0127] wherein, represents the final execution action of the autonomous vehicle, represents the action of mixed augmented guidance, represents the action output by the reinforcement learning agent;
[0128] S5. Design a progressive fading intervention mechanism to iteratively train the autonomous driving decision-making model, obtain the trained autonomous driving decision-making model, and deploy the trained autonomous driving decision-making model to the car to realize autonomous driving decision-making, as follows:
[0129] Based on the scaffolding theory in human pedagogy, a progressive fading intervention mechanism is designed;
[0130] Inspiration: To ensure the effectiveness of intervention in reinforcement learning and further improve safety and sampling efficiency, this embodiment proposes a new progressive fading intervention mechanism (PFIM), which is inspired by the scaffolding theory in human pedagogy. In the early stage of learning, the teacher provides reference for the students through demonstration, helping them to understand and imitate. As the learning progresses and the students' ability improves, the teacher gradually increases the students' autonomy and cooperates with the students to solve more complex problems. When the students' ability reaches a certain level, the teacher completely withdraws from the guidance and lets the students face the challenges independently. In this way, the students gradually transition from "imitation" to "learning" and finally "surpass" the teacher's ability;
[0131] Similarly, PFIM can be described as follows: in the initial learning stage of RL, when intervention is necessary, mixed augmented guidance completely takes over the vehicle, introduces RL into the autonomous driving framework, and provides high-quality samples for it to accelerate learning; then, in the stable learning stage, the RL agent has certain learning ability; while ensuring safety, the system switches to a mixed control mode, allowing mixed augmented guidance and the agent to jointly control the vehicle; this method not only reduces the agent's dependence on external intervention, but also creates more opportunities for autonomous exploration in real driving environments, thereby accelerating policy optimization and adaptability, and in the independent stage, the RL agent (reinforcement learning agent) has high decision-making ability and environmental adaptability, and the system completely hands over control to the RL agent, which further optimizes the driving strategy through autonomous exploration, thereby achieving surpassing of mixed augmented guidance.
[0132] The progressive fading intervention mechanism aims to guide the RL agent of the autonomous driving decision-making model to gradually transition from dependence on external intervention to completely autonomous decision-making, thereby completing the iterative training of the autonomous driving decision-making model;
[0133] Assuming that the RL agent achieves the optimal policy by interacting with the environment for C iterations, let L denote the intervention set, and PFIM can be divided into the following three intervention stages:
[0134] 1. Fully guided intervention: (0th to A th iteration);
[0135] wherein Lc represents the fully guided intervention, S t represents the state quantity at time T, represents the action quantity of the hybrid enhanced guidance HEG at time t;
[0136] 2. Collaborative control intervention: (A th to B th iteration);
[0137] wherein Ls represents the collaborative guidance intervention, represents the state quantity at time T;
[0138] 3. Autonomous exploration stage: (B th iteration and later);
[0139] wherein La is the autonomous exploration guidance; represents the empty set; and respectively represent the final time step of the intervention, and 0
[0140] S6. Based on the hybrid enhanced guidance and the progressive decay intervention mechanism, the reinforcement learning environment interaction experience is stored in the replay buffer to optimize the data quality and learning efficiency of the reinforcement learning.
[0141] Next, the technical solutions of the present application are further described in combination with specific embodiments:
[0142] Embodiment
[0143] Based on the Metadrive software and the driving simulator test, the mechanism knowledge enhanced automatic driving reinforcement learning method proposed in the present application is verified;
[0144] The performance of the PI-HGRL algorithm and the baseline algorithm proposed in this embodiment is evaluated by five different random seeds, and the results are as follows: Figure 4The SAC algorithm without human guidance shows the slowest learning speed. This is mainly because SAC relies entirely on the environment reward signal for exploration. SAC tends to perform inefficient random exploration, delaying the discovery of high-value policies and requiring additional training steps to balance exploration and exploitation. PI-HGRL outperforms baseline algorithms in both convergence speed and reward acquisition, demonstrating its advantages in learning efficiency and driving robustness. One possible reason is that SafeHIL-RL lacks a physical model as a safety backup. When the quality of human data decreases, its dynamic human-machine shared control mechanism can lead to poor coordination between "low-performance" agents and human drivers. Although PE-RLHF improves performance through a physical model-based safety backup mechanism, its high dependence on human intervention during the guidance phase and the use of a "0-1 switching" control strategy increase the uncertainty of human behavior. In contrast, PI-HGRL balances safety and humanization by combining a physical model and a risk perception-based task allocation strategy in HEG. In addition, PFIM gradually enhances the driving ability of the agent, ensuring that the agent maintains strong learning performance regardless of the quality of human guidance.
[0145] Learning efficiency is a key indicator for evaluating the ability of end-to-end autonomous driving systems to cope with complex and dynamic environments. To evaluate the learning efficiency of PI-HGRL compared to baseline methods, a comparative analysis is performed assuming that the baseline methods achieve their best reward R best The required training rounds are M, while PI-HGRL achieves R best The required rounds are N. Efficiency improvement is quantified by the following formula: K = M - N / M. The comparison results are shown in Table II. From this, it can be seen that the proposed algorithm outperforms the baseline methods in terms of data efficiency. Specifically, compared to SafeHIL-RL, PE-RLHF, and SAC, PI-HGRL reduces data requirements by 56.78%, 62.94%, and 71.59%, respectively. These results show that PI-HGRL has significant improvements in learning efficiency and resource utilization, especially in complex task scenarios.
[0146] Table 1 Learning efficiency comparison of the proposed algorithm and baseline algorithms
[0147] Contrast metric SAC SafeHIL-RL PE-RLHF M 630 752 761 N 179 325 282 K 71.59% 56.78% 62.94%
[0148] In summary, the present application fuses the physical model containing driving knowledge and human demonstration, dynamically optimizes the hybrid enhanced guidance strategy through the task delegation mechanism of driving risk perception, preferentially adopts human guidance in low-risk scenarios, and relies more on the physical model in high-risk situations to adapt to complex driving environments, improve data quality, and realize reliable and efficient human-machine collaboration.
[0149] Embodiment two:
[0150] The present embodiment provides a learning type autonomous driving system deeply integrating mechanism knowledge and human intelligence, which is used to realize the learning type autonomous driving method deeply integrating physical model and human intelligence described in embodiment one, comprising:
[0151] The model construction model is used to construct an autonomous driving decision model based on risk perception.
[0152] The risk calculation module is used to design a driving risk field of surrounding vehicles and roads, and calculate physical knowledge model risk, human driver guidance risk and reinforcement learning intelligent agent control risk based on the vehicle and road driving risk field, wherein the physical knowledge model risk and the human driver guidance risk constitute a hybrid enhanced guidance driving risk.
[0153] The hybrid enhanced guidance module is used to calculate a fusion weight based on the hybrid enhanced guidance driving risk, fuse the physical model action generated by the physical knowledge model and the guidance action generated by the human driver guidance module according to the fusion weight, and obtain a hybrid enhanced guidance action.
[0154] The execution output module is used to calculate a cooperative control weight based on the hybrid enhanced guidance driving risk and the reinforcement learning intelligent agent control risk, fuse the hybrid enhanced guidance action and the guidance action output by the reinforcement learning intelligent agent according to the cooperative control weight, and finally output the final execution action of the autonomous driving vehicle by the autonomous driving decision model.
[0155] The iterative training module is used to design a progressive attenuation intervention mechanism to iteratively train the autonomous driving decision model, obtain the trained autonomous driving decision model, and deploy the trained autonomous driving decision model to the car to realize autonomous driving decision.
[0156] Embodiment three:
[0157] The application provides a terminal device, including a memory, a processor and a computer program stored in the memory and capable of running on the processor, the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the learning type automatic driving method with deep fusion of physical model and human intelligence described in embodiment one is adopted.
[0158] It should be noted that the terminal device can adopt a computer device such as a desktop computer, a notebook computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory, for example, the terminal device can also include an input / output device, a network access device and a bus, etc.
[0159] Further, the processor can adopt a central processing unit (CPU), of course, according to the actual use condition, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), ready-to-program gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. can also be adopted, the general-purpose processor can adopt a microprocessor or any conventional processor, etc., and the application does not limit this.
[0160] Embodiment four:
[0161] The application provides a storage medium containing computer executable instructions, which are used for executing the learning type automatic driving method with deep fusion of physical model and human intelligence described in embodiment one when executed by a computer processor.
[0162] The computer program can be stored in a computer readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or some middleware form, etc., the computer readable medium includes any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc. capable of carrying the computer program code, and it should be noted that the computer readable medium includes but is not limited to the above components.
[0163] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended that the scope of the application be limited to the detailed description contained herein or the specific examples given herein, but rather that the scope of the application be determined by the appended claims, and their equivalents.
[0164] Those of ordinary skill in the art will appreciate that the above described terms are to be construed in accordance with their ordinary meanings in the present invention. When an element is referred to as being "connected", "coupled", "fixed", or "attached" to another element, it can be directly connected, coupled, fixed, or attached to the other element, or intervening elements can be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element, or intervening elements can be present. The terms "vertical", "horizontal", "upper", "lower", "left", "right", and similar terms as used herein are for descriptive purposes only and not meant to be limiting.
[0165] While embodiments of the application have been shown and described, it is to be understood that various modifications, substitutions, combinations, and variations can be made therein and by those of ordinary skill in the art without departing from the spirit and scope of the present application, which is defined by the following claims and their equivalents.
[0166] In the description of the specification, reference to "one embodiment", "an example", "a specific example", or the like means that a particular feature, structure, material, or characteristic being referred to is included in at least one embodiment or example of the disclosure. The appearances of the above expressions in various places in the specification are not necessarily referring to the same embodiment or example. Moreover, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
Claims
1. A learning automatic driving method in which a physical model is deeply integrated with human intelligence, characterized by, The method comprises the following steps: constructing a risk perception-based automatic driving decision model, wherein the automatic driving decision model comprises a physical knowledge model for representing vehicle longitudinal and lateral safety constraints, a human driver guidance module, and a reinforcement learning agent; designing a driving risk field for surrounding vehicles and roads, calculating a physical knowledge model risk based on the vehicle and road driving risk fields, respectively human driver guidance risk and reinforcement learning agent control risk wherein the physical knowledge model risk human driver guidance risk constitutes a hybrid augmented guided driving risk Hybrid augmented guidance driving risk based on Computing fusion weight λ t , according to fusion weight λ t Fusing physical model action generated by physical knowledge model With guidance action generated by human driver guidance module To get hybrid augmented guidance action Risk of guiding driving based on hybrid reinforcement and reinforcement learning agent control Computing a collaborative control weight τ t , according to the collaborative control weight τ t Fusing the hybrid reinforcement guided action and the reinforcement learning agent output guided action to output a final execution action of an autonomous vehicle by a final autonomous driving decision model designing a progressive attenuation intervention mechanism to iteratively train the automatic driving decision model, obtaining a trained automatic driving decision model, and deploying the trained automatic driving decision model to a car to realize automatic driving decision.
2. The physical model and human intelligence deeply integrated learning automatic driving method according to claim 1, characterized in that: The physical knowledge model comprises an intelligent driver model IDM and a MOBIL model, which are specifically constructed as follows: (21) The intelligent driver model IDM simulates the interaction between the automatic driving vehicle and the preceding vehicle, and adjusts the acceleration to maintain a safe following distance, and the expression is as follows: where, and are the acceleration and speed of the ego vehicle, a is the maximum acceleration, is the desired speed, g0 is the minimum safety gap, H is the safety time headway, δ is the speed decay exponent, g and are the actual gap and relative speed between the ego vehicle and the preceding vehicle, γ and β are predefined parameters; (22) the MOBIL model is used for lateral control, simulating the lane-changing behavior, and the expression is as follows: wherein, and respectively represent the desired acceleration, and are the current accelerations of the ego vehicle, the new following vehicle and the old following vehicle, respectively; p c is an aggressive parameter, and Δa th is an acceleration threshold for performing a lane change. 3.The physical model and human intelligence deeply integrated learning automatic driving method according to claim 2, characterized in that: designing a driving risk field for surrounding vehicles and roads, calculating a physical knowledge model risk based on the vehicle and road driving risk fields respectively human driver guidance risk and reinforcement learning agent control risk In particular as follows: (31) Driving risk field of surrounding vehicles The expression is: wherein, and respectively represent the field strength of the starting point and the vector of any point in the SVCS coordinate system of the i-th surrounding vehicle; v i , a ix and a iy are respectively the speed, longitudinal acceleration and lateral acceleration of the i-th surrounding vehicle; and are the angle between the X-axis and the Y-axis in the SVCS; η represents the lane line attenuation coefficient, when the end point of is located in the lane of the i-th surrounding vehicle, η = 1; otherwise, η = 0.1; (32) Driving risk field of a road As follows: where y AV , y f , y r , y c1 , and y c2 represent the lateral position of the ego vehicle, the positions of the left and right road boundaries, and the left and right lane marks, respectively; and σ j are the field strength, the protection distance parameter, and the scaling factor associated with each road feature j, respectively; Final dynamic artificial potential field O AV For evaluating the driving risk of the autonomous vehicle, the expression is as follows: wherein, and represents the coordinates of the autonomous vehicle in the i-th SVCS.
4. The physical model and human intelligence deeply integrated learning automatic driving method according to claim 3, characterized in that: The risk of hybrid-enhanced guided driving Calculate the fusion weight λ t According to the fusion weight λ t Actions generated from physical knowledge models Guided actions generated by the human driver guidance module By merging these elements, we obtain a hybrid enhanced guided action. Specifically as follows: In the formula, κ1 represents a driving risk scaling factor; Hybrid augmented guidance action is represented as: wherein, represents a mixed augmented guidance action, represents a human driver model guidance action, represents a physical model action, λ t represents a fusion weight. 5.The physical model and human intelligence deeply integrated learning automatic driving method according to claim 4, characterized in that: Risk of guiding driving based on hybrid reinforcement and reinforcement learning agent control Computing a collaborative control weight τ t , according to the collaborative control weight τ t Merging the hybrid reinforcement guided action and the reinforcement learning agent output guided action , the final autonomous driving decision model outputs the final execution action of the autonomous vehicle Specifically as follows: wherein λ t denotes the fusion weight, τ t denotes the weight of the collaborative control of the hybrid enhanced guidance and the reinforcement learning agent, κ2 denotes the driving risk scaling factor, denotes the driving risk of the vehicle under the reinforcement learning control, denotes the driving risk of the vehicle under the hybrid enhanced guidance; Final execution action This can be obtained by the following equation: wherein, represents a final execution action of an autonomous vehicle, represents a hybrid augmented guided action, represents an action output by a reinforcement learning agent. 6.The physical model and human intelligence deeply integrated learning automatic driving method according to claim 5, characterized in that: The design of the progressive attenuation intervention mechanism for iteratively training the automatic driving decision model specifically comprises three training stages: Assuming that the reinforcement learning agent in the automatic driving decision model realizes an optimal strategy through C iterations of interaction with the environment, and L represents a set of interventions; (61) In the complete guidance intervention stage, the first 0th to A th iterations are adopted: In the formula, Lc represents a complete guidance intervention, S t represents a state quantity at time T, represents an action quantity of a hybrid enhanced guidance HEG at time t; (62) In the collaborative control intervention stage, the A th to B th iterations are adopted: In the formula, Ls represents a synergistic guidance intervention, represents state quantity at the time instant; (63) In the autonomous exploration stage, the B th iteration to the C th iteration: where La is the autonomous exploration guide; denotes the empty set; and denotes the final time step of the intervention, and 0 < A < B < C. 7.The physical model and human intelligence deeply integrated learning automatic driving method according to claim 6, characterized in that: In the automatic driving decision model learning process, an actor-critic reinforcement learning framework is used to process a continuous action space, and the problem is modeled as a Markov decision process, at the current time step s t , the reinforcement learning agent performs an action a t , moves to the next state s t+1 , and obtains a reward r t ; The cumulative discounted return R over a time span T, considering a discount factor ξ ∈ (0, 1] is given by t is denoted by The reinforcement learning agent in s t The probability of choosing a t is defined by a policy π, and the expected cumulative reward of executing policy π starting from s t is denoted by a state value function V π (s t ). The expected cumulative reward of executing policy π from a state-action pair (s t , a t ) is denoted by an action value function (Q π (s t , a t ). where p represents the transition probability, and the optimal policy * Ensuring that the agent achieves the maximum state value or action value in any state or state-action pair, thereby continuously optimizing performance and accumulating rewards: Wherein, D is a replay buffer for storing interaction experience; The key information under collaborative control is stored in the replay buffer, and experience feedback is given to the reinforcement learning agent to accelerate strategy optimization: wherein G is a hybrid augmented guidance intervention discriminator, the key information includes a current time step s t , a final execution action a reward r t , a next state time step s t+1 , a fusion weight λ t , a hybrid augmented guidance and reinforcement learning intelligent agent collaborative control weight τ t , a hybrid augmented guidance action a human driver model guidance action a physical model action The entropy of the strategy is introduced into the objective function of the actor network to avoid local optimum, and the critic network is updated by minimizing the mean square Bellman error: Wherein, H represents the entropy of the strategy, φ and θ represent the parameters of the actor and critic networks respectively, α is a temperature parameter, and i is the index of the double critic network.
8. A learning automatic driving system deeply integrating mechanism knowledge with human intelligence, used to implement the learning automatic driving method deeply integrating physical model with human intelligence according to any one of claims 1 to 7, characterized in that, The method comprises the following steps: model construction, for constructing a risk perception-based automatic driving decision model; a risk calculation module, for designing a driving risk field of surrounding vehicles and roads, and calculating a physical knowledge model risk, a human driver guidance risk, and a reinforcement learning agent control risk based on the vehicle and road driving risk fields, wherein the physical knowledge model risk and the human driver guidance risk constitute a hybrid enhanced guidance driving risk; a hybrid enhanced guidance module, for calculating a fusion weight based on the hybrid enhanced guidance driving risk, and fusing a physical model action generated by the physical knowledge model and a guidance action generated by the human driver guidance module according to the fusion weight to obtain a hybrid enhanced guidance action; an execution output module, for calculating a collaborative control weight based on the hybrid enhanced guidance driving risk and the reinforcement learning agent control risk, and fusing the hybrid enhanced guidance action and a guidance action output by the reinforcement learning agent according to the collaborative control weight, so as to finally output an execution action of the automatic driving vehicle by the automatic driving decision model. An iterative training module is configured to iteratively train the automatic driving decision model by using the progressive attenuation intervention mechanism, and obtain a trained automatic driving decision model.
9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor loads and executes the computer program, and the physical model and human intelligence are deeply fused in the learning type automatic driving method of any one of claims 1 to 7.
10. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by the computer processor, are configured to perform the physical model and human intelligence deeply fused learning type automatic driving method of any one of claims 1 to 7.
