Automatic driving reinforcement learning decision method, device, equipment, storage medium and product

Abstract

This application relates to a method, apparatus, computer device, computer-readable storage medium, and computer program product for autonomous driving reinforcement learning decision-making. The method includes: determining the total risk based on a state vector; inputting the state vector into an action prediction model trained based on deep reinforcement learning to obtain candidate actions corresponding to the state vector; the state vector is constructed based on collected environmental information, and the total risk is obtained by superimposing the potential fields of multiple risk sources; inputting a state-action pair into an uncertainty decision-making model trained based on deep reinforcement learning to obtain an uncertainty metric; the state-action pair consists of a state vector and its corresponding candidate actions; calculating a fusion index based on the total risk and the uncertainty metric; determining the target region where the state-action pair is located based on the fusion index; and determining the final action used to control the vehicle's driving state based on the target region and the candidate actions. This method can provide a single, easily interpretable representation for decision-making safety assessment.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology for intelligent vehicles, and in particular to an autonomous driving reinforcement learning decision-making method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In autonomous driving decision-making systems, environmental risk perception and model uncertainty assessment are typically considered two independent technical paths. Existing methods often focus on a single dimension: either characterizing the objective degree of danger in a scene through risk potential fields or attention mechanisms, or estimating the model's confidence level in unknown states using ensemble learning or Bayesian inference. Although some research has attempted to simply combine the two, such as adding risk penalties and uncertainty penalties to the reward function simultaneously, or alternating between risk thresholds and uncertainty thresholds during decision-making, these approaches essentially remain at the later stage of splicing separate indicators and fail to establish an inherently unified evaluation mechanism.

[0003] This discrete architecture makes it difficult for the system to coordinate the inherent coupling relationship between the two types of information in time-series decision-making. Specifically, existing methods generally lack a unified quantitative framework that can simultaneously characterize the objective environmental risk and the subjective uncertainty of the model in the same indicator system. The model cannot distinguish between scenarios with high objective risk but model certainty and scenarios with low objective risk but model uncertainty, thus affecting the effective use of samples and the balanced optimization of strategies. Summary of the Invention

[0004] Based on this, it is necessary to provide an autonomous driving reinforcement learning decision-making method, device, computer equipment, computer-readable storage medium, and computer program product that can map objective environmental risks and subjective model uncertainties to the same quantitative scale to address the aforementioned technical problems.

[0005] Firstly, this application provides a reinforcement learning decision-making method for autonomous driving, including:

[0006] The total risk is determined based on the state vector. The state vector is input into the action prediction model trained by deep reinforcement learning to obtain the candidate action corresponding to the state vector. The state vector is constructed based on the collected environmental information and is used to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0007] The uncertainty metric is obtained by inputting state-action pairs into an uncertainty decision-making model trained based on deep reinforcement learning. A state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output of the action prediction model.

[0008] The fusion index is calculated based on the total risk and uncertainty measures. The fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk.

[0009] The target area where the state action pair is located is determined based on the fusion index, and the final action is determined based on the target area and candidate actions. The final action is used to control the vehicle driving state.

[0010] In one embodiment, determining the total risk based on the state vector includes:

[0011] By inputting the state vector into the Gaussian risk potential function of each risk source, the potential field of each risk source is obtained. The Gaussian risk potential function of each risk source is constructed based on environmental information.

[0012] The total risk is obtained by superimposing the potential fields of each risk source.

[0013] In one embodiment, the uncertainty decision-making model is constructed by a parallel evaluation network. The uncertainty decision-making model, trained using deep reinforcement learning, is input to state-action pairs to obtain uncertainty metrics, including:

[0014] The state-action pair is input into the evaluation network, and the uncertainty assessment results of each evaluation network are obtained. The uncertainty metric is calculated based on each uncertainty assessment result.

[0015] In one embodiment, determining the target region where the state-action pair is located based on fusion metrics includes:

[0016] When the fusion index of the state-action pair is less than or equal to the first fusion index threshold, the target area of ​​the state-action pair is determined as a safe zone.

[0017] When the fusion index of the state-action pair is greater than the first fusion index threshold and less than or equal to the second fusion index threshold, the target area of ​​the state-action pair is determined to be a safe area.

[0018] When the fusion index of the state-action pair is greater than the second fusion index threshold, the target region of the state-action pair is determined to be a safe region.

[0019] In one embodiment, determining the final action based on the target region and candidate actions includes:

[0020] The set of safe actions is determined based on the state vector. The set of safe actions includes at least one safe action, and the safe action is located within the safe distance and safe boundary determined based on the state vector.

[0021] If the target area is a safe area, and the set of safe actions contains candidate actions, then the candidate actions are determined as the final actions; or if the set of safe actions does not contain candidate actions, then the candidate actions are projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action.

[0022] When the target area is a caution zone, reference actions are selected from the set of safety actions, and the candidate actions and reference actions are weighted and fused. The weighted fusion result is then output as the final action.

[0023] If the target area is located in a danger zone, the candidate actions are projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action.

[0024] In one embodiment, it further includes:

[0025] Based on the final action and state vector, determine the state-action pair at the current sampling time, and calculate the fusion index of the state-action pair at the current sampling time based on the state-action pair at the current sampling time.

[0026] After controlling the vehicle's driving state according to the final action, the state vector at the next sampling time is obtained. Based on the state vector at the next sampling time, the immediate reward of the state-action pair at the current sampling time is determined. The immediate reward is used to construct the task reward function, so as to iteratively update the vehicle's control strategy based on the task reward function.

[0027] Based on the final action, state vector, fusion index of state-action pair at the current sampling time, state vector at the next sampling time, and immediate reward of state-action pair at the current sampling time, construct sample data tuples;

[0028] When the fusion index of the state-action pair at the current sampling time is less than or equal to the fusion index threshold, the sample data tuple is written into the first fusion risk experience pool. When the fusion index of the state-action pair at the current sampling time is greater than the fusion index threshold, the sample data tuple is written into the second fusion risk experience pool.

[0029] In one embodiment, it further includes:

[0030] According to the preset cycle and based on the fusion indicators, a preset number of sample data tuples are extracted from the first fusion risk experience pool and the second fusion risk experience pool respectively.

[0031] For each sample data tuple, the immediate reward is corrected according to a preset reward function to obtain a corrected immediate reward that includes task reward, risk penalty, and uncertainty penalty. The state vector at the next sampling time is input into a preset target action network to obtain candidate actions at the next sampling time. The state vector and candidate actions at the next sampling time are input into a preset target evaluation network to obtain the evaluation result at the next sampling time. The target action network is a replica model with the same structure as the action prediction model, and the target evaluation network is a replica model with the same structure as the uncertainty decision model.

[0032] Based on the final action and the state vector, determine the uncertainty metric of the sample data tuple;

[0033] The model parameters of the action prediction model and the uncertainty decision model are updated based on the final action, state vector, corrected immediate reward, uncertainty metric, and evaluation results at the next sampling time.

[0034] In one embodiment, the model parameters of the action prediction model and the uncertain decision-making model are updated based on the corrected immediate reward, the uncertainty metric, and the evaluation result at the next sampling time, including:

[0035] Based on the corrected immediate reward and the evaluation results at the next sampling time, calculate the temporal difference target value at the next sampling time;

[0036] The parameters of the uncertainty decision model are updated with the goal of minimizing the error between the uncertainty metric and the time-series difference objective value.

[0037] The parameters of the action prediction model are updated with the goal of maximizing the uncertainty metric.

[0038] Secondly, this application also provides an autonomous driving reinforcement learning decision-making device, comprising:

[0039] The candidate action determination module is used to determine the total risk based on the state vector. The state vector is input into the action prediction model trained by deep reinforcement learning to obtain the candidate actions corresponding to the state vector. The state vector is constructed based on the collected environmental information to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0040] The uncertainty determination module is used to input state-action pairs into an uncertainty decision model trained based on deep reinforcement learning, and obtain uncertainty metrics. A state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metrics are used to characterize the unreliability of the output of the action prediction model.

[0041] The fusion index determination module is used to calculate the fusion index based on the total risk and uncertainty measure. The fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk.

[0042] The final action determination module is used to determine the target area where the state action pair is located based on the fusion index, and to determine the final action based on the target area and candidate actions. The final action is used to control the vehicle's driving state.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0044] The total risk is determined based on the state vector. The state vector is input into the action prediction model trained by deep reinforcement learning to obtain the candidate action corresponding to the state vector. The state vector is constructed based on the collected environmental information and is used to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0045] The uncertainty metric is obtained by inputting state-action pairs into an uncertainty decision-making model trained based on deep reinforcement learning. A state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output of the action prediction model.

[0046] The fusion index is calculated based on the total risk and uncertainty measures. The fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk.

[0047] The target area where the state action pair is located is determined based on the fusion index, and the final action is determined based on the target area and candidate actions. The final action is used to control the vehicle driving state.

[0048] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0049] The total risk is determined based on the state vector. The state vector is input into the action prediction model trained by deep reinforcement learning to obtain the candidate action corresponding to the state vector. The state vector is constructed based on the collected environmental information and is used to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0050] The uncertainty metric is obtained by inputting state-action pairs into an uncertainty decision-making model trained based on deep reinforcement learning. A state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output of the action prediction model.

[0051] The fusion index is calculated based on the total risk and uncertainty measures. The fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk.

[0052] The target area where the state action pair is located is determined based on the fusion index, and the final action is determined based on the target area and candidate actions. The final action is used to control the vehicle driving state.

[0053] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0054] The total risk is determined based on the state vector. The state vector is input into the action prediction model trained by deep reinforcement learning to obtain the candidate action corresponding to the state vector. The state vector is constructed based on the collected environmental information and is used to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0055] The uncertainty metric is obtained by inputting state-action pairs into an uncertainty decision-making model trained based on deep reinforcement learning. A state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output of the action prediction model.

[0056] The fusion index is calculated based on the total risk and uncertainty measures. The fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk.

[0057] The target area where the state action pair is located is determined based on the fusion index, and the final action is determined based on the target area and candidate actions. The final action is used to control the vehicle driving state.

[0058] The aforementioned autonomous driving reinforcement learning decision-making method, device, computer equipment, computer-readable storage medium, and computer program product determine the total risk based on the state vector, determine the uncertainty metric through an uncertainty decision-making model, and calculate a fusion index based on the total risk and the uncertainty metric. This unifies the objective environmental risk and the subjective uncertainty of the model to the same scale, providing a single and easily interpretable representation for decision safety assessment. Furthermore, the fusion index determines the target area where the state-action pair is located. By calculating the fusion index, the risk and uncertainty of the state-action pair are simultaneously characterized and evaluated. By determining the target area, the risk and uncertainty of the state-action pair are divided, ensuring that state-action pairs with similar risks and uncertainties are located in the same target area. Finally, the final action is determined based on the target area and candidate actions. The target area allows for unified processing of both objective environmental risk and subjective model uncertainty, establishing an inherently unified evaluation mechanism for autonomous driving reinforcement learning decision-making. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is an application environment diagram of the reinforcement learning decision-making method for autonomous driving in one embodiment;

[0061] Figure 2 This is a flowchart illustrating an autonomous driving reinforcement learning decision-making method in one embodiment;

[0062] Figure 3 This is a flowchart illustrating the process of determining the target area where a state action pair is located in one embodiment;

[0063] Figure 4 This is a flowchart illustrating the process of determining the final action in one embodiment;

[0064] Figure 5 Here is a diagram of a reinforcement learning decision architecture in one embodiment;

[0065] Figure 6 This is a structural block diagram of an autonomous driving reinforcement learning decision-making device in one embodiment;

[0066] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] First, some of the terms appearing in the embodiments of this application will be explained:

[0069] Reinforcement Learning Decision-Making refers to the process of using reinforcement learning (RL) algorithms to directly output longitudinal and lateral control commands (such as acceleration and steering rate) for vehicles in autonomous driving scenarios, and iteratively optimizing decision strategies by interacting with the simulated or real environment and using reward signals as feedback.

[0070] Risk Potential Field: This abstracts environmental elements such as road boundaries, lane lines, static obstacles, and surrounding vehicles into risk sources, and constructs a scalar potential function in space that varies with distance, relative speed, etc., to quantify the objective degree of danger of the scene.

[0071] Model uncertainty refers to the subjective confidence of a decision model in the output of a current state action. It can be estimated based on the dispersion of the output of a multi-Q network integration and is used to characterize the unreliability of the model in out-of-distribution or sparse data scenarios. In this embodiment of the invention, the uncertainty of the action prediction model is characterized by calculating an uncertainty metric.

[0072] Risk-Uncertainty Fusion Index: A unified scalar index formed by normalizing and weighting the environmental risk potential field and model uncertainty. It is used to simultaneously measure the objective risk of the scenario and the subjective credibility of the strategy, and to divide the safe zone, cautious zone and dangerous zone accordingly.

[0073] Experience Pool (Replay Buffer): In reinforcement learning, a buffer is used to store interaction samples such as states, actions, rewards, and the next state. In this invention, a dual experience pool structure is adopted, which is divided according to the fusion index.

[0074] RSS (Responsibility Sensitive Safety) model: A formal safety model for autonomous driving. By constraining the longitudinal and lateral distances and acceleration / deceleration boundaries between vehicles, it provides a set of provably safe behaviors for safety filtering and takeover of RL policies.

[0075] Safety Intervention: During the execution phase, RSS-constrained pruning, projection, or interpolation is applied to the RL output actions to progressively increase the control weights of the rule model in high-risk, high-uncertainty scenarios, thereby avoiding serious consequences such as collisions.

[0076] In existing technologies, dynamic risk perception and reinforcement learning decision-making are typically modeled separately: one approach identifies high-risk targets by constructing risk potential fields and attention mechanisms, constraining vehicle behavior from an environmental perspective; another approach estimates uncertainty by integrating reinforcement learning models, switching to rule-based or empirical models in high-uncertainty scenarios. However, existing methods generally lack a unified quantitative framework that can simultaneously characterize both objective environmental risks and subjective model uncertainties within the same indicator system, making it difficult to systematically utilize these two types of information during the training and execution phases.

[0077] ① The uncertainty was not explicitly considered during the training phase, which may lead to overlearning of biased samples in areas that the model cannot understand;

[0078] ② The execution phase often uses simple threshold switching, which fails to smoothly adjust the intensity of safety intervention according to the level of risk and uncertainty;

[0079] ③ There is a lack of a mechanism to uniformly couple reinforcement learning strategies with formal security models such as RSS.

[0080] Therefore, the core technical problem of this application is: how to integrate the environmental risk potential field and model uncertainty, construct a unified risk uncertainty fusion index, and design a dual experience pool management and RSS safety intervention strategy based on the index to achieve a highly safe, robust and efficient training method for autonomous driving reinforcement learning decision-making.

[0081] Existing autonomous driving systems typically employ a layered architecture of perception, decision-making, planning, and control. The perception layer outputs information about surrounding traffic bodies and the road environment; the decision-making layer generates longitudinal and lateral behavioral intentions based on the perception results and planning objectives; the planning layer generates feasible trajectories; and the control layer executes trajectory tracking. Under this architecture, the safety and robustness of the decision-making method directly determine the quality of the vehicle's behavior in complex traffic environments.

[0082] With the development of deep reinforcement learning, an increasing number of studies are attempting to use RL methods to directly map states to control actions, enabling continuous decision-making for behaviors such as lane changing, overtaking, and following. Simultaneously, risk perception and safety assessment technologies are also rapidly evolving. Schemes based on dynamic risk indices, potential field modeling, and attention mechanisms can highlight high-risk targets in multi-objective scenarios. However, risk perception and RL decision-making are often weakly coupled: risk is primarily incorporated into RL through rewards / penalties or simple constraints, while the uncertainty of the RL model is rarely used to feed back into risk assessment and training data management.

[0083] On the other hand, existing work has recognized the importance of model uncertainty for autonomous driving safety and has begun to utilize methods such as Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3) to estimate Q-value variance, identify distributed scenarios, and switch to rule-based policies such as Intelligent Driver Model (IDM) and Minimizing Overall Braking Induced by Lane Changes (MOBIL) in high uncertainty regions. However, these methods only treat uncertainty as a switching switch and lack joint modeling with environmental risks; safety constraints mostly rely on empirical models and lack deep integration with formal models that can prove safety, such as RSS.

[0084] In terms of engineering implementation, autonomous driving systems designed for large-scale open road applications not only need to maintain decision-making efficiency and comfort in most common scenarios, but also must ensure safety redundancy in extreme long-tail scenarios. Therefore, a decision-making framework that can explicitly characterize and manage the combination of risk and uncertainty is needed to provide stronger interpretability and verifiability support for software and hardware safety certification.

[0085] Existing technology 1 proposes a method to assess the risk of surrounding traffic participants using multi-source perception information and to assign higher weights to high-risk targets through an attention mechanism, thereby constraining vehicle decision-making. A typical process of this scheme includes:

[0086] ① Based on radar, cameras, and other sensors, information such as the position, speed, and acceleration of various traffic targets is obtained;

[0087] ② Construct a dynamic risk index that reflects the target collision risk, approach speed, etc.;

[0088] ③ Use attention mechanisms to enhance feature extraction of high-risk targets and generate weighted scene representations;

[0089] ④ Input the scenario representation into the decision network or rule engine, and output vertical and horizontal decision instructions.

[0090] Existing technology can, to some extent, highlight high-risk targets and improve the safety of decision-making. However, its main feature is that it characterizes risk from the environmental perspective, without explicitly introducing the uncertainty of the reinforcement learning model, nor using a unified indicator to integrate risk and uncertainty.

[0091] Existing technology two proposes an autonomous driving decision-making system that incorporates multimodal inputs such as environmental semantics, behavioral intent, and rule information, and improves the interpretability of decision results through a modular structure. This solution typically includes:

[0092] ① Obtain multimodal information such as the vehicle's own status, the status of surrounding traffic participants, high-precision maps, and traffic rules;

[0093] ② Encode different modal information into feature vectors and fuse them to obtain a high-dimensional representation of the scene;

[0094] ③ Output decision results through interpretable policy networks (such as rule trees, hierarchical networks, etc.) and decompose and display the basis for decision-making;

[0095] ④ The decision results are then sent to the planning and control module for execution.

[0096] Existing technology 2 emphasizes multimodal input and decision interpretability, but still mainly focuses on the interpretation of environmental semantics and rules. It does not explicitly model the model uncertainty of reinforcement learning strategies, nor does it construct a dual experience pool and RSS security intervention mechanism with risk and uncertainty as the core.

[0097] In summary, the shortcomings of existing technologies are as follows:

[0098] (1) Risk perception is disconnected from model uncertainty, and there is a lack of a unified indicator system;

[0099] Existing technology one improves the ability to characterize environmental risks through dynamic risk perception and attention focus, but does not consider the subjective uncertainty of the decision model in out-of-distribution scenarios; Existing technology two emphasizes multimodal interpretability, but does not define a measurable model uncertainty index, which makes it impossible to evaluate the reliability of decisions from the perspective of model confidence.

[0100] (2) The training process lacks a sample selection and management mechanism based on risk and uncertainty;

[0101] Existing reinforcement learning decision-making methods often use a single experience pool to store samples uniformly. The sampling strategy is usually based only on TD-Error or random sampling, without combining environmental risks and model uncertainties to carry out fine-grained hierarchical management of samples. This can easily lead to problems such as overexploration in high-risk and high-uncertainty regions, affecting training stability, and underutilization of samples in low-risk and low-uncertainty regions, affecting policy convergence efficiency.

[0102] (3) The security intervention and rule / strategy switching are crude;

[0103] Some uncertainty-aware decision-making methods only hard switch to rule-based strategies when uncertainty exceeds a threshold, ignoring the joint impact of environmental risk levels and model uncertainty. Meanwhile, most schemes use empirical models or heuristic rules, lacking system coupling with provable safety models, making it difficult to provide strong support for engineering safety certification.

[0104] The autonomous driving reinforcement learning decision-making method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart TVs, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0105] In one exemplary embodiment, such as Figure 2 As shown, a reinforcement learning decision-making method for autonomous driving is provided, which is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0106] Step 202: Determine the total risk based on the state vector. Input the state vector into the action prediction model trained by deep reinforcement learning to obtain the candidate actions corresponding to the state vector. The state vector is constructed based on the collected environmental information and is used to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0107] For example, at each sampling time, environmental information is collected, including the vehicle's state, the surrounding vehicles' states, and road information. Based on the vehicle's state, the surrounding vehicles' states, and the road information, a unified state vector is defined for the vehicle and the surrounding targets. The state vector may include: two-dimensional position, velocity, acceleration, heading angle, relative distance, lane number, traffic light status, etc. For example, environmental information can be collected through visual acquisition devices such as cameras, or satellite detection devices such as radar.

[0108] Risk sources can include road boundaries, lane lines, vehicles traveling in the same direction, vehicles traveling in the opposite direction, and disabled vehicles. In order to characterize the objective degree of environmental danger, Gaussian risk potential functions are constructed for each type of risk source in this embodiment of the application to obtain the potential field of each type of risk source. The potential fields of each type of risk source are superimposed to obtain the total risk. Through the above Gaussian potential field modeling, the multi-source danger information in the environment can be uniformly quantified into continuous values, providing measurable input for subsequent fusion with model uncertainty.

[0109] The action prediction model uses an Actor policy network, which can output corresponding action decisions based on the input environment state. In this embodiment, the Actor policy network, also known as the policy network, is a core component in the deep reinforcement learning framework used to directly generate continuous action decisions based on the environment state.

[0110] In an exemplary embodiment, the Actor policy network takes a state vector as input and outputs candidate actions. Given a state vector, it can output a continuous action decision with a high expected cumulative reward. The candidate action is the original decision instruction directly calculated by the Actor policy network based on its previous learning experience and the state vector. The candidate action can be a two-dimensional vector containing longitudinal and lateral control variables. The longitudinal control variable can be represented by acceleration (e.g., positive values ​​for acceleration and negative values ​​for deceleration); the lateral control variable can be represented by steering wheel angle or steering rate (e.g., positive values ​​for left turns and negative values ​​for right turns). Assuming a following scenario, the Actor policy network receives a state vector, performs forward propagation based on its previous learning experience, and determines the optimal solution: maintain straight-line driving and gradually decelerate, i.e., outputs a candidate action of straight-line deceleration carrying both longitudinal and lateral control variables.

[0111] Step 204: Input the state-action pair into the uncertainty decision-making model trained based on deep reinforcement learning to obtain the uncertainty metric. The state-action pair consists of a state vector and the candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output result of the action prediction model.

[0112] For example, a state vector and a candidate action generated from the state vector are combined into a state-action pair, which represents the state vector and the candidate action corresponding to the state vector. That is, the state-action pair contains state vector data and candidate action data.

[0113] The state-action pair is input into the uncertainty decision model, and the uncertainty decision model outputs an uncertainty metric based on the state-action pair. In this embodiment, the uncertainty decision model outputs multiple Q values ​​through multiple parallel Critic networks. By constructing an uncertainty metric to characterize the differences between the Q values, the decision credibility of the action prediction model is determined. Here, the Q value is the output of each parallel network, that is, the evaluation value of each parallel network for the state-action pair, which is only used as an intermediate quantity to judge the reliability of the action prediction model.

[0114] In one embodiment, the uncertainty decision-making model is constructed in parallel by multiple evaluation networks, i.e., Critic networks, that have the same structure but different initial parameter values ​​and are updated independently. This model is used to output a quantitative index for a given state-action pair, which characterizes the reliability of the action prediction model's decision on that state-action pair.

[0115] At each point where uncertainty assessment of candidate actions is required, the current state vector is concatenated with the candidate actions output by the action prediction model to form a state-action vector. This vector contains both environmental state information and policy decision information, serving as input to the uncertainty decision-making model.

[0116] The state-action pair vectors are input in parallel to all evaluation networks. Each evaluation network can be an independent multi-layer fully connected feedforward neural network, consisting of an input layer, two hidden layers, and an output layer. The input layer is only responsible for data reception and distribution; the first hidden layer uses a linear rectified activation function; the second hidden layer also uses a linear rectified activation function; the output layer uses a linear activation function and outputs the Q-value for the state-action pair.

[0117] Because each evaluation network uses different random seeds for parameter initialization during training and updates its parameters independently via gradient descent, although they share the same structure, their evolution paths differ, resulting in multiple evaluation models with certain differences. When the same state-action pair is input into these evaluation networks, the Q-values ​​output by each network are not exactly the same.

[0118] After all evaluation networks complete their forward computation in parallel, they each output their Q-value for the state-action pair. At this point, the uncertainty decision model collects all Q-values, calculates the dispersion among them, and normalizes the dispersion results to obtain an uncertainty metric. A larger dispersion indicates more significant disagreement among the evaluation networks regarding the state-action pair, meaning the current state-action pair is located in a sparsely distributed region of the training data or differs significantly from the training samples, resulting in lower decision reliability for the action prediction model in this region. Conversely, a smaller dispersion indicates that the opinions of the evaluation networks tend to be consistent, meaning the current state-action pair is located in a region familiar and trusted by the model, resulting in higher decision reliability.

[0119] Step 206: Calculate the fusion index based on the total risk and uncertainty measure. The fusion index is used to characterize the unreliability of the output of the action prediction model and the scenario risk.

[0120] For example, the aforementioned total risk and uncertainty measures are normalized, and then the normalized total risk and uncertainty measures are weighted and fused to obtain a fusion index. The fusion index characterizes the combined risk of the unreliability of the action prediction model output and the scenario risk in vehicle driving. A smaller fusion index indicates a lower combined risk, while a larger fusion index indicates a higher combined risk. Here, the combined risk includes the total scenario risk of vehicle operation and the unreliability of the candidate actions output by the action prediction model.

[0121] Step 208: Determine the target area where the state action pair is located based on the fusion index, and determine the final action based on the target area and candidate actions. The final action is used to control the vehicle driving state.

[0122] For example, by comparing the fusion index with a preset fusion index threshold, the target area corresponding to the state-action pair is determined. This target area is not a fixed area, but a virtual concept used to characterize the risk level of autonomous driving. In this embodiment, it may include a safe zone, a cautious zone, and a danger zone. After determining the target area of ​​the state-action pair, the final action is determined based on candidate actions according to a preset adjustment strategy.

[0123] In the aforementioned reinforcement learning decision-making method for autonomous driving, the total risk is determined based on the state vector, and the uncertainty metric is determined through an uncertainty decision-making model. A fusion index is calculated based on the total risk and the uncertainty metric, unifying the objective environmental risk and the model's subjective uncertainty to the same scale, providing a single, easily interpretable representation for decision safety assessment. Then, the target region where the state-action pair is located is determined based on the fusion index. The calculation of the fusion index simultaneously characterizes and evaluates the risk and uncertainty of the state-action pair. By determining the target region, the risk and uncertainty of the state-action pair are divided, ensuring that state-action pairs with similar risks and uncertainties are located in the same target region. Finally, the final action is determined based on the target region and candidate actions. The objective environmental risk and the model's subjective uncertainty can be uniformly processed based on the target region, establishing an inherently unified evaluation mechanism for reinforcement learning decision-making in autonomous driving.

[0124] In one embodiment of this application, determining the total risk based on the state vector includes:

[0125] By inputting the state vector into the Gaussian risk potential function of each risk source, the potential field of each risk source is obtained. The Gaussian risk potential function of each risk source is constructed based on environmental information.

[0126] The total risk is obtained by superimposing the potential fields of each risk source.

[0127] For example, risk sources are identified based on environmental information, and Gaussian risk potential functions for each risk source are constructed. The state vector is input into the Gaussian risk potential function of each risk source to obtain the potential field of each risk source. The potential fields of each risk source are superimposed to obtain the total risk potential field, and the total risk is determined. Here, risk sources are objective elements in the vehicle-type environment that may pose a threat to the safety of the vehicle. Therefore, based on environmental information, we can start from the road structure, traffic participants, traffic rules, etc., to find objective elements that can be represented by mathematical functions and dynamically updated based on the vehicle state as risk sources.

[0128] In one embodiment, for the current sampling time t, a unified state vector s for the vehicle and surrounding targets is defined. t This includes: two-dimensional position, velocity, acceleration, heading angle, relative distance, lane number, traffic light status, etc.

[0129] To characterize the objective degree of environmental hazard, this application's embodiments construct a Gaussian risk potential function for each type of risk source j (such as road boundaries, lane lines, vehicles traveling in the same direction, vehicles traveling in the opposite direction, disabled vehicles, etc.):

[0130]

[0131] in, For the risk source potential field, The risk source intensity coefficient, A comprehensive distance metric considering both relative distance and relative speed between the vehicle and the risk source. This is the parameter for the risk diffusion scale.

[0132] The total risk is obtained by superimposing the potential fields of all risk sources:

[0133]

[0134] in, For the overall risk field, This is a collection of road boundaries / lane lines / static obstacles. For risk sources such as road boundaries, lane lines, and static obstacles, For the weekly vehicle risk set, For the risk source of Zhou Che, For rule risk items defined based on speed limits, restricted areas, etc.

[0135]

[0136]

[0137] The speeding rule only penalizes the portion of the speed exceeding the limit. , These are the weight coefficients for the corresponding items. For longitudinal velocity, Speed ​​limits for roads, Indicates a speed scale; To mitigate lane rule risks, vehicles will be penalized for entering the emergency lane or changing lanes across a solid line. These are signs indicating violations of traffic rules. By modeling different risk sources separately, road structure risks, vehicle interaction risks, and traffic rule risks can be distinguished in subsequent analyses.

[0138] In one embodiment of this application, the uncertainty decision-making model is constructed by a parallel evaluation network. The uncertainty metric is obtained by inputting state-action pairs into the uncertainty decision-making model trained using deep reinforcement learning, including:

[0139] The state-action pair is input into the evaluation network, and the uncertainty assessment results of each evaluation network are obtained. The uncertainty metric is calculated based on each uncertainty assessment result.

[0140] For example, an uncertainty decision-making model can consist of multiple parallel evaluation networks. The state-action pair is input into each evaluation network, the mean and variance of the output values ​​of each evaluation network are determined, the consistency of the output results of each evaluation network is judged, and then the uncertainty of the output results of the action prediction model is determined by normalization.

[0141] In one embodiment, an ensemble reinforcement learning structure comprising K parallel Critic networks is employed. For any state-action pair... The Q-values ​​of each Critic network output The mean and variance of the ensemble Q-values ​​are defined as follows:

[0142]

[0143]

[0144] in, The mean of Q-values The variance of the Q value, This is the Q value.

[0145] Further define the normalized uncertainty measure:

[0146]

[0147] in, This is a measure of the model's uncertainty. To prevent small constants with a denominator of zero, denoted as the standard deviation of the Q value. The larger the value, the greater the disagreement among the integrated critic networks regarding the action pair in this state, and the lower the reliability of the action prediction model's output in this region.

[0148] In one exemplary embodiment, such as Figure 3 As shown, step 208 includes steps 302 to 306. Wherein:

[0149] Step 302: When the fusion index of the state-action pair is less than or equal to the first fusion index threshold, the target area of ​​the state-action pair is determined as a safe area.

[0150] Step 304: When the fusion index of the state-action pair is greater than the first fusion index threshold and less than or equal to the second fusion index threshold, the target area of ​​the state-action pair is determined to be a safe area.

[0151] Step 306: When the fusion index of the state-action pair is greater than the second fusion index threshold, the target area of ​​the state-action pair is determined as a safe area.

[0152] For example, embodiments of this application normalize the total risk and uncertainty metrics:

[0153]

[0154] in, The normalized total risk value. This is the normalized measure of uncertainty. These represent the highest risk, lowest risk, maximum uncertainty, and minimum uncertainty, respectively. These values ​​can be obtained through offline statistics.

[0155] Redefining the fusion index of risk and uncertainty measures:

[0156]

[0157] in, State-action pair The fusion index, with adjustable weights α and β, is used to balance the importance of environmental risk and uncertainty measures. The size of the state-action pair is used to divide the state-action pair into:

[0158] Safe Zone: ;

[0159] Caution zone: ;

[0160] Danger Zone: .

[0161] in, The threshold for the first fusion metric. The threshold for the second fusion metric. and The value of can be set by those skilled in the art based on requirements, and no specific limitation is made here in the embodiments of this application.

[0162] In one exemplary embodiment, such as Figure 4 As shown, step 208 further includes steps 402 to 408. Wherein:

[0163] Step 402: Determine a set of safe actions based on the state vector. The set of safe actions includes at least one safe action, and the safe action is located within the safe distance and safe boundary determined based on the state vector.

[0164] In this embodiment, the longitudinal safety distance and lateral safety boundary between vehicles are calculated by introducing an RSS model. A set of safe actions is constructed based on the longitudinal safety distance and lateral safety boundary. The establishment of the set of safe actions provides further decision-making factors for vehicle action decisions.

[0165] For example, a permissible longitudinal safe distance or range of motion is determined. If the actual distance between the vehicle and the vehicle in front is greater than or equal to the required safe distance, the longitudinal movement is considered safe. In this case, the permissible range of longitudinal movement is relatively wide, which can include various operations ranging from appropriate deceleration to moderate acceleration. If the actual distance is less than the required safe distance, the longitudinal movement is considered unsafe. In this case, the longitudinal movement of the vehicle must be restricted, and only deceleration can be performed. Furthermore, the deceleration needs to reach a certain intensity to ensure that the safe distance does not decrease or begins to increase.

[0166] For example, when following distance is insufficient, the only allowed longitudinal movement of the vehicle is deceleration, and the deceleration must be controlled within a reasonable range.

[0167] Determine the permissible lateral safety boundaries or range of motion. In lane-keeping scenarios, the vehicle's lateral position must always remain within the range defined by the left and right lane lines, and a certain safety margin must be maintained between the vehicle and the lane lines. In other words, steering maneuvers should not cause the vehicle to cross the lane lines or deviate from the lane. Based on the current lateral position and the distance to the lane boundaries, a permissible steering rate range can be derived to ensure that the vehicle does not cross the lane boundaries within a specified time.

[0168] In scenarios involving an intention to change lanes, an additional assessment of the safety of the lane change is required. If the lane change operation would pose a risk of collision with vehicles in adjacent lanes, then turning in that direction is prohibited; if the lane change is safe, then the corresponding turning operation is permitted.

[0169] Step 404: If the target area is a safe area, and the set of safe actions contains candidate actions, then the candidate actions are determined as the final actions; or if the set of safe actions does not contain candidate actions, then the candidate actions are projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action.

[0170] For example, the target area where the state-action pair is located is determined based on the fusion index. When the state-action pair is located in the safe zone, it is determined whether the candidate action exists in the set of safe actions constructed based on the RSS responsibility-sensitive safety model. If it exists, the candidate action is directly output as the final action. If it does not exist, the candidate action is projected into the set of safe actions, the candidate action is adjusted according to the projection result, and then the final action is generated.

[0171] Projecting candidate actions onto the set of safe actions can be done in two dimensions. For the longitudinal acceleration dimension, the longitudinal acceleration value of the candidate action is compared with the upper and lower boundaries of the safe longitudinal interval. If the longitudinal acceleration is less than the minimum value of the safe interval, the longitudinal acceleration is corrected to the minimum value of the safe interval; if the longitudinal acceleration is greater than the maximum value of the safe interval, the longitudinal acceleration is corrected to the maximum value of the safe interval; if the longitudinal acceleration is already within the safe interval, the original value is left unchanged.

[0172] For the lateral steering ratio dimension, the same processing method is used: compare the lateral steering ratio of the candidate action with the upper and lower boundaries of the safe lateral interval. If it exceeds the boundary, it is corrected to the nearest boundary value; if it is within the interval, it remains unchanged.

[0173] Combining the corrected values ​​from the two dimensions yields the projected action. This projected action is the point in the set of safe actions that is closest to the candidate action.

[0174] Based on the projected action, the out-of-dimension of the candidate action is corrected to the nearest boundary value, while keeping the dimension within the interval unchanged. The target action obtained from the projection is directly used as the final action output.

[0175] Step 406: If the target area is a caution zone, select reference actions from the set of safe actions, perform weighted fusion on the candidate actions and reference actions, and output the weighted fusion result as the final action.

[0176] For example, when the state action pair is located in the caution zone, reference actions are selected from the set of safe actions, and the reference actions and candidate actions are weighted and fused according to preset weights to obtain the final action.

[0177] In this embodiment of the invention, the reference action can be determined in a variety of ways, such as: selecting the point in the set of safe actions that is closest to the candidate action as the reference action; selecting the optimal action in a certain preset direction in the set of safe actions; or pre-defining several typical reference points in the set of safe actions.

[0178] Based on the fusion index of the state-action pair, corresponding interpolation weights are constructed. When the fusion index just exceeds the safe zone threshold and enters the cautious zone, the candidate action should have a larger weight, and the reference action should have a smaller weight, i.e., a slight correction to the original decision. As the fusion index increases and gradually approaches the danger zone threshold, the weight of the candidate action gradually decreases, and the weight of the reference action gradually increases, i.e., the intensity of safety intervention gradually increases. When the fusion index reaches the danger zone threshold, the weight of the reference action should be close to complete takeover.

[0179] After determining the interpolation weights, the reference action and candidate action are weighted and fused according to the following formula to obtain the final action.

[0180]

[0181] in, In order to be in The reference action obtained through optimization.

[0182] Step 408: If the target area is located in a danger zone, project the candidate actions into the set of safe actions, and output the data in the set of safe actions that is closest to the projection as the final action.

[0183] For example, when the state action pair is located in the danger zone, it is determined whether the candidate action exists in the set of safety actions constructed based on the RSS responsibility-sensitive safety model; if it does not exist, the candidate action is projected to the closest position in the set of safety actions and then the final action is generated; if it exists, the candidate action is directly output as the final action example; if there is no feasible projection, the vehicle is controlled by the emergency action given by the RSS model.

[0184] When a state-action pair is located in a danger zone, an attempt is made to project it into the set of safe actions. The goal of the projection is to find a point within the safe set that is closest to the original candidate action, as shown in the following formula:

[0185]

[0186] Among them, A RSSe Represents a set of safety actions. This represents the minimum amount of control modification required between the corrected action and the original action.

[0187] The projection operation uses the same method as the safe zone: the two dimensions are processed independently. The difference is that the projection operation in the safe zone is mainly for reminders and is only used as a reference. The probability of the candidate action causing danger to the vehicle is small. However, the projection operation in the danger zone needs to find an action within the set of safe actions as the final action. Otherwise, it may cause danger to the vehicle.

[0188] In one embodiment, during the execution phase, this application introduces an RSS model to calculate the longitudinal safety distance and lateral safety boundary between vehicles in order to construct a set of safety actions. Taking a longitudinal following scenario as an example, the safe distance given by RSS can be expressed as:

[0189]

[0190] in, For RSS longitudinal safety distance, For the speed of the following vehicle, The speed of the vehicle in front. For system response time, This is the minimum braking acceleration that the vehicle can achieve. This represents the maximum possible braking acceleration of the vehicle in front. Combining lateral clearance and lateral dynamic constraints, a set of safe actions satisfying the RSS condition can be constructed. .

[0191] For example, a set of security actions can be constructed as follows:

[0192] The minimum safe distance is calculated based on a responsibility-sensitive safety model. If the actual distance is sufficient, the permissible range of longitudinal movement is wide; if the actual distance is insufficient, deceleration is the only option, limiting the longitudinal range to a deceleration interval. Based on lane-keeping requirements, ensuring the vehicle does not cross the lane lines, the permissible steering rate range is derived. The intersection of the permissible longitudinal acceleration interval and the permissible lateral steering rate interval yields a two-dimensional rectangular region. Any movement within this region simultaneously satisfies both longitudinal and lateral safety requirements, constituting a set of safe movements.

[0193] For the candidate actions of the output In this application, different intervention strategies are implemented according to the region where the fusion index is located:

[0194] safe zone ( ):like If the conditions are met, the operation is executed directly; otherwise, a slight projection correction is performed.

[0195] Caution zone ( ): Construction depends on interpolation weights A weighted fusion is performed between RL actions and RSS safety actions.

[0196] Danger Zone ( ):like If the RSS constraint is not satisfied, project it to the nearest position in the set of safe actions.

[0197] If no feasible projection exists, then the emergency takeover control is entirely provided by the RSS model.

[0198] For example, the RSS model operates in each decision cycle according to the following steps:

[0199] Two-dimensional position, velocity, acceleration, heading angle, relative distance, and other data are extracted from the state vector, and then the longitudinal velocity of the vehicle, the maximum and minimum braking deceleration of the vehicle, the longitudinal velocity of the vehicle in front, the maximum braking deceleration of the vehicle in front, the actual relative distance between the vehicle and the vehicle in front, the lateral position of the vehicle, the lane boundary, and the relative position and velocity of target vehicles in adjacent lanes are calculated.

[0200] The RSS model calculates the minimum longitudinal safety distance to be maintained at the current moment based on the vehicle's speed, the speed of the vehicle in front, the system reaction time, the minimum braking deceleration of the vehicle, and the maximum possible braking deceleration of the vehicle in front. The calculated safety distance threshold is compared with the actual relative distance: if the actual distance is less than the safety threshold, the longitudinal state is deemed unsafe.

[0201] Based on the safety distance constraint, the permissible longitudinal acceleration range of the vehicle is derived: the upper limit of this range can be a non-positive deceleration value, and the lower limit can be the minimum braking deceleration that the vehicle can achieve. This range is the longitudinal safe operating interval.

[0202] For lane-keeping scenarios, the RSS model treats the left and right lane lines as insurmountable boundaries, restricting the vehicle's lateral position to within the lane lines while maintaining a safety margin. For lane-changing scenarios, the RSS model evaluates the relative position, relative speed, and geometry of the vehicle and the target vehicle in the adjacent lane, constructing a two-dimensional safety envelope. A lane-changing maneuver is considered safe only if the vehicle's predicted trajectory falls entirely within this area and does not overlap with the target vehicle's envelope.

[0203] Based on the lateral safety boundary, the range of lateral steering rates that the vehicle is allowed to perform is derived, i.e., the lateral safety action range.

[0204] The intersection of the longitudinal and lateral safe action intervals yields the complete set of safe actions at the current moment. Any action within this set satisfies the provable safety condition: regardless of how surrounding vehicles react within a reasonable range, no collision attributable to the vehicle will occur.

[0205] Receive candidate actions output by the action prediction model and determine whether the longitudinal acceleration and lateral steering rate of the candidate action both fall within the above set of safe actions.

[0206] If the candidate action is already in the safe set: directly output it as the final action.

[0207] If the candidate action is not within the safe set, then different levels of intervention are performed based on the target region to which the fusion indicator belongs.

[0208] When the state-action pair is located in the safe zone, the candidate action is projected onto the boundary of the safe action set, and the action closest to the original candidate action is selected for the smallest correction.

[0209] When the state-action pair is located in the caution zone, a reference action that is consistent with the direction of the candidate action is selected from the set of safe actions. The interpolation weight is calculated based on the fusion index value, and the candidate action and the reference action are weighted and fused to obtain the final action.

[0210] When the state action is located in the danger zone, the candidate action is directly projected to the point with the closest Euclidean distance in the safe action set; if the safe action set is empty or there is no feasible projection, the RSS model takes over completely and outputs the preset emergency braking and lane keeping instructions.

[0211] The final action obtained after the above-mentioned judgment, correction, fusion or takeover operations is sent to the execution module, which converts it into throttle, braking and steering control signals to complete the control of the vehicle's driving state.

[0212] Through the above-described process, the embodiments of this application achieve deep coupling between the provable security model and reinforcement learning decision-making: the fusion index drives continuous adjustment of intervention intensity, avoiding control abrupt changes caused by hard switching; the abstract security constraints are transformed into specific action boundaries, providing verifiable security guarantees for the learning strategy; and at the same time, good interpretability and engineering certification feasibility are maintained.

[0213] Through the above design, the embodiments of this application achieve the integration of metrics. The driven continuous safety intervention mechanism is smoother and more adjustable compared to simple hard switching with high uncertainty.

[0214] In one embodiment of this application, it further includes:

[0215] Based on the final action and state vector, determine the state-action pair at the current sampling time, and calculate the fusion index of the state-action pair at the current sampling time based on the state-action pair at the current sampling time.

[0216] After controlling the vehicle's driving state according to the final action, the state vector at the next sampling time is obtained. Based on the state vector at the next sampling time, the immediate reward of the state-action pair at the current sampling time is determined. The immediate reward is used to construct the task reward function, so as to iteratively update the vehicle's control strategy based on the task reward function.

[0217] Based on the final action, state vector, fusion index of state-action pair at the current sampling time, state vector at the next sampling time, and immediate reward of state-action pair at the current sampling time, construct sample data tuples;

[0218] When the fusion index of the state-action pair at the current sampling time is less than or equal to the fusion index threshold, the sample data tuple is written into the first fusion risk experience pool. When the fusion index of the state-action pair at the current sampling time is greater than the fusion index threshold, the sample data tuple is written into the second fusion risk experience pool.

[0219] For example, the data related to the final action generated during the historical operation are written into the fusion risk experience pool as sample data tuples. In subsequent operations, the action prediction model and the uncertainty decision model can be iteratively updated based on the data in the fusion risk experience pool to improve the model prediction accuracy. Specifically, after controlling the vehicle to drive according to the final action, the state vector of the next sampling moment can be obtained. The state vector of the next sampling moment is packaged and stored with the data of the current sampling moment, which is convenient for adjusting the model parameters according to the data changes between the two sampling moments.

[0220] In one embodiment, this application constructs a composite reward function that includes task reward, risk penalty, and uncertainty penalty:

[0221]

[0222] in, Used to encourage the completion of traffic tasks such as lane changing, overtaking, and following other vehicles. The penalty weighting coefficient is used. This reward design guides the strategy to favor actions with low risk, low uncertainty, and good task results. The task reward function... The contribution of the current action to the completion of driving tasks such as lane changing, overtaking, and following can be defined as a weighted combination of task progress, lane keeping effect, following distance maintenance effect, and traffic efficiency.

[0223]

[0224] in, The path advance is used to indicate whether the vehicle is moving towards the mission objective. This refers to the lateral deviation relative to the centerline of the target lane. This is the current following distance. To maintain the desired following distance, Current vehicle speed For the desired vehicle speed, These are the weighting coefficients. In this patent, the task reward function... This represents the specific value in the current state-action pair, which can be considered as the original immediate reward. The final single-step reward used during training is a composite reward. .

[0225] Each step in the interaction with the environment generates a sample tuple. .in, This represents the state at the current time t. For the state at the next moment, For the action at the current moment, The immediate reward for the current state action can be considered as a task reward function; in this patent, this value is as designed above. , This represents the fusion metric for the current state-action pair. During training, this value is continuously evaluated in real time for the current behavior, and based on... Divide the samples into different experience pools:

[0226] when The sample tuple is written into the low-fusion-risk experience pool. ;

[0227] when At that time, the sample tuple is written into the high-fusion-risk experience pool. .

[0228] In one embodiment of this application, it further includes:

[0229] According to the preset period, a preset number of sample data tuples are extracted from the first fusion risk experience pool and the second fusion risk experience pool respectively, based on the fusion index.

[0230] For each sample data tuple, the immediate reward is corrected according to a preset reward function to obtain a corrected immediate reward that includes task reward, risk penalty, and uncertainty penalty. The state vector at the next sampling time is input into a preset target action network to obtain candidate actions at the next sampling time. The state vector and candidate actions at the next sampling time are input into a preset target evaluation network to obtain the evaluation result at the next sampling time. The target action network is a replica model with the same structure as the action prediction model, and the target evaluation network is a replica model with the same structure as the uncertainty decision model.

[0231] Based on the final action and the state vector, determine the uncertainty metric of the sample data tuple;

[0232] The model parameters of the action prediction model and the uncertainty decision model are updated based on the final action, state vector, corrected immediate reward, uncertainty metric, and evaluation results at the next sampling time.

[0233] For example, the target action network is a parameter-lag-updated copy of the action prediction model, and the target evaluation network is a parameter-lag-updated copy of the uncertain decision model. The target action network has the same architecture as the action prediction model, except that the model parameters are updated later than those of the action prediction model. Similarly, the target evaluation network has the same architecture as the uncertain decision model, except that the model parameters are updated later than those of the uncertain decision model.

[0234] The target action network is the same as the action prediction model, except that its parameters are updated later and at a lower frequency. When the action prediction model is optimized in each iteration, the model parameters of the target action network are relatively stable, which can guide the update of the action prediction model to be relatively stable. When the action prediction model is updated by itself, each update may cause drastic changes in the model parameters. The target evaluation network is similar to the uncertainty decision model. By setting up a replica model, the model update is ensured to be relatively stable.

[0235] During the training phase, stratified sampling is performed from two types of experience pools according to a preset ratio, for example, using... The probability from Mid-sampling, with The probability from Mid-sampling, and It can be gradually increased as training progresses.

[0236] Within each experience pool, this invention implements priority sampling based on a fusion metric. For the i-th sample, its sampling probability can be defined as:

[0237]

[0238] in, For sampling probability, For temperature parameters, This is the fusion metric for the sample. This design allows samples with lower fusion metrics to be used for updates more frequently within the same experience pool, thereby enhancing the stability of the policy in low-fusion-risk regions while retaining adequate learning on high-fusion-risk samples.

[0239] In one embodiment of this application, the model parameters of the action prediction model and the uncertainty decision model are updated based on the modified immediate reward, the uncertainty metric, and the evaluation result at the next sampling time, including:

[0240] Based on the corrected immediate reward and the evaluation results at the next sampling time, calculate the temporal difference target value at the next sampling time;

[0241] The parameters of the uncertainty decision model are updated with the goal of minimizing the error between the uncertainty metric and the time-series difference objective value.

[0242] The parameters of the action prediction model are updated with the goal of maximizing the uncertainty metric.

[0243] For example, the network parameters of the action prediction model and the uncertainty decision model are updated respectively in the following manner.

[0244] 1) Calculate the target value of the time series difference.

[0245] The corrected immediate reward is combined with the evaluation result obtained at the next sampling time through the target evaluation network.

[0246] For example, the modified immediate reward is added to the product of a preset discount factor and the evaluation result at the next sampling time to obtain the temporal difference target value. This temporal difference target value represents the reasonable expected return that the current state action should have after comprehensively considering the actual return of the current step and the estimated return of the future. The discount factor is applied to the modified immediate reward to adjust the long-term return target of the model.

[0247] 2) Update the parameters of the uncertainty decision-making model.

[0248] The uncertainty decision-making model consists of multiple parallel and structurally identical Critic networks. For each Critic network, the uncertainty metric of its output state-action pair at the current sampling time is compared with the aforementioned temporal difference objective value, and the difference between the two is calculated, which is the temporal difference error. The parameters of the Critic network are updated using the gradient backpropagation algorithm, with the goal of minimizing this temporal difference error, for example, by minimizing the mean squared error loss function. Since multiple Critic networks are updated independently with different initial parameters, they gradually learn estimation functions with certain differences. This difference forms the basis for subsequent estimation of model uncertainty.

[0249] During the update process, the update intensity can be weighted according to the fusion index of the samples. For samples with lower fusion indices, i.e., lower risk and more deterministic models, higher update weights are assigned to enable the model to learn from safe and reliable samples first; for samples with higher fusion indices, lower update weights are assigned to avoid the model overfitting noisy data in high-risk and high-uncertainty regions.

[0250] 3) Update the parameters of the action prediction model.

[0251] The action prediction model aims to maximize the estimated output action of the uncertain decision-making model. The state vector at the current sampling moment is input into the action prediction model to obtain candidate actions at that moment. This candidate action is then paired with the current state vector to form a state-action pair, which is input into the updated uncertain decision-making model to obtain the estimated result corresponding to that action. This estimated result reflects the expected cumulative return that can be obtained by executing the candidate action under the current policy.

[0252] The autonomous driving reinforcement learning decision-making method proposed in this application aims to improve the valuation result by calculating the policy gradient, i.e., calculating the gradient of the action prediction model output relative to its network parameters, and using the valuation result to guide the direction of this gradient. The parameters of the action prediction model are updated through a gradient ascent method. The updated action prediction model will be more inclined to output actions that can obtain higher valuations, i.e., those actions that are judged by the uncertainty decision-making model to have higher long-term returns.

[0253] Through these two parallel and coupled update processes, the uncertain decision-making model continuously learns how to more accurately evaluate the value of state-action pairs, while the action prediction model continuously adjusts its strategy to output actions evaluated as having higher value. This co-evolution ultimately enables the action prediction model to output optimal decisions that balance task efficiency, environmental safety, and model reliability in various driving scenarios.

[0254] A complete embodiment will be provided to illustrate the autonomous driving reinforcement learning decision-making method proposed in this application:

[0255] Figure 5 Here is a diagram of a reinforcement learning decision architecture in one embodiment, such as Figure 5 As shown, in practice, the autonomous driving reinforcement learning decision-making method proposed in this application embodiment can be implemented by constructing the following functional modules:

[0256] 1) Environmental perception and state construction module;

[0257] 2) Risk potential field construction and decomposition module;

[0258] 3) Uncertainty-aware reinforcement learning decision-making module;

[0259] 4) Risk and uncertainty fusion assessment module;

[0260] 5) Dual experience pool management and priority sampling module;

[0261] 6) RSS-based security intervention and action correction module;

[0262] 7) Execution module.

[0263] The leftmost module is the environment perception and state construction module, which takes as input the vehicle's state, surrounding vehicle states, and road information from sensors / perception algorithms, and outputs a unified state vector. .

[0264] The state vector is fed into the risk potential field construction and decomposition module, which outputs the total risk. Its decomposition; on the other hand, it is fed into the uncertainty-aware reinforcement learning decision-making module to output candidate actions. And intermediate quantities related to uncertainty.

[0265] The risk-uncertainty fusion assessment module simultaneously receives and uncertainty measure Calculate fusion index The results of the regional division are presented.

[0266] The dual experience pool management and priority sampling module writes interaction samples into the low-fusion-risk experience pool according to the fusion index. Experience pool with high integration risks It then returns the sampled batches to the RL decision module for training.

[0267] The RSS-based security intervention and action correction module simultaneously receives candidate actions. Integration indicators and environmental conditions Calculate the RSS security set Output the final action .

[0268] The execution module will The command is sent to the vehicle chassis for execution, and the state transition and reward information are fed back to the dual experience pools and the RL module.

[0269] like Figure 5 As shown in the embodiments of this application, the overall architecture of the autonomous driving reinforcement learning decision-making method mainly includes an environment perception and state construction module, a risk potential field construction and decomposition module, an uncertainty perception reinforcement learning decision-making module, a risk uncertainty fusion evaluation module, a dual experience pool management and priority sampling module, an RSS-based safety intervention and action correction module, and an execution and feedback module. The environment perception and state construction module is used to acquire the state and road structure information of the vehicle and surrounding traffic bodies, and construct the decision state vector s. t The risk potential field construction and decomposition module constructs potential fields for road risk and vehicle risk based on lane boundaries, static obstacles, and surrounding vehicle information, and then superimposes these fields to obtain the total risk E. tot (s t The uncertainty-aware reinforcement learning decision-making module employs a continuous action reinforcement learning algorithm with an integrated critic structure, given a state s. tNext output candidate action a RL (s t Simultaneously, the uncertainty metric U(s) is estimated by outputting a multi-Q network. t ,a t The risk-uncertainty fusion assessment module receives the total risk E. tot (s t ) and uncertainty U(s t ,a t The two are normalized and weighted to obtain the fusion index Φ(s). t ,a t Based on this, the safe zone, cautious zone, and danger zone are divided, and a composite reward system is constructed, including task reward, risk penalty, and uncertainty penalty. The dual experience pool management and priority sampling module classifies interaction samples according to fusion indicators, constructs a low-fusion-risk experience pool and a high-fusion-risk experience pool, and provides training batch data to the reinforcement learning decision module according to a set ratio and priority weight. The RSS-based safety intervention and action correction module adjusts the current state based on the action. t Calculate the RSS longitudinal safety distance and the set of safe actions, for candidate action a RL (s t Under the constraints of fusion metrics and RSS, make corrections or replacements, and output the final action a. t The execution and feedback module will handle the final action a. t The commands are converted into throttle, braking, and steering control commands and applied to the vehicle. The execution results and reward information are then fed back to the aforementioned modules, thus forming an integrated decision-making closed loop that incorporates risk potential field, model uncertainty, dual experience pool management, and RSS safety intervention.

[0270] When implementing the project, it can be done according to Figure 5 The modules shown are divided into functional partitions and software deployments on electronic control units (ECUs) or domain controllers.

[0271] Combining the above modules, the complete method flow of this invention can be summarized as follows:

[0272] 1) At each decision / sampling moment, the environment perception and state construction module outputs the current state vector s. t The risk potential field construction and decomposition module is based on s t Calculate total risk The uncertainty-aware reinforcement learning decision-making module provides candidate actions. And estimate the uncertainty measure. .

[0273] 2) The risk uncertainty fusion assessment module calculates fusion indicators. It also completes the division of safe zone, cautious zone and danger zone, and constructs a composite reward function that includes task reward, risk penalty and uncertainty penalty for subsequent strategy updates.

[0274] 3) The RSS-based security intervention and action correction module adjusts the current state s. t Calculate the RSS longitudinal safety distance and safety action set, and integrate the indicators. The candidate action is evaluated under the combined effect of the RSS constraint. Make corrections or replacements to obtain the final action a. t .

[0275] 4) The execution and feedback module will execute the final action a t The commands are converted into throttle, brake, and steering control commands, which are then applied to the vehicle, resulting in an immediate reward. t and the state vector s at the next time step t+1 In this process, the information such as the current state, action, reward, and next state is organized into sample data tuples. Write it into the dual experience pool, where This is the fusion index of the current state-action pair. The next state vector s t+1 This is used to construct temporal difference (TD) objectives in subsequent training phases. Specifically, it uses the action output by the policy network and the Q-value output by the critic network in the next state to calculate the target reward for the current sample, thereby ensuring that the Actor network and Critic network can continuously update using historical state transition relationships without re-interacting with the environment.

[0276] 5) Periodically sample batches of data from the dual experience pools in a stratified and weighted manner, and update the Actor / Critic network using a composite reward function. For example, during the training phase, small batches of experience samples are drawn from the low-fusion-risk experience pool and the high-fusion-risk experience pool at a preset ratio. Prioritize the selection of integration indicators Smaller samples (i.e., those with lower environmental risk and model uncertainty) are used for training. For each sample, the immediate reward is first modified into a composite reward function according to the reward design described above, i.e.:

[0277]

[0278] Then use the next state vector and target policy network Get the next action And by calculating the corresponding target Q-value through a target critic network, a time-difference target is constructed:

[0279]

[0280] Where γ is the discount factor, d i This is an indicator of whether the sample is in a terminated state. The target critic network updates its parameters by minimizing the weighted mean square error between the current Q-value and the target Q-value.

[0281]

[0282] Among them, w i Different weights can be set according to the fusion index of the samples to achieve risk-uncertainty-based priority training. The policy (Actor) network then performs gradient ascent updates on the same batch of samples, aiming to maximize the Q-value of the critic network's output action for the current policy, thus obtaining... Higher security rewards are generated within the state space covered by the experience pool. Through the above method, the experience pool stores... The samples are not only used to record execution results and fusion metrics, but also provide the basic data for constructing target rewards, evaluating policy quality, and implementing risk-uncertainty weighted training during the update process of the Actor / Critic network.

[0283] 6) Repeat the above decision execution and training steps. As the samples in the experience pool become richer and the parameters of the Actor / Critic network are continuously updated, the autonomous driving reinforcement learning decision-making method proposed in this application can gradually converge to a safe optimization solution that takes into account task efficiency, environmental risk and model uncertainty, while ensuring the formal safety constraints of RSS.

[0284] Compared to existing technologies 1 and 2, and other traditional autonomous driving decision-making methods, the autonomous driving reinforcement learning decision-making method proposed in this application has the following advantages:

[0285] ① Unified quantitative framework: By integrating risk uncertainty indicators, objective environmental risks and subjective model uncertainties are unified to the same scale, providing a single, interpretable quantitative basis for decision-making safety assessment.

[0286] ② Safe and efficient training process: By utilizing dual experience pools and a priority sampling mechanism based on fusion metrics, reinforcement learning focuses on consolidating low-fusion-risk samples in the early stages of training and appropriately expands high-fusion-risk scenarios in the later stages of training, thus balancing safety and policy generalization ability.

[0287] ③ Continuously adjustable safety intervention: Introducing the RSS formal safety model, and interpolating and projecting RL actions under the drive of fusion indicators, to achieve a continuous transition from almost no intervention to complete takeover, avoiding the comfort and stability problems caused by simple hard switching.

[0288] ④ Strong engineering verifiability: The fusion indicators, experience pool strategies, and RSS constraints all have clear physical meanings and formal expressions, which facilitates quantitative analysis and verification in functional safety and expected functional safety demonstrations, making them suitable for engineering implementation and certification.

[0289] Without departing from the basic concept of this invention, each module may have the following substitutions or equivalent modifications, all of which should be considered to fall within the protection scope of this invention:

[0290] 1) Uncertainty estimation methods for the model can be replaced by Bayesian neural networks, Monte Carlo Dropout, etc., instead of ensemble Q networks;

[0291] 2) The risk potential field can be replaced by an exponentially decaying, polynomially decaying, or learning-based risk network output;

[0292] 3) The number of dual experience pools can be expanded to multi-level risk uncertainty segmented experience pools, or continuous importance weights can be used to replace discrete pool division;

[0293] 4) The RSS model can be replaced by other behavioral safety models with formal security guarantees, as long as they still satisfy the idea of ​​continuous intervention based on fusion indicators;

[0294] 5) Reinforcement learning algorithms can be replaced by any continuous action RL method such as DDPG, TD3, or SAC, as long as the overall framework of uncertainty awareness + dual experience pools + RSS intervention is maintained.

[0295] The above alternative solutions, which take the risk uncertainty fusion index as the core idea of ​​this application, do not constitute a substantial change to the linkage training data management and RSS security intervention, and are all equivalent variations of this application.

[0296] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0297] Based on the same inventive concept, this application also provides an autonomous driving reinforcement learning decision-making device for implementing the aforementioned autonomous driving reinforcement learning decision-making method. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the autonomous driving reinforcement learning decision-making device provided below can be found in the limitations of the autonomous driving reinforcement learning decision-making method described above, and will not be repeated here.

[0298] In one exemplary embodiment, such as Figure 6 As shown, an autonomous driving reinforcement learning decision-making device is provided, comprising: a candidate action determination module, an uncertainty determination module, a fusion index determination module, and a final action determination module, wherein:

[0299] The candidate action determination module is used to determine the total risk based on the state vector. The state vector is input into the action prediction model trained by deep reinforcement learning to obtain the candidate actions corresponding to the state vector. The state vector is constructed based on the collected environmental information to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources.

[0300] The uncertainty determination module is used to input state-action pairs into an uncertainty decision model trained based on deep reinforcement learning, and obtain uncertainty metrics. A state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metrics are used to characterize the unreliability of the output of the action prediction model.

[0301] The fusion index determination module is used to calculate the fusion index based on the total risk and uncertainty measure. The fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk.

[0302] The final action determination module is used to determine the target area where the state action pair is located based on the fusion index, and to determine the final action based on the target area and candidate actions. The final action is used to control the vehicle's driving state.

[0303] In one embodiment of this application, the candidate action determination module shown is used for:

[0304] By inputting the state vector into the Gaussian risk potential function of each risk source, the potential field of each risk source is obtained. The Gaussian risk potential function of each risk source is constructed based on environmental information.

[0305] The total risk is obtained by superimposing the potential fields of each risk source.

[0306] In one embodiment of this application, the uncertainty determination module is used for:

[0307] The state-action pair is input into the evaluation network, and the uncertainty assessment results of each evaluation network are obtained. The uncertainty metric is calculated based on each uncertainty assessment result.

[0308] In one embodiment, determining the target region where the state-action pair is located based on fusion metrics includes:

[0309] When the fusion index of the state-action pair is less than or equal to the first fusion index threshold, the target area of ​​the state-action pair is determined as a safe zone.

[0310] When the fusion index of the state-action pair is greater than the first fusion index threshold and less than or equal to the second fusion index threshold, the target area of ​​the state-action pair is determined to be a safe area.

[0311] When the fusion index of the state-action pair is greater than the second fusion index threshold, the target region of the state-action pair is determined to be a safe region.

[0312] In one embodiment of this application, the final action determination module is used for:

[0313] The set of safe actions is determined based on the state vector. The set of safe actions includes at least one safe action, and the safe action is located within the safe distance and safe boundary determined based on the state vector.

[0314] If the target area is a safe area, and the set of safe actions contains candidate actions, then the candidate actions are determined as the final actions; or if the set of safe actions does not contain candidate actions, then the candidate actions are projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action.

[0315] When the target area is a caution zone, reference actions are selected from the set of safety actions, and the candidate actions and reference actions are weighted and fused. The weighted fusion result is then output as the final action.

[0316] If the target area is located in a danger zone, the candidate actions are projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action.

[0317] In one embodiment of this application, a model update module is further included, used for:

[0318] Based on the final action and state vector, determine the state-action pair at the current sampling time, and calculate the fusion index of the state-action pair at the current sampling time based on the state-action pair at the current sampling time.

[0319] After controlling the vehicle's driving state according to the final action, the state vector at the next sampling time is obtained. Based on the state vector at the next sampling time, the immediate reward of the state-action pair at the current sampling time is determined. The immediate reward is used to construct the task reward function, so as to iteratively update the vehicle's control strategy based on the task reward function.

[0320] Based on the final action, state vector, fusion index of state-action pair at the current sampling time, state vector at the next sampling time, and immediate reward of state-action pair at the current sampling time, construct sample data tuples;

[0321] When the fusion index of the state-action pair at the current sampling time is less than or equal to the fusion index threshold, the sample data tuple is written into the first fusion risk experience pool. When the fusion index of the state-action pair at the current sampling time is greater than the fusion index threshold, the sample data tuple is written into the second fusion risk experience pool.

[0322] In one embodiment of this application, the model update module is further configured to:

[0323] According to the preset period, a preset number of sample data tuples are extracted from the first fusion risk experience pool and the second fusion risk experience pool respectively, based on the fusion index.

[0324] For each sample data tuple, the immediate reward is corrected according to a preset reward function to obtain a corrected immediate reward that includes task reward, risk penalty, and uncertainty penalty. The state vector at the next sampling time is input into a preset target action network to obtain candidate actions at the next sampling time. The state vector and candidate actions at the next sampling time are input into a preset target evaluation network to obtain the evaluation result at the next sampling time. The target action network is a replica model with the same structure as the action prediction model, and the target evaluation network is a replica model with the same structure as the uncertainty decision model.

[0325] Based on the final action and the state vector, determine the uncertainty metric of the sample data tuple;

[0326] The model parameters of the action prediction model and the uncertainty decision model are updated based on the final action, state vector, corrected immediate reward, uncertainty metric, and evaluation results at the next sampling time.

[0327] In one embodiment of this application, the model update module is further configured to:

[0328] Based on the corrected immediate reward and the evaluation results at the next sampling time, calculate the temporal difference target value at the next sampling time;

[0329] The parameters of the uncertainty decision model are updated with the goal of minimizing the error between the uncertainty metric and the time-series difference objective value.

[0330] The parameters of the action prediction model are updated with the goal of maximizing the uncertainty metric.

[0331] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an autonomous driving reinforcement learning decision-making method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0332] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0333] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0334] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0335] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0336] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0337] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0338] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0339] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A reinforcement learning decision-making method for autonomous driving, characterized in that, The method includes: The total risk is determined based on the state vector. The state vector is input into the action prediction model trained based on deep reinforcement learning to obtain the candidate action corresponding to the state vector. The state vector is constructed based on the collected environmental information and is used to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources. An uncertainty metric is obtained by inputting a state-action pair into an uncertainty decision-making model trained based on deep reinforcement learning. The state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output of the action prediction model. A fusion index is calculated based on the total risk and the uncertainty metric, and the fusion index is used to characterize the unreliability of the output of the action prediction model and the scenario risk. The target area where the state action pair is located is determined based on the fusion index, and the final action is determined based on the target area and the candidate actions. The final action is used to control the vehicle driving state.

2. The method as described in claim 1, characterized in that, The total risk is determined based on the state vector, including: The state vector is input into the Gaussian risk potential function of each risk source to obtain the potential field of each risk source. The Gaussian risk potential function of each risk source is constructed based on the environmental information. The total risk is obtained by superimposing the potential fields of each of the aforementioned risk sources.

3. The method as described in claim 1, characterized in that, The uncertainty decision-making model is constructed by a parallel evaluation network. The uncertainty metric obtained by taking state-action pairs as input and training the uncertainty decision-making model based on deep reinforcement learning includes: The state-action pair is input into the evaluation network, and the uncertainty assessment results of each evaluation network are obtained. The uncertainty metric is calculated based on each uncertainty assessment result.

4. The method as described in claim 1, characterized in that, Determining the target region where the state-action pair is located based on the fusion index includes: When the fusion index of the state-action pair is less than or equal to the first fusion index threshold, the target area of ​​the state-action pair is determined to be a safe area. When the fusion index of the state-action pair is greater than the first fusion index threshold and less than or equal to the second fusion index threshold, the target region of the state-action pair is determined to be a caution region. When the fusion index of the state-action pair is greater than the second fusion index threshold, the target area of ​​the state-action pair is determined to be a danger zone.

5. The method as described in claim 4, characterized in that, Determining the final action based on the target region and the candidate actions includes: A set of safety actions is determined based on the state vector, the set of safety actions includes at least one safety action, and the safety action is located within a safety distance and safety boundary determined based on the state vector; If the target area is a safe zone, and the candidate action is included in the set of safe actions, then the candidate action is determined as the final action; or if the candidate action is not included in the set of safe actions, then the candidate action is projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action. If the target area is a caution zone, a reference action is selected from the set of safety actions, and the candidate action and the reference action are weighted and fused. The weighted fusion result is then output as the final action. If the target area is located in a danger zone, the candidate actions are projected onto the set of safe actions, and the data in the set of safe actions that is closest to the projection is output as the final action.

6. The method as described in claim 1, characterized in that, The method further includes: Based on the final action and the state vector, determine the state-action pair at the current sampling time, and calculate the fusion index of the state-action pair at the current sampling time based on the state-action pair at the current sampling time. After controlling the vehicle's driving state according to the final action, the state vector at the next sampling time is obtained. Based on the state vector at the next sampling time, the instant reward of the state-action pair at the current sampling time is determined. The instant reward is used to construct the task reward function to iteratively update the vehicle's control strategy based on the task reward function. Based on the final action, the state vector, the fusion index of the state-action pair at the current sampling time, the state vector at the next sampling time, and the immediate reward of the state-action pair at the current sampling time, construct a sample data tuple; When the fusion index of the state-action pair at the current sampling time is less than or equal to the fusion index threshold, the sample data tuple is written into the first fusion risk experience pool; when the fusion index of the state-action pair at the current sampling time is greater than the fusion index threshold, the sample data tuple is written into the second fusion risk experience pool.

7. The method as described in claim 6, characterized in that, Also includes: According to the preset period, a preset number of sample data tuples are extracted from the first fusion risk experience pool and the second fusion risk experience pool respectively, based on the fusion index. For each sample data tuple, the immediate reward is corrected according to a preset reward function to obtain a corrected immediate reward that includes task reward, risk penalty, and uncertainty penalty. The state vector at the next sampling time is input into a preset target action network to obtain candidate actions at the next sampling time. The state vector at the next sampling time and the candidate actions at the next sampling time are input into a preset target evaluation network to obtain the evaluation result at the next sampling time. The target action network is a replica model with the same structure as the action prediction model, and the target evaluation network is a replica model with the same structure as the uncertainty decision model. Based on the final action and the state vector, determine the uncertainty metric of the sample data tuple; The model parameters of the action prediction model and the uncertainty decision model are updated based on the final action, the state vector, the corrected immediate reward, the uncertainty metric, and the evaluation result at the next sampling time.

8. The method as described in claim 7, characterized in that, Based on the corrected immediate reward, the uncertainty metric, and the evaluation result at the next sampling time, update the model parameters of the action prediction model and the uncertainty decision model, including: Based on the corrected immediate reward and the evaluation result at the next sampling time, calculate the temporal difference target value at the next sampling time; The parameters of the uncertainty decision model are updated with the goal of minimizing the error between the uncertainty metric and the time-series difference target value. The parameters of the action prediction model are updated with the goal of maximizing the uncertainty metric.

9. An autonomous driving reinforcement learning decision-making device, characterized in that, The device includes: The candidate action determination module is used to determine the total risk based on the state vector. The state vector is input into the action prediction model trained based on deep reinforcement learning to obtain the candidate action corresponding to the state vector. The state vector is constructed based on the collected environmental information to represent the vehicle state. The total risk is a potential field, which is obtained by superimposing the potential fields of multiple risk sources. An uncertainty determination module is used to input a state-action pair into an uncertainty decision model trained based on deep reinforcement learning to obtain an uncertainty metric. The state-action pair consists of a state vector and a candidate action corresponding to the state vector. The uncertainty metric is used to characterize the unreliability of the output of the action prediction model. The fusion index determination module is used to calculate the fusion index based on the total risk and the uncertainty metric, wherein the fusion index is used to characterize the unreliability of the output results of the action prediction model and the scenario risk; The final action determination module is used to determine the target area where the state action pair is located based on the fusion index, and to determine the final action based on the target area and the candidate actions. The final action is used to control the vehicle driving state.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images