Autonomous driving decision-making method fusing action-level risk prediction and safety lookahead constraints

Abstract

The application relates to an automatic driving decision-making method fusing action-level risk prediction and safety look-ahead constraint, and relates to the technical field of automatic driving. The application acquires a current traffic state and constructs a candidate action set; a state-action joint coding network jointly encodes and extracts state-action joint features of the traffic state and the candidate action; an action-level risk prediction model obtains the risk of the current action in the future according to the state-action joint features; a reinforcement learning strategy network reuses traffic state features provided by a state encoder and gives strategy probabilities of the candidate actions according to the traffic state features; the risk prediction model and the reinforcement learning strategy network are jointly scored in terms of risk strategy, and a final action is selected according to the joint score and safety look-ahead constraint. The application jointly uses the risk prediction result and the reinforcement learning strategy output for action scoring; and further introduces look-ahead safety verification before the action output, thereby improving the decision safety in a left-turn scene at a signal-free intersection.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method that integrates action-level risk prediction and safety forward-looking constraints. Background Technology

[0002] In left-turn scenarios at unsignalized intersections, autonomous vehicles lack traffic light protection and need to dynamically interact with oncoming straight-going vehicles, oncoming right-turning vehicles, and vehicles moving forward at the same speed. If a vehicle turns left too early, it is prone to collisions with oncoming vehicles; if it waits too long, it will lead to decreased traffic efficiency or exceeding the time limit.

[0003] Existing reinforcement learning decision-making methods can acquire driving strategies through trial and error, but their safety mainly relies on indirect constraints from reward functions, lacking explicit judgment of the future collision risk for each candidate action. In complex interaction scenarios, the policy network may output actions with high short-term gains but high collision risks. Therefore, how to introduce action-level risk prediction and hard safety look-ahead constraints into the reinforcement learning decision-making process is a key issue in improving the safety of autonomous driving intersection decisions. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this invention provides an autonomous driving decision-making method that integrates action-level risk prediction and safety forward-looking constraints.

[0005] In a first aspect, the present invention provides an autonomous driving decision-making method that integrates action-level risk prediction and safety forward-looking constraints, comprising: Obtain the current traffic status and construct a set of candidate actions; A state-action joint encoding network is constructed to jointly encode traffic states and candidate actions, and extract state-action joint features; A supervised learning model is used to train an action-level risk prediction model. The action-level risk prediction model takes the joint features of state and action as input and outputs the probability that the current action will cause a collision in the short time domain in the future. The reinforcement learning policy network reuses the traffic state features provided by the state encoder of the action-level risk prediction model, and gives the policy probability of candidate actions based on the traffic state features. The risk prediction model and the reinforcement learning policy network are used to jointly score the risk policy, and the final action is selected based on the joint score and the safety look-ahead constraint.

[0006] Furthermore, the state-action joint coding network includes a state encoder and an action encoder. The state encoder encodes traffic states to obtain traffic state codes, and the action encoder encodes actions to obtain action codes. Finally, the encoding results of the two are concatenated to obtain joint state-action features. The entire process is represented as follows: , in, For state encoders, For motion encoders, For feature splicing operations, the state encoder processes the environmental vehicle state through masking and multilayer perceptron during the traffic state encoding process, in order to mask invalid vehicles and extract the characteristics of valid vehicles as interactive features.

[0007] Furthermore, the state encoder includes a vehicle state encoding branch and an environmental vehicle state encoding branch. The vehicle state encoding branch encodes the vehicle state to obtain vehicle state features. For example, the vehicle state encoding branch uses a multilayer perceptron. The encoding process performed by the environmental vehicle state encoding branch includes: encoding the states of all environmental vehicles to obtain environmental vehicle features; determining the effective environmental vehicles that influence the vehicle's driving decisions based on the relative positions of the vehicle and environmental vehicles, designing corresponding masks, processing the environmental vehicle features using the masks, and retaining the states of the effective environmental vehicles; then processing the masked environmental vehicle features using a multilayer perceptron, aggregating the features of each effective environmental vehicle through mean pooling to obtain the final environmental vehicle features, and concatenating the final environmental vehicle features with the vehicle state features to obtain the traffic state code.

[0008] Furthermore, the process of generating a learnable attention mask includes the following steps: For each environmental vehicle, calculate its interaction feature vector with the driver vehicle. These interaction features include, but are not limited to: the relative position, relative speed, relative heading angle, estimated collision time, and minimum safe distance between the driver and environmental vehicles. A multilayer perceptron is used to perform a nonlinear mapping on these interaction features to obtain a high-dimensional interaction feature representation. The interaction feature vector is then input into an attention weight network, where learnable parameters are used to automatically calculate the attention weights for the corresponding vehicles. ,in, The sigmoid activation function is used, and W, w, b1, and b2 are network parameters learned through training; the attention weights are... It reflects the degree of influence of the i-th environmental vehicle on the current driving decision of the vehicle; the environmental vehicle encoder extracts features of the state of each environmental vehicle to obtain environmental vehicle features, and then uses the attention weight to perform weighted mean aggregation of the environmental vehicle features; in order to capture different types of interaction relationships in traffic scenes, a multi-head attention mechanism is adopted, and multiple independent attention heads are set up, each attention head learning the interaction patterns of different subspaces.

[0009] Furthermore, pre-set spatial or spatiotemporal filtering rules for forming the mask include: marking an environmental vehicle as a valid vehicle when the longitudinal distance between the vehicle and the surrounding vehicle is less than a preset threshold, the lateral distance is less than the width of the adjacent lane, or the expected collision time between the two vehicles is less than a safety threshold.

[0010] Furthermore, the action-level risk prediction model is implemented based on a risk prediction network, which includes a backbone network and a total risk prediction head, a collision risk prediction head, and a lane risk prediction head connected to the backbone network, respectively predicting the total risk, collision risk, and lane departure risk. The risk prediction network's overall risk prediction head, collision risk prediction head, and lane risk prediction head decompose and model the risk sources, predicting the overall collision risk, lane departure risk, and vehicle-to-vehicle collision risk respectively. ; ; in, This represents the vehicle-to-vehicle collision probability output by the risk prediction model. This indicates whether a car-to-car collision will occur within K steps from the current time t, with a value of 1 indicating that a collision has occurred. This represents the lane departure collision probability output by the risk prediction model. This indicates whether a lane-crossing collision will occur within K steps from the current time t, with a value of 1 indicating that a collision has occurred. The total risk prediction head integrates lane departure risk and vehicle-to-vehicle collision risk to obtain the total risk: , When any sub-risk increases, the total risk increases simultaneously.

[0011] Furthermore, the reinforcement learning-based decision network outputs Gaussian distribution parameters for longitudinal acceleration and rotation angle, decomposing the candidate action probabilities into two independent distributions, as follows:

[0012] The actor network provided information on traffic conditions. Longitudinal acceleration is used below The probability, The actor network provided information on traffic conditions. The following uses longitudinal acceleration The probability of; Convert the logarithm of the probability to a logarithmic probability: .

[0013] Furthermore, by combining strategy preferences and risk penalties, a joint score is calculated for each candidate action: , in, For the joint scoring of candidate actions, For the strategy item weights, This is the risk penalty coefficient; Select the action with the highest combined score from the safe candidate actions: ; If the predicted risk of the highest-scoring action is less than the set risk threshold, then the highest-scoring action is directly output; if the predicted risk of the highest-scoring action is higher than the risk threshold, then the top-m candidate actions are traversed from high to low according to the joint score, and the top-m candidate actions are processed in the environment replica. Step-by-step safety look-ahead verification, output the first action that satisfies the safety look-ahead verification. If none of the Top-m candidate actions pass the verification, then execute the conservative backoff action.

[0014] Furthermore, during the training process, the trigger frequency of various risks during reinforcement learning training is statistically analyzed; the simulation environment dynamically adjusts the probability of scene generation, increases the proportion of scenes with poor reinforcement learning performance, and allows the reinforcement learning strategy to be trained on weaknesses, accelerating the recognition of risk sources.

[0015] Secondly, the present invention provides an autonomous driving decision-making system that integrates action-level risk prediction and safety look-ahead constraints, comprising: multiple interconnected network devices, each network device including: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit storing a computer program, and the processing unit implementing the autonomous driving decision-making method that integrates action-level risk prediction and safety look-ahead constraints by running the computer program stored in the storage unit.

[0016] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements the autonomous driving decision-making method that integrates action-level risk prediction and safety forward-looking constraints.

[0017] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art: This application combines reinforcement learning (policy network) and risk prediction mechanisms to ensure the safety and rationality of driving. The network employs a dual-path structure, processing policy and risk separately. Based on decision and risk, and combined with safety prospective analysis, action decisions are made, transforming collision risk from a post-event reward / penalty mechanism into a pre-execution action-level probability prediction. A risk decomposition model distinguishes between lane risk and vehicle collision risk, improving model interpretability. A joint scoring mechanism simultaneously considers policy benefits and action risks. Top-m action prospective validation forms hard safety constraints, effectively reducing collision risks in complex interaction scenarios. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an autonomous driving decision-making method that integrates action-level risk prediction and safety look-ahead constraints, provided as an embodiment of the present invention; Figure 2 This invention provides a framework for an autonomous driving decision-making method that integrates action-level risk prediction and safety forward-looking constraints. Figure 3 A schematic diagram of a state-action joint coding network provided in an embodiment of the present invention; Figure 4 A schematic diagram of a state encoder provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an autonomous driving decision-making device that integrates action-level risk prediction and safety forward-looking constraints, provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] Example 1 like Figure 1 and Figure 2 As shown, this invention proposes an autonomous driving decision-making method that integrates action-level risk prediction and safety look-ahead constraints. This invention models risk as the probability of future collisions under state-action conditions and uses the risk prediction results and reinforcement learning policy output together for action scoring. Furthermore, a short-time look-ahead verification mechanism is introduced before action output to suppress high-risk actions and improve decision-making safety in unsignalized intersection left-turn scenarios. Specifically, the technical solution of this invention includes the following steps: Step 1: Obtain the current traffic status and construct a set of candidate actions.

[0024] The current traffic status consists of the vehicle status and the environmental vehicle status. Real-time acquisition of vehicle status information and all environmental vehicle status information is used for decision-making in the current step. The content of vehicle status information and environmental vehicle status information includes the position, speed and heading angle of each vehicle at each time. , in, Indicates the current traffic status. This indicates the vehicle's position, speed, and heading angle, among other things. It indicates the environmental vehicle position, speed, and heading angle.

[0025] The candidate motion consists of longitudinal acceleration and front wheel steering angle, and is represented as: , in, Indicates candidate control actions, For the longitudinal acceleration of the vehicle, The front wheel steering angle of the vehicle. This application decomposes the steering action of the vehicle into longitudinal acceleration and the front wheel steering angle that controls the vehicle's steering trajectory.

[0026] Step 2: Construct a state-action joint encoding network to jointly encode traffic states and candidate actions, extracting state-action joint features so that subsequent models can perform action-level risk prediction based on these joint features. Specifically, the vehicle state, the environmental vehicle state, and candidate actions are input into the state-action joint encoding network to obtain state-action joint features.

[0027] In the specific implementation process, such as Figure 3 As shown, the state-action joint coding network includes a state encoder and an action encoder. The state encoder encodes traffic states to obtain traffic state codes, and the action encoder encodes actions to obtain action codes. Finally, the encoding results of the two are concatenated to obtain joint state-action features. The entire process is represented as follows: , in, For state encoders, For motion encoders, This is a feature concatenation operation. In the traffic state encoding process, the state encoder processes the environmental vehicle state through masking and a multilayer perceptron to filter out invalid vehicles and extract the characteristics of valid vehicles as interactive features.

[0028] Furthermore, such as Figure 4 As shown, the state encoder includes a vehicle state encoding branch and an environmental vehicle state encoding branch. The vehicle state encoding branch encodes the vehicle state to obtain vehicle state features; for example, the vehicle state encoding branch uses a multilayer perceptron. The environmental vehicle state encoding branch performs the encoding process of encoding all environmental vehicle states to obtain environmental vehicle features; during implementation, all encoded environmental vehicle states are unified to the vehicle coordinate system and adopted in the form of relative vehicle states.

[0029] Next, based on the relative positions of the vehicle and the surrounding vehicles, the effective environmental vehicles that influence the vehicle's driving decisions are determined, and corresponding masks are designed. The environmental vehicle features are processed using the masks, while preserving the states of the effective environmental vehicles. Then, the masked environmental vehicle features are processed by a multilayer perceptron, and the features of each effective environmental vehicle are aggregated by mean pooling to obtain the final environmental vehicle features. The final environmental vehicle features and the vehicle's state features are then concatenated to obtain the traffic state code.

[0030] In practice, spatial or spatiotemporal filtering rules for forming a mask are pre-defined. When the longitudinal distance between the vehicle and the surrounding vehicles is less than a preset threshold (e.g., 50 meters), and the lateral distance is less than the width of the adjacent lane, or the expected collision time between the two vehicles is less than a safety threshold, the vehicle is marked as a valid vehicle.

[0031] In the environmental vehicle state encoding branch, to further enhance the model's ability to perceive key environmental vehicles and adaptively learn the importance weights of each environmental vehicle under different traffic scenarios, this invention introduces a learnable attention mask mechanism to replace the traditional rule-based hard mask. Specifically, the generation process of the learnable attention mask includes the following steps: For each ambient vehicle, calculate its interaction feature vector with the driver vehicle. These interaction features include, but are not limited to: the relative position, relative speed, relative heading angle, estimated collision time, and minimum safe distance between the driver and ambient vehicles. A high-dimensional interaction feature representation is obtained by nonlinearly mapping these interaction features using a multilayer perceptron. The interaction feature vector is then input into an attention weight network, where the attention weights for the corresponding vehicles are automatically calculated using learnable parameters. ,in, The sigmoid activation function is used, and W, w, b1, and b2 are network parameters learned through training. The attention weights... This reflects the influence of the i-th environmental vehicle on the driver's current driving decision; a larger weight value indicates a more critical environmental vehicle. Environmental vehicle features are obtained by extracting features from the state of each environmental vehicle using an environmental vehicle encoder. Then, the environmental vehicle features are weighted and aggregated using the attention weights. Based on mean pooling, an attention mechanism or graph neural network is introduced to further implement weighted mean pooling, thereby highlighting the features of critical vehicles that pose the greatest threat to the driver. The model can adaptively focus on critical vehicles that significantly influence the driver's decision (such as closely following vehicles or vehicles cutting in), while suppressing interference from distant or irrelevant vehicles, thus improving the discriminative power of state encoding and the accuracy of risk prediction. Optionally, to capture different types of interaction relationships in traffic scenarios, this invention further employs a multi-head attention mechanism, setting up multiple independent attention heads. Each attention head learns the interaction patterns of different subspaces, and finally, the outputs of multiple heads are concatenated or weighted summed to obtain richer aggregated environmental vehicle features.

[0032] Step 3: Train the action-level risk prediction model using supervised learning. The action-level risk prediction model takes the joint state-action features as input and outputs the probability that the current action will cause a collision in the short time domain in the future.

[0033] In specific implementation, the action-level risk prediction model is based on a risk prediction network, which includes a backbone network and a total risk prediction head, a collision risk prediction head, and a lane risk prediction head connected to the backbone network, predicting the total risk, collision risk, and lane departure risk, respectively. The risk prediction network models the relationship between state-action joint features and risks.

[0034] The risk prediction network uses its overall risk prediction head, collision risk prediction head, and lane risk prediction head to decompose and model the risk sources, predicting the overall collision risk, lane departure risk, and vehicle-to-vehicle collision risk, respectively.

[0035] ; ; in, This represents the vehicle-to-vehicle collision probability output by the risk prediction model. This indicates whether a car-to-car collision will occur within K steps from the current time t. A value of 1 indicates that a collision has occurred. This represents the lane departure collision probability output by the risk prediction model. This indicates whether a lane-crossing collision will occur within K steps from the current time t, with a value of 1 indicating a collision has occurred. Furthermore, the risk prediction network first outputs a vehicle-to-vehicle collision risk score. Lane crossing collision risk score The risk score is then mapped to a probability using the Sigmoid function: , , in, This represents the Sigmoid function. This represents the parameters of the risk prediction model.

[0036] The total risk prediction head integrates lane departure risk and vehicle-to-vehicle collision risk to obtain the total risk: , When any sub-risk increases, the total risk increases simultaneously. Risk decomposition modeling facilitates the explanation of risk sources.

[0037] Step 4: The reinforcement learning policy network reuses the traffic state features provided by the state encoder and gives the policy probability of candidate actions based on the traffic state features.

[0038] The reinforcement learning strategy employs an actor network, whose outputs Gaussian distribution parameters (mean μ and variance σ) for longitudinal acceleration and rotation angle. This decomposes the candidate action probabilities into two independent distributions, significantly simplifying the dimensionality of the network output, reducing training difficulty, and simultaneously ensuring the continuity of the action space. This can be represented as: , The actor network provided information on traffic conditions. The following uses longitudinal acceleration The probability, The actor network provided information on traffic conditions. The following uses longitudinal acceleration The probability of.

[0039] Convert the logarithm of the probability to a logarithmic probability: .

[0040] Step 5: Perform joint risk policy scoring on the risk prediction model and the reinforcement learning policy network, and select the final action based on the joint score and safety look-ahead constraints.

[0041] Based on the risk prediction model and the decision probabilities provided by the reinforcement learning policy network, a joint score is calculated for each candidate action by combining policy preferences and risk penalties. , in, For the joint scoring of candidate actions, For the strategy item weights, The risk penalty coefficient is used; the joint score preserves the decision preferences of the reinforcement learning strategy while introducing risk assessment from the risk prediction network.

[0042] Select the action with the highest combined score from the safe candidate actions: , If the predicted risk of the highest-scoring action is less than a set risk threshold, then the highest-scoring action is directly output. If the predicted risk of the highest-scoring action is higher than the risk threshold, then the top-m candidate actions are traversed from highest to lowest according to the joint score, and the top-m candidate actions are processed in the environment replica. First, perform a safety look-ahead verification and output the first action that satisfies the verification. In practice, the top-m candidate actions are processed in a replica of the environment. The safety look-ahead verification step is performed. The safety look-ahead verification function is defined as follows: if no collision occurs in the previous Kv steps, the safety look-ahead verification function value of the candidate action in the current traffic state is 1; otherwise, the value is 0. The first action that satisfies the safety look-ahead verification function value of 1 is output. If none of the Top-m candidate actions pass the verification, a conservative backoff action is executed.

[0043] The final action is converted into the vehicle's longitudinal acceleration and front wheel steering angle, which are used to make a safe left turn decision for the autonomous vehicle.

[0044] The decision-making and planning algorithm framework for the autonomous driving system provided in this application combines reinforcement learning (policy network) and risk prediction mechanisms to ensure the safety and rationality of driving. The network in this application adopts a dual-path network structure, processing policy and risk separately. Based on the decision and risk, and combined with safety prospective analysis, action decisions are made, transforming collision risk from a post-event reward / penalty mechanism into a pre-execution action-level probability prediction. A risk decomposition model distinguishes between lane risk and vehicle collision risk, improving model interpretability. A joint scoring mechanism simultaneously considers policy benefits and action risks. Top-m action prospective verification forms hard safety constraints, effectively reducing collision risks in complex interaction scenarios.

[0045] The baseline policy network, based on reinforcement learning, is responsible for policy generation. The state encoder first extracts features from the traffic state. The baseline policy network (actor network) receives the encoded state and outputs the probability distribution of actions.

[0046] The risk prediction model uses a state-action joint encoding network to jointly encode traffic states and candidate actions. Three specific prediction heads are used for fine-grained risk assessment: a collision risk prediction head specifically predicts the risk of collisions with other vehicles, a lane risk prediction head predicts lane departure or lane-related risks, and a total risk prediction head integrates the risks of vehicle collisions and vehicle-related risks to obtain the total risk.

[0047] The decision fusion layer conducts joint scoring of strategy risks: , First item From the baseline policy network, representing the original expected value of the action. The second term... Derived from the risk prediction model, this represents risk penalties (the greater the risk, the more points are deducted). While pursuing optimal strategy, high-risk actions must be penalized.

[0048] It also introduces safety controls and look-ahead mechanisms to handle complex or high-risk situations: it determines whether the predicted risk of a candidate action is below a certain risk threshold; if the risk of a candidate action is too high, it initiates a safety look-ahead mechanism. The top m actions with the highest joint scores are selected for further analysis, performing Kv-step look-ahead predictions. This involves not only looking at the current step but also extrapolating Kv steps forward to assess the long-term consequences of the actions.

[0049] This application does not blindly trust policy networks, but instead introduces a specialized risk prediction model to participate in decision-making. This risk prediction model can break down risks into collision and lane risks, making it more accurate. A safety-forward approach is designed to find safer strategies through multi-step predictions in high-risk situations, preventing accidents caused by short-sighted behavior.

[0050] In practical applications, reinforcement learning policy networks often lack intuition about danger due to sparse rewards or blind exploration, i.e., weak cognition; while risk prediction networks, if trained solely on human driving data (supervised learning), lack awareness of extreme danger edge scenarios (long-tail scenarios). Traditional RL rewards are often sparse (e.g., +100 for reaching the finish line, -1000 for collision). RL struggles to attribute the final collision penalty to specific actions taken a few steps prior. This application injects the output of the risk prediction head as a dense penalty into the reward of each step of the reinforcement learning process. The actor network receives feedback about the specific source of risk at each step. If the actor network's decision attempts to cross the line, even without a collision, The penalty will also immediately correct the policy through backpropagation. When the collision risk exceeds the limit, the algorithm will forcibly reduce the policy's exploration space, forcing reinforcement learning to prioritize learning how to avoid obstacles. This invention utilizes risk sources to enhance reinforcement learning training. When the reinforcement learning policy has a weak awareness of risk, it uses fine-grained risk sources provided by the action-level risk prediction model to provide dense gradient guidance and safety constraints for reinforcement learning.

[0051] This application supports risk-driven, targeted, and precise learning by statistically analyzing the trigger frequency of various risks during reinforcement learning training. If reinforcement learning performs well in lane risk but poorly in collision risk, the simulation environment dynamically adjusts the scenario generation probability, increasing the proportion of high-collision-risk scenarios. This allows the reinforcement learning strategy to be trained on its weaknesses, accelerating its recognition of specific risk sources.

[0052] Reinforcement learning can enhance the training of risk prediction models. Leveraging the exploratory capabilities of RL, high-quality edge case data can be generated to feed back into the risk prediction model. When a risk model is trained on normal human driving data, its predictions become inaccurate when encountering "dangerous actions" explored by RL. In the early stages of RL training, actors will perform many dangerous actions. Collecting these trajectories generated by RL exploration, especially those high-risk trajectories that ultimately lead to simulated collisions, and adding them to the training set of the risk prediction network for fine-tuning, allows for the use of reinforcement learning's trial-and-error capabilities to supplement the risk prediction model with accident / edge case samples, improving its prediction accuracy in edge situations.

[0053] In terms of architecture design, the "state encoding network" of RL and the "state-action encoding network" of the risk network share the underlying traffic feature extraction layer. The gradients generated by RL in optimizing long-term cumulative rewards are fed back to the shared layer, helping the risk model extract features that are more relevant to decision-making and have physical dynamic significance, rather than simply fitting surface pixels or statistical features in human data.

[0054] These two processes alternate or run in conjunction during training, forming a double-helix upward closed loop: The risk model defines the safety boundaries for reinforcement learning: through fine-grained penalties and constraints, it prevents reinforcement learning from learning badly or causing catastrophic collisions in the early stages of training, accelerating the convergence of reinforcement learning to a safe policy. Reinforcement learning explores unknown boundaries for the risk model: through continuous trial and error and adversarial exploration, it breaks the distribution limitations of human driving data, providing the risk model with long-tail hazardous scenario data, making its predictions increasingly accurate. Ultimately, as training progresses, the reinforcement learning policy itself gradually enhances its awareness of risk (internalizing risk constraints), and the risk prediction network also becomes extremely sensitive; both together support the final safety control action output in the architecture diagram.

[0055] Example 2 like Figure 5As shown, this embodiment of the invention provides an autonomous driving decision-making device that integrates action-level risk prediction and safety look-ahead constraints, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit serving as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to the autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints in this embodiment of the invention. The processing unit implements the aforementioned autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints by running the software program, computer-executable program, and module stored in the storage unit, including: Obtain the current traffic status and construct a set of candidate actions; A state-action joint encoding network is constructed to jointly encode traffic states and candidate actions, and extract state-action joint features; A supervised learning model is used to train an action-level risk prediction model. The action-level risk prediction model takes the joint features of state and action as input and outputs the probability that the current action will cause a collision in the short time domain in the future. The reinforcement learning policy network reuses the traffic state features provided by the state encoder of the action-level risk prediction model, and gives the policy probability of candidate actions based on the traffic state features. The risk prediction model and the reinforcement learning policy network are used to jointly score the risk policy, and the final action is selected based on the joint score and the safety look-ahead constraint.

[0056] Of course, the computer program stored in the storage unit of the autonomous driving decision-making system that integrates action-level risk prediction and safety look-ahead constraints provided in the embodiments of the present invention is not limited to the method operation described above, but can also execute related operations in the autonomous driving decision-making method that integrates action-level risk prediction and safety look-ahead constraints provided in any embodiment of the present invention.

[0057] Example 3 This invention provides a computer-readable storage medium storing a computer program. When executed, the computer program implements the autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints, comprising: Obtain the current traffic status and construct a set of candidate actions; A state-action joint encoding network is constructed to jointly encode traffic states and candidate actions, and extract state-action joint features; A supervised learning model is used to train an action-level risk prediction model. The action-level risk prediction model takes the joint features of state and action as input and outputs the probability that the current action will cause a collision in the short time domain in the future. The reinforcement learning policy network reuses the traffic state features provided by the state encoder of the action-level risk prediction model, and gives the policy probability of candidate actions based on the traffic state features. The risk prediction model and the reinforcement learning policy network are used to jointly score the risk policy, and the final action is selected based on the joint score and the safety look-ahead constraint.

[0058] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0059] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0060] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0061] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An autonomous driving decision-making method that integrates action-level risk prediction and safety forward-looking constraints, characterized in that, include: Obtain the current traffic status and construct a set of candidate actions; A state-action joint encoding network is constructed to jointly encode traffic states and candidate actions, and extract state-action joint features; A supervised learning model is used to train an action-level risk prediction model. The action-level risk prediction model takes the joint features of state and action as input and outputs the probability that the current action will cause a collision in the short time domain in the future. The reinforcement learning policy network reuses the traffic state features provided by the state encoder of the action-level risk prediction model, and gives the policy probability of candidate actions based on the traffic state features. The risk prediction model and the reinforcement learning policy network are used to jointly score the risk policy. The final action is selected based on the joint score and the safety look-ahead constraint. The joint score is calculated for each candidate action by combining policy preference and risk penalty. ， in, For the joint scoring of candidate actions, For the strategy item weights, This is the risk penalty coefficient; Select the action with the highest combined score from the safe candidate actions: ； If the predicted risk of the highest-scoring action is less than the set risk threshold, then the highest-scoring action is directly output; if the predicted risk of the highest-scoring action is higher than the risk threshold, then the top-m candidate actions are traversed from high to low according to the joint score, and the top-m candidate actions are processed in the environment replica. Step-by-step safety look-ahead verification, output the first action that satisfies the safety look-ahead verification. If none of the Top-m candidate actions pass the verification, then execute the conservative backoff action.

2. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 1, characterized in that, The state-action joint coding network includes a state encoder and an action encoder. The state encoder encodes traffic states to obtain traffic state codes, and the action encoder encodes actions to obtain action codes. Finally, the encoding results of the two are concatenated to obtain joint state-action features. The entire process is represented as follows: , in, For state encoders, For motion encoders, For feature splicing operations, the state encoder processes the environmental vehicle state through masking and multilayer perceptron during the traffic state encoding process, in order to mask invalid vehicles and extract the characteristics of valid vehicles as interactive features.

3. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 2, characterized in that, The state encoder includes a vehicle state encoding branch and an environmental vehicle state encoding branch. The vehicle state encoding branch encodes the vehicle state to obtain vehicle state features. The vehicle state encoding branch uses a multilayer perceptron. The encoding process performed by the environmental vehicle state encoding branch includes: encoding the states of all environmental vehicles to obtain environmental vehicle features; determining the effective environmental vehicles that influence the vehicle's driving decisions based on the relative positions of the vehicle and environmental vehicles, designing corresponding masks, processing the environmental vehicle features using the masks, and retaining the states of the effective environmental vehicles; then processing the masked environmental vehicle features using a multilayer perceptron, aggregating the features of each effective environmental vehicle through mean pooling to obtain the final environmental vehicle features, and concatenating the final environmental vehicle features with the vehicle state features to obtain the traffic state code.

4. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 3, characterized in that, The process of generating a learnable attention mask includes the following steps: For each environmental vehicle, calculate its interaction feature vector with the driver vehicle. These interaction features include, but are not limited to: the relative position, relative speed, relative heading angle, estimated collision time, and minimum safe distance between the driver and environmental vehicles. A multilayer perceptron is used to perform a nonlinear mapping on these interaction features to obtain a high-dimensional interaction feature representation. The interaction feature vector is then input into an attention weight network, where learnable parameters are used to automatically calculate the attention weights for the corresponding vehicles. ,in, The sigmoid activation function is used, and W, w, b1, and b2 are network parameters learned through training; the attention weights are... It reflects the degree of influence of the i-th environmental vehicle on the current driving decision of the vehicle; the environmental vehicle encoder extracts features of the state of each environmental vehicle to obtain environmental vehicle features, and then uses the attention weight to perform weighted mean aggregation of the environmental vehicle features; in order to capture different types of interaction relationships in traffic scenes, a multi-head attention mechanism is adopted, and multiple independent attention heads are set up, each attention head learning the interaction patterns of different subspaces.

5. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 3, characterized in that, Pre-defined spatial or spatiotemporal filtering rules for forming a mask include: marking an environmental vehicle as a valid vehicle when the longitudinal distance between the vehicle and the surrounding vehicle is less than a preset threshold and the lateral distance is less than the width of the adjacent lane, or when the expected collision time between the two vehicles is less than a safety threshold.

6. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 1, characterized in that, The action-level risk prediction model is implemented based on a risk prediction network, which includes a backbone network and a total risk prediction head, a collision risk prediction head, and a lane risk prediction head connected to the backbone network, which respectively predict the total risk, collision risk, and lane departure risk. The risk prediction network's overall risk prediction head, collision risk prediction head, and lane risk prediction head decompose and model the risk sources, predicting the overall collision risk, lane departure risk, and vehicle-to-vehicle collision risk respectively. ； ； in, This represents the vehicle-to-vehicle collision probability output by the risk prediction model. This indicates whether a car-to-car collision will occur within K steps from the current time t, with a value of 1 indicating that a collision has occurred. This represents the lane departure collision probability output by the risk prediction model. This indicates whether a lane-crossing collision will occur within K steps from the current time t, with a value of 1 indicating that a collision has occurred. The total risk prediction head integrates lane departure risk and vehicle-to-vehicle collision risk to obtain the total risk: ， When any sub-risk increases, the total risk increases simultaneously.

7. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 1, characterized in that, The basic decision network of reinforcement learning outputs Gaussian distribution parameters for longitudinal acceleration and rotation angle, decomposing the candidate action probability into two independent distributions, as follows: The actor network provided information on traffic conditions. The following uses longitudinal acceleration The probability, The actor network provided information on traffic conditions. Longitudinal acceleration is used below The probability of; Convert the logarithm of the probability to a logarithmic probability: 。 8. The autonomous driving decision-making method integrating action-level risk prediction and safety look-ahead constraints according to claim 1, characterized in that, During training, the trigger frequency of various risks during reinforcement learning training is statistically analyzed; the simulation environment dynamically adjusts the probability of scene generation, increases the proportion of scenes with poor reinforcement learning performance, and allows the reinforcement learning strategy to be trained on weaknesses, thereby accelerating the recognition of risk sources.

9. An autonomous driving decision-making device integrating action-level risk prediction and safety forward-looking constraints, comprising: At least one processing unit, the processing unit being connected to a storage unit via a bus unit, characterized in that the storage unit stores a computer program, and the processing unit implements the autonomous driving decision-making method of any one of claims 1-8 by running the computer program stored in the storage unit.

Images