A safe robust reinforcement learning-based automatic driving decision method for intersections without signal control

Abstract

This invention provides a decision-making method for autonomous driving at intersections without signal control based on robust reinforcement learning. The method includes: before entering an intersection without signal control, collecting state information of the vehicle and other surrounding vehicles; identifying and extracting the states of all potential conflicting vehicles based on dynamic screening rules; inputting the states of the vehicle and all potential conflicting vehicles into a feature extraction network to calculate a high-dimensional feature vector representing the complexity of the environment; inputting the high-dimensional feature vector into a robust reinforcement learning decision-making framework, and generating the optimal control action command for the current moment through mapping calculations in the policy network within the framework; controlling the accelerator pedal opening and brake pedal opening in real time using the optimal control action command to achieve longitudinal motion control of the vehicle; repeating the above steps until the autonomous vehicle safely passes through the intersection without signal control. This invention further improves the reliability and performance of autonomous driving decision-making at intersections without signal control.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method for signalless intersections based on safety robust reinforcement learning. Background Technology

[0002] The core of autonomous driving lies in the coordinated operation of three major modules: perception, decision-making, and control, enabling vehicles to drive autonomously and safely. The decision-making module plays a crucial bridging role, processing environmental information output by the perception module (such as the status of the vehicle and surrounding vehicles, obstacle information, traffic signals, etc.) into reasonable high-level decision-making instructions, which are then executed by the control module. In scenarios involving intersections without traffic lights (hereinafter referred to as uncontrolled intersections), the lack of unified coordination by traffic lights leads to highly dynamic and interactive behaviors of traffic participants such as vehicles, pedestrians, and non-motorized vehicles, posing more stringent technical requirements for autonomous driving decision-making systems. The decision-making system not only needs to accurately perceive and predict the intentions and trajectories of all traffic participants, but also needs to balance traffic efficiency, safety intervals, and energy consumption in multi-objective optimization, and possess robustness to cope with perception uncertainties, preventing decision-making biases caused by calculation errors in the perception module. This makes the decision-making problem at uncontrolled intersections a recognized challenge and hot topic in the field of autonomous driving.

[0003] Deep Reinforcement Learning (DRL) combines the complex function solving capabilities of deep learning with the autonomous exploration capabilities of reinforcement learning. It learns optimal policies through real-time interaction between the agent and the environment, and has been widely applied in the field of autonomous driving decision-making. DRL's advantages lie in its sequential decision-making capabilities and its optimization characteristics geared towards long-term cumulative returns. However, existing DRL-based methods still suffer from problems in practical applications, such as low sample efficiency, poor performance in the early stages of training, and insufficient security of the policies obtained during training. Especially in the high-uncertainty environment of real-world intersections, training failures are easily caused, making it difficult to guarantee the reliability and safety of the policies.

[0004] On the one hand, current deep reinforcement learning-based decision-making methods typically employ relatively simple feature preprocessing methods for input states. Existing technologies often directly concatenate abstract physical states output by the perception module (such as the position, speed, and orientation of the vehicle and surrounding vehicles) or raw sensor data (such as images and point clouds) as the state input to the reinforcement learning network. To improve the quality of feature representation, some solutions introduce dedicated feature extraction networks, such as using CNNs to extract spatial features from bird's-eye view (BEV) images, or combining RNNs to capture temporal motion information to model the trajectory evolution and potential intentions of traffic participants. However, such simple feature extraction networks based on CNNs or RNNs suffer from insufficient representation capabilities, limited generalization performance, and low sensitivity to safety-critical information in complex scenarios like intersections without signalized traffic control, making it difficult to meet the stringent robustness and reliability requirements of practical deployment. Furthermore, the design of these feature extraction networks often lacks systematic theoretical analysis and design motivation, resulting in unclear internal transformation mechanisms. More critically, the training of such feature extraction networks is usually highly coupled with the downstream reinforcement learning policy network, with parameters jointly updated under a unified optimization objective. This coupled training method means that small fluctuations in the feature extractor can significantly alter the representation distance in the state space, thereby interfering with the convergence of the state value function. At the same time, the coupled training process is not interpretable, which increases the difficulty of debugging, fault tracing, and security verification, further hindering the progress of this type of method towards engineering implementation.

[0005] On the other hand, regarding the selection of reinforcement learning algorithms, the Soft Actor-Critic (SAC) algorithm has become one of the mainstream technical routes in complex intersection decision-making tasks due to its superior sampling efficiency and environment generalization ability in handling high-dimensional continuous action spaces. SAC is based on the maximum entropy reinforcement learning framework and, by introducing an adaptive temperature regularization mechanism, maximizes both cumulative reward and policy entropy, thus achieving a good balance between exploration and exploitation. Simultaneously, its value network ensures the accuracy of value assessment by minimizing the time difference error of the Bellman equation. However, the optimization objective of SAC determines that it is essentially a "reward-driven" method, where safety can only be integrated into the reward function as a negative feedback term. This approach leads to a "reward offsetting" phenomenon in multi-objective optimization, where the safety penalty resulting from a high-risk action may be offset by other high-value tasks (such as traffic efficiency rewards), causing the agent to statistically still consider dangerous behavior to have a high overall value. This optimization mechanism, lacking explicit safety constraints, makes it difficult for the learned policy to achieve a robust balance between conservatism and efficiency, and fails to establish an effective safety baseline. Furthermore, the SAC algorithm lacks sufficient robustness to perceived uncertainty. At intersections without signal control, perception modules often face uncertainties such as occlusion, noise, and prediction errors. Although the maximum entropy mechanism of SAC improves its robustness in action space exploration, its value network and policy network themselves lack the ability to resist interference from input distribution shifts. Once the perception information deviates (such as incorrect prediction of the target vehicle's intent or position drift), the policy network is highly susceptible to distribution shifts or incorrect output decisions, thereby violating safety constraints and even causing catastrophic consequences. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] Based on this, the present invention provides a decision-making method for autonomous driving at intersections without signal control based on safety robust reinforcement learning, so as to further improve the reliability and performance of autonomous driving decision-making at intersections without signal control.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, this invention provides a decision-making method for unsignalized intersections based on safety robust reinforcement learning, comprising:

[0010] Step S1: Before the autonomous vehicle enters an intersection without traffic control, it collects traffic status information of itself and other vehicles around it through sensors; the status information of the autonomous vehicle includes: position, orientation angle, speed and acceleration; the status information of other vehicles around it includes: position, orientation angle and speed.

[0011] Step S2: Identify and extract the status information of all potential conflicting vehicles in the vicinity according to the dynamic filtering rules for potential conflicting vehicles;

[0012] Step S3: Input the state information of the vehicle and all potential conflicting vehicles into the feature extraction network to calculate a high-dimensional feature vector representing the complexity of the environment; wherein, the feature extraction network includes a vehicle temporal feature extraction module and a feature fusion module;

[0013] Step S4: Input the high-dimensional feature vector into the safe and robust reinforcement learning decision framework, and generate the optimal control action command at the current time through the mapping calculation of the policy network in the framework;

[0014] The security-robust reinforcement learning decision framework includes an input layer, a historical data buffer, a task performance module, a security performance module, a policy network, and an adversarial network.

[0015] Step S5: Control the accelerator pedal opening and brake pedal opening in real time through optimal control action commands to achieve longitudinal motion control of the vehicle;

[0016] Step S6: Repeat steps S1-S5 until the autonomous vehicle safely passes through the uncontrolled intersection.

[0017] (III) Beneficial Effects

[0018] As can be seen from the above technical solution, the beneficial effects of the autonomous driving decision-making method for signalless intersections based on safety robust reinforcement learning proposed in this invention are as follows:

[0019] 1. This invention constructs a highly interpretable feature extraction network. By combining a vehicle temporal feature extraction module and a feature fusion module, the transmission mechanism of feature vectors between different levels is clarified, enabling the deconstruction of target vehicle dynamics and environmental spatial occupancy features from raw data. This clear information flow gives the output high-dimensional feature vectors strong representational capabilities and allows researchers to adjust them in segments based on experimental results, effectively supporting the training convergence of reinforcement learning networks in complex environments.

[0020] 2. The robust reinforcement learning decision-making framework designed in this invention specifically addresses the pain points of insufficient perception robustness and inadequate security of trained strategies in existing technologies. By introducing adversarial network noise attack training and a long-term security cost constraint mechanism, it significantly enhances the anti-disturbance capability of autonomous driving systems in high-risk, highly interference scenarios such as intersections without signal control. While ensuring efficient decision-making, it effectively reduces the trigger rate of dangerous accidents during the training and deployment phases, demonstrating stronger scenario-specificity. Attached Figure Description

[0021] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0022] Figure 1 This is a schematic diagram illustrating the principle of the signalless intersection autonomous driving decision-making method of the present invention;

[0023] Figure 2 This is a schematic diagram of the vehicle temporal feature extraction module of the present invention;

[0024] Figure 3 This is a schematic diagram of the feature fusion module of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of the secure and robust reinforcement learning decision framework of the present invention;

[0026] Figure 5 This is a schematic diagram of the training process for the secure and robust reinforcement learning decision framework of this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1 As shown, the working principle of the autonomous driving decision-making method for intersections without signal control in this invention is as follows: Before the autonomous vehicle enters the intersection, it collects raw environmental data through onboard sensors, and the perception module analyzes and extracts the spatial position, orientation angle, and speed of surrounding vehicles, as well as the real-time position, speed, and acceleration of the vehicle itself. Subsequently, based on the potential conflict vehicle screening rules, the state information of potential conflict vehicles is identified and extracted, and input into the feature extraction network to calculate a high-dimensional feature vector representing the complexity of the environment. This high-dimensional feature vector is further input into a safe and robust reinforcement learning decision framework, and the optimal control action command for the current moment is generated through the mapping calculation of the policy network. This command will control the accelerator pedal opening and brake pedal opening in real time to achieve longitudinal motion control of the vehicle. For the lateral motion control of the vehicle, it is completed by existing path planning and underlying control. This solution does not limit its specific implementation method, but only serves as an external function to cooperate with this invention, so as to jointly achieve safe passage of the vehicle in intersections without signal control.

[0029] This invention provides a decision-making method for autonomous driving at signalless intersections based on safety robust reinforcement learning, specifically including:

[0030] Step S1: Before the autonomous vehicle enters an intersection without traffic control, it collects traffic status information of itself and other vehicles around it through sensors; the status information of the autonomous vehicle includes: position, orientation angle, speed and acceleration; the status information of other vehicles around it includes: position, orientation angle and speed.

[0031] Step S2: Identify and extract the status information of all potential conflicting vehicles in the vicinity according to the dynamic filtering rules for potential conflicting vehicles;

[0032] Centered on the vehicle, a detection range is defined based on a preset radius. Only traffic participants within this detection range are included in the initial selection set of potential conflict vehicles. If the number of vehicles within the detection range is insufficient (i.e., the number of vehicles is less than...), the system will not detect any potential conflicts. If the number of vehicles within the detection range exceeds a preset maximum vehicle count threshold, then a false stationary vehicle at a fixed preset coordinate at the edge of the detection range will be used as a placeholder to supplement the vehicle count. Centered on the vehicle, the segment is cut off based on spatial proximity (Euclidean distance), retaining... The closest target that has the greatest impact on the vehicle is considered as a potential conflict vehicle.

[0033] The purpose of this step is to balance real-time computation with the semantic integrity of the scene, providing a dimension-constant input for the subsequent feature extraction network. Defining the detection range based on a preset radius is to filter out irrelevant distant traffic participants and reduce computational load. To address the uncertainty of the number of potentially conflicting vehicles within the detection range, placeholders are introduced when the number of vehicles is insufficient; when the number of vehicles exceeds the limit, the nearest targets that have the greatest impact on the vehicle are retained, solving the input adaptation problem of the feature extraction network caused by the dynamic changes in the number of surrounding vehicles.

[0034] Step S3: Input the state information of the vehicle and all potentially conflicting vehicles into the feature extraction network to calculate a high-dimensional feature vector representing the complexity of the environment; wherein, the feature extraction network includes a vehicle temporal feature extraction module (Part A) and a feature fusion module (Part B); specifically including:

[0035] Step S301: Transfer the vehicle's status information Status information of all potential conflict vehicles Organized as potential conflict vehicles Personal information Dynamic information and related information Among them, superscript Indicates from time Time sequences, for example ; Indicates the preset number of past time steps; , , , and These represent the times of the vehicle. of Axis coordinates Axis coordinates, orientation angle, velocity, and acceleration; The number indicating a potential conflict vehicle (i.e., an environmental vehicle); , , , and Representing environmental vehicles At any moment of Axis coordinates Axis coordinates, orientation angle, velocity Axial components and velocity Axial components; , , ; , , Representing environmental vehicles At any moment Other vehicles Related information ( Axis coordinate difference (axis coordinate difference, orientation angle)

[0036] Step S302: Move the environmental vehicle Personal information Dynamic information and related information Inputting the vehicle's temporal feature extraction module, it predicts the features for the next single step. ;

[0037] like Figure 2 As shown, the vehicle temporal feature extraction module is divided into three branches: the state branch, the dynamic branch, and the association branch. In the state branch, the vehicle... Personal information The high-dimensional state feature vector of the vehicle is obtained by inputting the multilayer perceptron (MLP). As shown below:

[0038]

[0039] in, This represents a multilayer perceptron; express The parameters of the MLP network used in the operation;

[0040] In dynamic branch lines, vehicles Dynamic information The high-dimensional dynamic feature vector of the vehicle is obtained by inputting the multilayer perceptron (MLP). Then, a gated recurrent neural network (GRU) is used to capture... Vehicle time-series dynamic characteristics As shown below:

[0041]

[0042]

[0043] in, This represents a multilayer perceptron; express The parameters of the MLP network used in the operation; This represents a gated recurrent neural network; The hidden state represents the vehicle's temporal dynamic characteristics from the previous moment; express The parameters of the GRU network used in the computation;

[0044] In the associated branch roads, environmental vehicles Related information High-dimensional correlation feature vectors are obtained by inputting the multilayer perceptron (MLP). Then, a gated recurrent neural network (GRU) is used to capture... Vehicle time-series dynamic characteristics Next, the cross-attention mechanism is used to obtain information about other vehicles in the target environment. The influence weight, and use this weight to... The target environment vehicle is obtained by performing a weighted summation. Characteristics affected by the environment As shown in the following formula:

[0045]

[0046]

[0047]

[0048]

[0049] in, express The parameters of the MLP network used in the operation; This represents the hidden state of the vehicle regarding the temporal dynamic features at the previous moment; express The parameters of the GRU network used in the computation; This represents the softmax function; Represents the weight parameters of the query matrix in the attention mechanism; Represents the weight parameters of the key matrix in the attention mechanism; Indicates the dimension of the key matrix; Represents matrix dot product; Represents the value matrix;

[0050] Then, the vehicle's time-series dynamic characteristics and target vehicle Characteristics affected by the environment After splicing, dimensionality reduction is achieved through MLP integration. , Then, with the vehicle's high-dimensional state feature vector After concatenation, long-term dynamic features are captured using a Long Short-Term Memory (LSTM) neural network. ; and will Inputting the MLP yields the prediction vector for the next time step. Simultaneously, this MLP branches out a prediction head branch, which then predicts the vector... The parsed reconstruction is a Gaussian distribution prediction vector representing the probability of future vehicle space occupancy. As shown in the following formula:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] in, Indicates splicing; express The parameters of the MLP network used in the operation; Represents a Long Short-Term Memory (LSTM) neural network; superscript Indicates the past at the current time step The time window for the step size; This represents the output of hidden features in long-term memory; This represents the output of hidden features in short-term memory; express and The parameters of the LSTM network used in the operation; This represents the input of hidden features in long-term memory; express The parameters of the MLP network used in the operation; express The parameters of the MLP network used in the operation; Indicates vehicle Location ( coordinate, The future time-time Gaussian distribution mean vector of coordinates and orientation angle; Indicates position ( coordinate, The future time-time variance vector of the coordinates and the orientation angle; Indicates position ( coordinate, The future time-time covariance vector of coordinates and orientation angle;

[0060] Step S303: Roll the predictions of the future vehicles obtained by the vehicle temporal feature extraction module. Feature vectors at each time step The input feature fusion module yields a high-dimensional feature vector used as input to a secure and robust reinforcement learning decision framework. .

[0061] like Figure 3 As shown, at each time step, the vehicle temporal feature extraction module is used to perform forced teaching (teacher forcing) rolling prediction for each environmental vehicle to obtain future results. Prediction vector at each time step Then, the future space occupied by the vehicle is obtained through MLP dimensionality enhancement. As shown in the following formula:

[0062]

[0063] in, Indicates calculation The parameters of the MLP network used.

[0064] Teacher forcing is a technique used to train recurrent neural networks for sequence generation tasks. Its core principle is that during the training phase, the decoder... Instead of using the actual predicted output from the previous time step, the input at each step is directly fed with the true labels from the training set. In this way, each prediction step is based on the "known correct answer," thus avoiding the accumulation of early prediction bias over time and enabling the network to learn the conditional probability distribution of the target sequence more directly.

[0065] Next, the future space occupancy characteristics of each vehicle in the environment are input into a multi-layer attention mechanism; specifically, the vehicle's state information is input first. The vector obtained after feature extraction by MLP Environmental vehicles Personal information The vector obtained after feature extraction by MLP Vehicle weights are obtained by performing scaled dot product attention calculation. Then calculate the vehicle's... and various environmental vehicles Future space occupancy characteristics Spatial weights are obtained by scaling dot product attention calculation. Finally, based on vehicle weights With space occupied weight Calculate each environmental vehicle The future position of the vehicle's influence vector As shown below:

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] in, Indicates calculation The MLP network parameters used; Indicates calculation The MLP network parameters used; The query matrix weight parameters represent the impact of the computational environment on the vehicle. The key matrix weight parameters represent the impact of the computational environment on the vehicle. The dimension of the key matrix represents the impact of the computational environment on the vehicle. The value matrix weight parameters represent the impact of the environmental vehicles on the vehicle. The query matrix weight parameters represent the impact of vehicle-occupied space in the computing environment on the vehicle itself; The key matrix weight parameters represent the impact of vehicle-occupied space in the computational environment on the vehicle itself. The dimension of the key matrix represents the impact of vehicle-occupied space in the computational environment on the vehicle itself; The value matrix weight parameters represent the impact of vehicle-occupied space in the computational environment on the vehicle itself.

[0072] Finally, the vector will be affected The result obtained after MLP With the feature vector of the vehicle The concatenated features are then linearly transformed to map the concatenated feature dimensions to the same dimension as the input layer of the secure and robust reinforcement learning decision framework, resulting in a high-dimensional output feature vector. As shown in the following formula:

[0073]

[0074] in, Indicates calculation The MLP network parameters used; It represents a linear transformation.

[0075] Step S4: Input the high-dimensional feature vector into the safe and robust reinforcement learning decision framework, and generate the optimal control action command at the current time through the mapping calculation of the policy network in the framework;

[0076] like Figure 4 As shown, the secure and robust reinforcement learning decision framework designed in this invention includes an input layer, a historical data buffer, a task performance module, a security performance module, a policy network, and an adversarial network.

[0077] The historical data buffer is used to store tuple data sampled by the reinforcement learning agent in a simulation environment. The data is collected, stored, and shuffled for use by offline policy reinforcement learning networks for training. Indicates state; This indicates interference from noise, i.e. Indicates actions taken after being disturbed by noise; Indicates a reward; Indicate cost; Indicates the next moment, that is Indicates the state at the next moment;

[0078] The task performance evaluation module has the following functions: first, it accurately represents the task's reward by quantifying the expected cumulative reward of actions performed in the current state; second, it effectively suppresses evaluation bias, ensuring that the fitting effect of the value function converges to the true value. This module includes two value networks (i.e., task value networks) and two corresponding target networks (i.e., target value networks), each employing a neural network architecture consisting of two fully connected layers. The use of two task value networks in this module is to alleviate the problem of value "overestimation." The input to the task value network is the high-dimensional feature vector output by the feature extraction network. The output dimension is a one-dimensional value vector, used to measure the magnitude of the action's value. The loss function design for updating the task value network is based on the principle of temporal difference, that is, combining the output of the target value network with the current policy entropy to calculate the temporal difference target value of the task value under the current state-action pair; then, using this temporal difference target value as a benchmark, the task value network loss function is constructed by calculating the temporal difference error between the predicted value and the target value, as shown in the following formula:

[0079]

[0080]

[0081] in, Indicates the number of the task value network; Representing the task value network The loss function; Representing the task value network Weight parameters; Representing the task value network Calculate the expected cumulative reward estimate of the output; This represents a historical data buffer. This represents tuple data sampled from the buffer; Indicates the number of the target value network; This represents the temporal difference target value of the task value network; Represents the reward function; Indicates the reward discount factor; Represents the target value network Weight parameters; Represents the target value network Calculate the expected cumulative reward estimate of the output; Represents the coefficient of the entropy regularization term; Represents policy entropy; This represents the output of the policy network action.

[0082] The safety performance evaluation module has the following functions: first, to quantify the expected multi-step costs after taking actions in the current state; and second, to improve the noise resistance and accuracy of the evaluation process, making the fitted curve of the safety performance evaluation approximate the true cost distribution. This module includes two value networks (i.e., safety cost networks) and their corresponding two target networks (i.e., target cost networks), each employing a neural network architecture consisting of two fully connected layers. The use of two safety cost networks in this module is to mitigate the problem of cost "overestimation." The input to the safety cost network is the high-dimensional feature vector output by the feature extraction network. The output is a one-dimensional safety cost index. The loss function for updating the safety cost network is also based on the temporal difference principle, that is, using the target cost network to calculate the temporal difference target value of the safety cost under the current state-action pair; then, using this temporal difference target value as a benchmark, the safety cost network loss function is constructed by calculating the temporal difference error between the predicted value and the target value, as shown in the following formula:

[0083]

[0084]

[0085] in, Indicates the number of the security cost network; Indicates security cost network The loss function; Indicates security cost network Weight parameters; Indicates security cost network Calculate the expected cumulative reward estimate of the output; This represents tuple data sampled from the history buffer; Indicates the number of the target cost network; This represents the temporal difference objective value of the security cost network; Represents the cost function; Represents the target cost network Weight parameters; Represents the target cost network Calculate the expected cumulative reward estimate of the output.

[0086] The policy network aims to optimize policy performance to achieve a preset task objective. This network consists of two fully connected layers that receive real-time environmental state input and calculate the optimal action to take in the current context. Based on maximizing task performance as the optimization objective, and constrained by safety cost and policy entropy, it utilizes the Lagrange duality method to transform the constrained non-convex optimization problem into a single-objective unconstrained convex optimization problem. The loss function for policy network updates is then constructed using this transformation, as shown in the following equation:

[0087]

[0088] in, The loss function of the policy network is represented; This represents the policy network weight parameters.

[0089] Adversarial networks (ANNs) consist of two fully connected layers, capable of capturing environmental state features in real time and calculating the optimal perturbation term that reduces the agent's decision-making performance. The design of ANNs is based on the robust Markov decision process (R-MDP) theory. Its core logic is to pre-apply the optimal perturbation attack to the decision-making agent in the current state during the training process. By forcing the agent to iterate its policy under extreme constraints, it is forced to learn a robust policy for that state. Depending on the actual algorithm configuration, the ANN can choose its attack method: when the attack target is task performance, the ANN operates in Maximum Reward (MR) mode. Exploiting the agent's profit-seeking instinct, it maliciously guides the agent's observed state by generating perturbation noise aimed at increasing the expected cumulative reward, inducing misjudgments of task value and leading the agent to engage in dangerous profit-seeking behavior that violates safety constraints in pursuit of falsely high returns. When the attack target is security performance, the adversarial network (ANN) operates in Maximum Cost (MC) attack mode. This mode directly disrupts the state representation by inputting noise into the observed state to increase the accumulated security cost over multiple steps. This directly undermines the agent's perception of environmental risks, inducing the agent to directly engage in illegal behavior or collision risks under conditions of decision-making failure. Based on this game theory mechanism, the loss function of the ANN under both attack modes is designed as follows:

[0090]

[0091]

[0092] in, This represents the loss function of the adversarial network under the maximum reward attack mode; The loss function of the adversarial network under the maximum cost attack mode is denoted as . This indicates the noise interference of the adversarial network on the state; This represents the weight parameters of the adversarial network.

[0093] In the security robust reinforcement learning decision framework, all networks (including policy network, adversarial network, task value network, security cost network, objective value network, and objective cost network) use the Adam optimizer, with a learning rate of 0.0001, a buffer size of 100,000, a batch size of 128, and a discount factor of [missing information]. The value is 0.99. All networks consist of two fully connected layers with 64 and 64 neurons respectively. The activation function between the fully connected layers is the ReLU function, and the output layer of the policy network uses the Tanh activation function to limit the range of action outputs.

[0094] The status includes information about the vehicle itself and surrounding vehicles, i.e. ,in This represents the upper limit of the number of vehicles in the detection range. The vehicle information is a 7-dimensional vector, including the vehicle's current position, orientation angle, velocity, and acceleration, which can be represented as... Environmental vehicle information is a 5-dimensional vector, including environmental vehicles. The current position, orientation angle, and velocity can be represented as: After being processed by the feature extraction network, all the above state information will be mapped into a 128-dimensional high-dimensional feature vector and input into the secure and robust reinforcement learning decision framework.

[0095] The decision-making process of the safety robust reinforcement learning decision-making framework is achieved through longitudinal control of the vehicle, specifically by controlling the accelerator and brake pedals. Therefore, the policy network output action space of the safety robust reinforcement learning decision-making framework is a one-dimensional continuous space ranging from [-1, 1]. When the output action value is in (0, 1), the action is applied to the accelerator pedal opening; when the value is in [-1, 0], the action is applied to the brake pedal opening.

[0096] The intelligent decision-making process based on a security-robust reinforcement learning decision-making framework is divided into a training phase and a deployment phase. The training process is as follows: Figure 5 As shown. During the training phase, all reinforcement learning-related network parameters are first initialized, and then... In the step-by-step loop, the current policy network is trained using sampled tuple data. The sampled state needs to be subjected to anti-antibody noise attacks to simulate perception perturbations. After processing by the feature extraction module, it is input into the policy network to complete action sampling. When the amount of data in the historical data buffer reaches a preset threshold, the network parameter update process is triggered: the gradient updates of the task value network and the safety cost network are executed sequentially; the parameters of the policy network are optimized based on the above value assessment; the entropy regularization coefficient and the safety constraint regularization coefficient in the loss function of the policy network are dynamically adjusted simultaneously to balance exploration efficiency and safety constraints; finally, the target network (i.e., the target value network and the target cost network) is softly updated. In the deployment phase, the globally optimal or converged policy network parameters generated in the training phase are loaded and frozen for updates; the policy network receives the high-dimensional feature vector processed by the feature extraction module in real time and calculates the optimal control action to achieve precise control of the throttle and brake actuators of the autonomous vehicle.

[0097] In autonomous driving missions, ensuring zero collisions is a core safety benchmark, and Time to Collision (TTC), as a commonly used autonomous driving safety indicator, can effectively characterize potential collision risks. This invention establishes a cost evaluation mechanism for dynamic targets ahead of the vehicle's driving path. This mechanism combines the relative distance and spatial orientation information between vehicles, and calculates the TTC index between the vehicle and surrounding vehicles in real time, using it as the basis for cost function calculation, as shown in the following formula:

[0098]

[0099]

[0100]

[0101]

[0102] in, This indicates the distance between the center of mass of the vehicle and other vehicles in the environment. This represents the projected distance between the outer edges of the vehicle and the surrounding vehicles along the vehicle's direction of travel. In this embodiment, it is assumed that the vehicle and the surrounding vehicles have the same outer dimensions. Indicates the length of the vehicle; Indicates the width of the vehicle; This represents the orientation angle of the surrounding vehicle relative to the vehicle itself, ranging from [0, 90°]. Indicates the absolute speed of the vehicle in the environment; Indicates the absolute speed of the vehicle; This represents the projected velocity of the vehicle relative to its own orientation angle in the environment. Represents the cost function; This represents the cost weighting coefficient; Indicates the absolute position of the vehicle in the environment; This refers to the area within a certain angle and distance in front of the vehicle.

[0103] The reward function of this invention includes four aspects: safety, comfort, timeliness, and action guidance.

[0104] Safety rewards are based on collisions between the vehicle and other vehicles in the environment. When a collision occurs, the system will impose a severe penalty on the agent. ;in, This indicates a security reward.

[0105] Comfort rewards aim to optimize vehicle dynamics by quantifying the coherence and smoothness of accelerator and brake pedal operations. This mechanism uses the increment of action between adjacent time steps as the core evaluation metric: when the rate of change between the control command (accelerator or brake intensity) in the current time step and the action in the previous time step exceeds a preset smoothing threshold, the system will impose a corresponding numerical penalty on the agent based on the magnitude of this difference. This is illustrated below:

[0106]

[0107] in, Indicates a comfort reward; express The actions taken by the agent at any given moment; This indicates the sampling time step of the simulation platform; This represents the threshold at which the system imposes a comfort penalty.

[0108] The timeliness-based reward module comprises two aspects: an incentive mechanism based on vehicle speed and a guidance mechanism based on task round progress. Regarding speed control, the system implements dynamic rewards and penalties by monitoring the mapping relationship between vehicle speed and preset speed limits: when the vehicle speed is within the legal speed limit range, the agent is given a positive reward to incentivize it to maintain efficient power output; once the speed exceeds the speed limit threshold, a corresponding numerical penalty is immediately applied to avoid safety risks caused by excessive speed. In terms of round progress, the system calculates the proportion of the current distance traveled to the total round target progress and applies continuous progress penalties to stagnant or slow-moving states. As shown below:

[0109]

[0110]

[0111] in, Indicates a reward based on vehicle speed; Represents rewards based on round progression; Indicates the current vehicle speed; This indicates the vehicle speed limit in the current scenario; Indicates the distance the vehicle has traveled in the current round; This indicates the total distance the vehicle needs to travel in the current round; and These represent the weighting coefficients for speed rewards and penalties, respectively.

[0112] The action guidance reward module consists of acceleration guidance rewards and deceleration guidance rewards, aiming to guide the agent to make efficient and safe decisions through dense rewards. Acceleration guidance rewards are primarily determined by vehicle speed and TTC (Traffic Tethering Control) status. When the environment meets a preset safety threshold, this reward mechanism incentivizes the agent to accelerate appropriately while ensuring safety, effectively avoiding a prolonged local optimum stalemate. Deceleration guidance rewards focus on risk warnings. When the TTC indicator triggers a danger threshold, enhanced negative feedback guides the agent to take timely braking and deceleration actions to avoid collision risks.

[0113]

[0114]

[0115] in, This indicates an accelerated introductory reward; Indicates a deceleration-guided reward; Indicates the speed threshold; and These represent the acceleration and deceleration thresholds for TTC, respectively.

[0116] Therefore, the overall reward function The expression is:

[0117]

[0118] in, , , , , and These represent the weight coefficients for safety, comfort, timeliness, and action-guided rewards, which are adjusted during training based on the training progress to guide the agent's policy learning process.

[0119] The intelligent decision-making process based on a security-robust reinforcement learning decision-making framework is divided into a training phase and a deployment phase, specifically including:

[0120] Step 1: Pre-training data collection;

[0121] In the data preparation phase, a multi-vehicle collision scenario without signal control was first constructed based on the Carla high-fidelity simulation platform. The autonomous vehicle used the A* (A-Star) algorithm for path planning and integrated a path polygon collision detection algorithm. By calculating the spatiotemporal overlap between the environmental vehicle bounding boxes and the autonomous vehicle's planned path, the system accurately identified potential collision risks. At the underlying execution level, a PID control algorithm was used as the autonomous vehicle's kinematic controller for path following control. Throughout the simulation, multi-step historical trajectory tuple data, containing state information of all environmental vehicles and the autonomous vehicle, was cyclically collected at a preset frequency and stored in the simulation platform's memory (historical data buffer).

[0122] Step 2: Pre-training the feature extraction network;

[0123] In the pre-training phase of the feature extraction network, the core task focuses on optimizing the parameters and shaping the representation capabilities of the vehicle temporal feature extraction module. In this phase, utilizing the massive sequence data from the previous pre-training data acquisition phase, the vehicle's historical trajectory sequence is used as input features, and the corresponding future multi-step predicted trajectory is used as supervision information. A forced teaching strategy guides the feature fusion module to continuously predict future states, and a supervised learning task is constructed through an autoregressive mechanism. Within this framework, the maximum negative log-likelihood function is selected as the loss function. Its mathematical expression is as follows:

[0124]

[0125] in, Represents a logarithmic function; Indicates the actual trajectory; This represents the predicted Gaussian distribution vector output by the vehicle temporal feature extraction module. Included predicted trajectories; It represents the likelihood function of the actual trajectory sample in the future relative to the trajectory predicted by the network; This represents the network parameters of the vehicle temporal feature extraction module; Indicates the environmental vehicle number; This indicates the total number of vehicles in the scene.

[0126] After obtaining the loss values ​​of the batch data, the ADAM optimizer is used to perform stochastic gradient descent to iteratively update the weight parameters of the vehicle temporal feature extraction module.

[0127] Step 3: Pre-train the policy network;

[0128] After the feature extraction network is pre-trained, the system enters the pre-training stage of the policy network. To maintain the stability of the learned low-level representations, the system freezes the network parameters of the vehicle temporal feature extraction module and uses the Behavior Cloning (BC) method from imitation learning to achieve joint training of the feature extraction network and the policy network. Considering that the action space of the safety robust reinforcement learning decision framework of this invention is a continuous action space, and the action distribution output by the policy network follows a Gaussian distribution, the Kullback-Leibler divergence (KL divergence) between the action distribution generated by the pre-trained policy network and the real action distribution in the buffer data is selected as the core loss function, as shown in the following equation:

[0129]

[0130] in, and Let represent the mean and variance of the Gaussian distribution of the action output of the pre-trained policy network, respectively. Indicates the parameters of the pre-trained policy network; This indicates the action of collecting pre-training data; Indicates state; This represents the pre-trained dataset.

[0131] After obtaining the loss value of the batch data, the ADAM optimizer is used to perform stochastic gradient descent, and the gradient is backpropagated to the feature fusion module and the pre-trained policy network. The weight parameters of both are updated iteratively to enhance the initial perception and decision-making ability for specific complex scenarios. During this period, the vehicle temporal feature extraction module is in a frozen state, and its internal parameters remain unchanged.

[0132] Step 4: Training the secure and robust reinforcement learning decision-making framework;

[0133] After completing all the above pre-training steps, the secure robust reinforcement learning decision framework is trained. Adversarial networks are introduced at this stage, therefore the state prior to this stage... With action There is no interference. The training process of the secure and robust reinforcement learning decision-making framework designed in this invention is as follows: Figure 5 As shown.

[0134] During the training startup phase, the initialization settings for each network parameter, optimizer data, and historical data buffer are first completed. Specifically, the network parameters of the pre-trained policy network are directly initialized for this policy network. Then, a sampling loop is entered, controlling the policy network to perform interactive sampling in the simulation environment, and the acquired sampled tuple data, after being interfered with by the adversarial network, is processed. The data is stored in the historical data buffer. In each training round, the system will monitor the current cumulative time steps in real time. The subsequent parameter update process will only be triggered when the number of sampling rounds (or steps) reaches the preset pre-sampling threshold.

[0135] After the update condition is triggered, a batch of data of a specified size is randomly selected from the historical data buffer to perform the network parameter learning process.

[0136] First, the system optimizes the task value network. The policy network calculates the next-time action and policy entropy, then the target value network calculates the action-state value of the next-time state. Combining these two factors yields the temporal difference target (TD Target) for the current state-action pair. Subsequently, using this target value as a benchmark, the task value network loss function is constructed by calculating the temporal difference error (TD Error) between the predicted and target values. Finally, gradient descent updates the task value network parameters using gradient backpropagation.

[0137] Subsequently, the safety cost network is optimized. The target cost network is used to calculate the TDTarget of the action in the current state. Based on this target value, the TD Error between the predicted value and the target value is calculated to construct the safety cost network loss function. Finally, the parameters of the safety cost network are updated using gradient descent through gradient backpropagation.

[0138] After updating the task value network and security cost network, the system performs optimization steps for the policy network. The loss function of the policy network is calculated, and the policy network parameters are updated through backpropagation to optimize the policy in a direction that satisfies the optimization objective.

[0139] The next step is to update the adversarial network. Based on the actual situation, select the appropriate adversarial network attack mode: first, calculate the loss function of the adversarial network, then update the adversarial network parameters to ensure it can effectively and robustly attack the policy network and adapt to changes in policy and environment.

[0140] Next, constraint adjustment and soft updates of the target value network and target cost network are performed. To balance the exploration and optimization intensity of the strategy, the entropy regularization coefficient is dynamically adjusted by calculating the loss function of the entropy regularization coefficient, thus updating this coefficient. Similarly, to balance the exploration intensity of the optimal strategy and the strength of security constraints, the system dynamically adjusts the safety constraint regularization coefficient by calculating the loss function of the cost constraint coefficient, thus adjusting the strength of the safety constraints. The loss functions for the two regularization coefficients are shown below.

[0141]

[0142]

[0143] in, Represents the coefficient of the entropy regularization term; This indicates the noise interference of the adversarial network on the state; Represents the coefficient of the regularization term for safety constraints; This represents the loss function with entropy regularization coefficients; This represents the loss function for the regularization coefficients of the safety constraints. The threshold representing policy entropy; Indicates the safety constraint threshold; This indicates the lower limit of the preset cost item coefficient; and This represents the adjustment coefficient of the cost term coefficient loss function.

[0144] At the end of each update cycle, a soft update operation is performed on each target network (i.e., the target value network and the target cost network). By slowly tracking the parameter changes of the main networks (i.e., the task value network and the safety cost network), the numerical stability of the entire training process is ensured. The soft update process is as follows:

[0145]

[0146]

[0147] in, This represents the soft update coefficient.

[0148] Step 5: Deployment of a secure and robust reinforcement learning decision-making framework;

[0149] According to the above iterative update process, each network in the reinforcement learning decision-making method framework of this invention will converge to the optimal parameters one by one. Finally, after reaching the optimal policy or the upper limit of the number of iterations, the parameters of each network are frozen, and the policy network is taken as the decision module in the whole scheme. This decision module (policy network) receives the high-dimensional feature vector processed by the feature extraction module in real time and calculates the optimal control action command to achieve precise control of the throttle and brake actuators of the autonomous vehicle.

[0150] Step S5: Control the accelerator pedal opening and brake pedal opening in real time through optimal control action commands to achieve longitudinal motion control of the vehicle;

[0151] Step S6: Repeat steps S1-S5 until the autonomous vehicle safely passes through the uncontrolled intersection.

[0152] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A decision-making method for automated driving at signalless intersections based on safety robust reinforcement learning, characterized in that, include: Step S1: Before the autonomous vehicle enters an intersection without traffic control, it collects traffic status information of itself and other vehicles around it through sensors; the status information of the autonomous vehicle includes: position, orientation angle, speed and acceleration; the status information of other vehicles around it includes: position, orientation angle and speed. Step S2: Identify and extract the status information of all potential conflicting vehicles in the vicinity according to the dynamic filtering rules for potential conflicting vehicles; The dynamic screening rule for potential conflict vehicles is as follows: A detection range is defined centered on the vehicle, based on a preset radius. Only traffic participants within this detection range are included in the initial selection set of potential conflict vehicles. If the number of vehicles within the detection range is insufficient, i.e., the number of vehicles is less than [a certain value], [further details are needed]. If the number of vehicles exceeds a preset maximum vehicle count threshold, then false stationary vehicles at fixed preset coordinates at the edge of the detection range will be used as placeholders to supplement the vehicle count. Centered on the vehicle, the segment is cut off based on spatial proximity, retaining The closest target that has the greatest impact on the vehicle is considered as a potential conflict vehicle; Step S3: Input the state information of the vehicle and all potential conflicting vehicles into the feature extraction network to calculate a high-dimensional feature vector representing the complexity of the environment; wherein, the feature extraction network includes a vehicle temporal feature extraction module and a feature fusion module; Step S4: Input the high-dimensional feature vector into the safe and robust reinforcement learning decision framework, and generate the optimal control action command at the current time through the mapping calculation of the policy network in the framework; The security-robust reinforcement learning decision framework includes an input layer, a historical data buffer, a task performance module, a security performance module, a policy network, and an adversarial network. Step S5: Control the accelerator pedal opening and brake pedal opening in real time through optimal control action commands to achieve longitudinal motion control of the vehicle; Step S6: Repeat steps S1-S5 until the autonomous vehicle safely passes through the uncontrolled intersection.

2. The method according to claim 1, characterized in that, Step S3 specifically includes: Step S301: Transfer the vehicle's status information Status information of all potential conflict vehicles Organized as potential conflict vehicles Personal information Dynamic information and related information ; where superscript Indicates from time Time sequence, that is ; Indicates the preset number of past time steps; , , , and These represent the times of the vehicle. of Axis coordinates Axis coordinates, orientation angle, velocity, and acceleration; This indicates the number of a potential conflict vehicle, which is an environmental vehicle. , , , and Representing environmental vehicles At any moment of Axis coordinates Axis coordinates, orientation angle, velocity Axial components and velocity Axial components; , , ; , , Representing environmental vehicles At any moment Other vehicles of Axis coordinate difference The difference in axis coordinates and the included angle of orientation; Step S302: Move the environmental vehicle Personal information Dynamic information and related information Inputting the vehicle's temporal feature extraction module, it predicts the features for the next single step. ; Step S303: Roll the predictions of the future vehicles obtained by the vehicle temporal feature extraction module. Feature vectors at each time step The input feature fusion module yields a high-dimensional feature vector used as input to a secure and robust reinforcement learning decision framework. .

3. The method according to claim 2, characterized in that, In step S302, the vehicle temporal feature extraction module is divided into three branches: a state branch, a dynamic branch, and an association branch. In the state branch, the vehicle... Personal information The high-dimensional state feature vector of the vehicle is obtained by inputting the multilayer perceptron (MLP). As shown below: in, Represents a multilayer perceptron; express The parameters of the MLP network used in the operation; In dynamic branch lines, vehicles Dynamic information The high-dimensional dynamic feature vector of the vehicle is obtained by inputting the multilayer perceptron (MLP). Then, a gated recurrent neural network (GRU) is used to capture... Vehicle time-series dynamic characteristics As shown below: in, express The parameters of the MLP network used in the operation; This represents a gated recurrent neural network; The hidden state represents the vehicle's temporal dynamic characteristics from the previous moment; express The parameters of the GRU network used in the computation; In the associated branch roads, environmental vehicles Related information High-dimensional correlation feature vectors are obtained by inputting the multilayer perceptron (MLP). Then, a gated recurrent neural network (GRU) is used to capture... Vehicle time-series dynamic characteristics Next, the cross-attention mechanism is used to obtain information about other vehicles in the target environment. The influence weight, and use this weight to... The target environment vehicle is obtained by performing a weighted summation. Characteristics affected by the environment As shown in the following formula: in, express The parameters of the MLP network used in the operation; This represents the hidden state of the vehicle regarding the temporal dynamic features at the previous moment; express The parameters of the GRU network used in the computation; This represents the softmax function; Represents the weight parameters of the query matrix in the attention mechanism; Represents the weight parameters of the key matrix in the attention mechanism; Indicates the dimension of the key matrix; Represents matrix dot product; Represents the value matrix; Then, the vehicle's time-series dynamic characteristics and target vehicle Characteristics affected by the environment After splicing, dimensionality reduction is achieved through MLP integration. , Then, with the vehicle's high-dimensional state feature vector After concatenation, long-term dynamic features are captured using a Long Short-Term Memory (LSTM) neural network. ; and will Inputting the MLP yields the prediction vector for the next time step. Simultaneously, the MLP branches out a prediction head branch, which then predicts the vector... The parsed reconstruction is a Gaussian distribution prediction vector representing the probability of future vehicle space occupancy. As shown in the following formula: in, Indicates splicing; express The parameters of the MLP network used in the operation; Represents a Long Short-Term Memory (LSTM) neural network; superscript Indicates the past at the current time step The time window for the step size; This represents the output of hidden features in long-term memory; This represents the output of hidden features in short-term memory; express and The parameters of the LSTM network used in the operation; This represents the input of hidden features in long-term memory; express The parameters of the MLP network used in the operation; express The parameters of the MLP network used in the operation; , and Representing vehicles of coordinate, The mean vector of the Gaussian distribution of coordinates and orientation angle at future moments; , and Indicates vehicle of coordinate, Future time-major variance vector of coordinates and orientation angle; , and Indicates vehicle of coordinate, The future time-time covariance vector of coordinates and orientation angle.

4. The method according to claim 3, characterized in that, In step S303, at each time step, the vehicle temporal feature extraction module is used to perform forced teaching rolling prediction for each environmental vehicle to obtain future... Prediction vector at each time step Then, the future space occupied by the vehicle is obtained through MLP dimensionality enhancement. As shown in the following formula: in, Indicates calculation The MLP network parameters used; Next, the future space occupancy characteristics of each vehicle in the environment are input into a multi-layer attention mechanism; specifically, the vehicle's state information is input first. The vector obtained after feature extraction by MLP Environmental vehicles Personal information The vector obtained after feature extraction by MLP Vehicle weights are obtained by performing scaled dot product attention calculation. Then calculate the vehicle's... and various environmental vehicles Future space occupancy characteristics Spatial weights are obtained by scaling dot product attention calculation. Finally, based on vehicle weights With space occupied weight Calculate each environmental vehicle The future position of the vehicle's influence vector As shown below: in, Indicates calculation The MLP network parameters used; Indicates calculation The MLP network parameters used; The query matrix weight parameters represent the impact of the computational environment on the vehicle. The key matrix weight parameters represent the impact of the computational environment on the vehicle. The dimension of the key matrix represents the impact of the computational environment on the vehicle. The value matrix weight parameters represent the impact of the environmental vehicles on the vehicle. The query matrix weight parameters represent the impact of vehicle-occupied space in the computing environment on the vehicle itself; The key matrix weight parameters represent the impact of vehicle-occupied space in the computational environment on the vehicle itself. The dimension of the key matrix represents the impact of vehicle-occupied space in the computational environment on the vehicle itself; The value matrix weight parameters represent the impact of vehicle-occupied space in the computational environment on the vehicle itself. Finally, the vector will be affected The result obtained after MLP With the feature vector of the vehicle The concatenated features are then linearly transformed to map the concatenated feature dimensions to the same dimension as the input layer of the secure and robust reinforcement learning decision framework, resulting in a high-dimensional output feature vector. As shown in the following formula: in, Indicates calculation The MLP network parameters used; It represents a linear transformation.

5. The method according to claim 4, characterized in that, In step S4, when describing the safe and robust reinforcement learning decision framework, the historical data buffer is used to store tuple data sampled by the reinforcement learning agent in the simulation environment. The data is collected, stored, and shuffled in order for use and training by offline policy reinforcement learning networks; among which, Indicates state; This indicates interference from noise, i.e. Indicates actions taken after being disturbed by noise; Indicates a reward; Indicate cost; Indicates the next moment, that is Indicates the state at the next moment; The task performance evaluation module includes two task value networks and two corresponding target value networks. Each network adopts a neural network architecture consisting of two fully connected layers. The input to the task value network is a high-dimensional feature vector output by the feature extraction network. The output dimension is a one-dimensional value vector, used to measure the magnitude of the action's value. The loss function for updating the task value network is designed based on the principle of temporal difference, that is, by combining the output of the target value network with the current policy entropy, the temporal difference target value of the task value under the current state-action pair is calculated. Subsequently, based on this temporal difference target value, the task value network loss function is constructed by calculating the temporal difference error between the predicted value and the target value, as shown in the following formula: in, Indicates the number of the task value network; Representing the task value network The loss function; Representing the task value network Weight parameters; Representing the task value network Calculate the expected cumulative reward estimate of the output; This represents a historical data buffer. This represents tuple data sampled from the buffer; Indicates the number of the target value network; This represents the temporal difference target value of the task value network; Represents the reward function; Indicates the reward discount factor; Represents the target value network Weight parameters; Represents the target value network Calculate the expected cumulative reward estimate of the output; Represents the coefficient of the entropy regularization term; Represents policy entropy; This represents the output of the policy network action; The security performance evaluation module includes two security cost networks and two corresponding target cost networks. Each network employs a neural network architecture consisting of two fully connected layers. The input to the security cost network is a high-dimensional feature vector output by the feature extraction network. The output is a one-dimensional safety cost index. The loss function for updating the safety cost network is based on the temporal difference principle, that is, using the target cost network to calculate the temporal difference target value of the safety cost under the current state-action pair. Then, using this temporal difference target value as a benchmark, the safety cost network loss function is constructed by calculating the temporal difference error between the predicted value and the target value, as shown in the following formula: in, Indicates the number of the security cost network; Indicates security cost network The loss function; Indicates security cost network Weight parameters; Indicates security cost network Calculate the expected cumulative reward estimate of the output; This represents tuple data sampled from the history buffer; Indicates the number of the target cost network; This represents the temporal difference objective value of the security cost network; Represents the cost function; Represents the target cost network Weight parameters; Represents the target cost network Calculate the expected cumulative reward estimate of the output; The policy network consists of two fully connected layers, capable of receiving real-time environmental state input and calculating the optimal action to be taken in the current context. Based on maximizing task performance as the optimization objective, and constrained by safety cost and policy entropy, it is implemented using the Lagrange duality method. This transforms the constrained non-convex optimization problem into a single-objective unconstrained convex optimization problem, and the loss function for policy network updates is constructed accordingly, as shown in the following equation: in, The loss function of the policy network is represented; Indicates the policy network weight parameters; The adversarial network (ANN) consists of two fully connected layers, capable of capturing environmental state features in real time and calculating the optimal perturbation term that reduces the agent's decision-making performance. The ANN's design is based on robust Markov decision process theory, and its core logic is to pre-apply the optimal perturbation attack for the decision-making agent in the current state during the training process. By subjecting the agent to policy iteration under extreme constraints, it is forced to learn a robust policy for that state. Depending on the actual algorithm configuration, the ANN employs two attack methods: when the target is task performance, the ANN operates in a maximum reward attack mode, leveraging the agent's profit-seeking instinct to generate a perturbation term aimed at increasing the expected cumulative reward. Interference noise is maliciously guided to influence the agent's observed state, inducing it to misjudge the task's value and thus engage in dangerous profit-seeking behavior that violates security constraints in pursuit of falsely high gains. When the attack target is security performance, the adversarial network operates in maximum-cost attack mode. By directly disrupting the state representation, interference noise aimed at increasing the accumulated security cost over multiple steps is input into the observed state, directly disrupting the agent's perception of environmental risks and inducing the agent to directly engage in illegal behavior or collision risks under a state of decision-making failure. Based on the above game theory mechanism, the loss function of the adversarial network under the two attack modes is designed as follows: in, This represents the loss function of the adversarial network under the maximum reward attack mode; The loss function of the adversarial network under the maximum cost attack mode is denoted as . This indicates that the adversarial network is subject to noise interference with the state. Indicates the weight parameters of the adversarial network; The status includes information about the vehicle itself and surrounding vehicles, i.e. ,in This represents the upper limit of the number of vehicles in the environment within the detection range; the vehicle information is a 7-dimensional vector, including the vehicle's current position, orientation angle, velocity, and acceleration, represented as... Environmental vehicle information is a 5-dimensional vector, including environmental vehicles. The current position, orientation angle, and velocity are represented as follows: After being processed by the feature extraction network, all the above state information will be mapped into a 128-dimensional high-dimensional feature vector and input into the secure and robust reinforcement learning decision framework. The decision-making of the safety robust reinforcement learning decision-making framework is achieved through the longitudinal control of the vehicle, that is, the control of the accelerator pedal and brake pedal. Therefore, the policy network output action space of the safety robust reinforcement learning decision-making framework is a one-dimensional continuous space with a range of [-1,1]. When the output action value is in (0,1), the action is applied to the accelerator pedal opening; when the value is in [-1,0], the action is applied to the brake pedal opening.

6. The method according to claim 5, characterized in that, In the security-robust reinforcement learning decision framework of step S4, the parameter optimizer for all networks—policy network, adversarial network, task value network, security cost network, objective value network, and objective cost network—is the Adam optimizer, with a learning rate set to 0.0001; discount factor... The value is 0.99; all networks consist of two fully connected layers with 64 and 64 neurons respectively; the activation function between the fully connected layers is the ReLU function, and the output layer of the policy network uses the Tanh activation function to limit the range of action output.

7. The method according to claim 6, characterized in that, In step S4, a cost assessment mechanism for dynamic targets ahead of the vehicle's driving path is established. This mechanism combines the relative distance and spatial orientation information between vehicles, and calculates the TTC index between the vehicle and surrounding vehicles in real time, using it as the basis for cost function calculation, as shown in the following formula: in, This indicates the distance between the center of mass of the vehicle and other vehicles in the environment. This represents the projected distance between the outer edges of the vehicle and the surrounding vehicles along the vehicle's direction of travel. In this embodiment, it is assumed that the vehicle and the surrounding vehicles have the same outer dimensions. Indicates the length of the vehicle; Indicates the width of the vehicle; This represents the orientation angle of the surrounding vehicle relative to the vehicle itself, ranging from [0, 90°]. Indicates the absolute speed of the vehicle in the environment; Indicates the absolute speed of the vehicle; This represents the projected velocity of the vehicle relative to its own orientation angle in the environment. Represents the cost function; This represents the cost weighting coefficient; Indicates the absolute position of the vehicle in the environment; This refers to the area within a certain angle and distance in front of the vehicle. The reward function includes four aspects: safety, comfort, timeliness, and action guidance. Safety rewards are based on collisions between the vehicle and other vehicles in the environment. When a collision occurs, the system will impose a severe penalty on the agent. ;in, This indicates a security reward; Comfort rewards optimize vehicle dynamics by quantifying the continuity and smoothness of accelerator and brake pedal operations. This mechanism uses the increment of motion between adjacent time steps as the core evaluation metric: when the rate of change between the control command of the current time step (accelerator or brake intensity) and the action of the previous time step exceeds a preset smoothing threshold, the system will impose a corresponding numerical penalty on the agent based on the magnitude of this difference; as shown below: in, Indicates a comfort reward; express The actions taken by the agent at any given moment; This indicates the sampling time step of the simulation platform; This represents the threshold at which the system imposes a comfort penalty. The timeliness reward module comprises two aspects: an incentive mechanism based on vehicle speed and a guidance mechanism based on task round progress. Regarding speed control, the system implements dynamic rewards and penalties by monitoring the mapping relationship between vehicle speed and preset speed limits: when the vehicle speed is within the legal speed limit range, the agent is given a positive reward to incentivize it to maintain efficient power output; once the vehicle speed exceeds the speed limit threshold, a corresponding numerical penalty is immediately applied to avoid safety risks caused by excessive speed. In terms of round progress, the system calculates the proportion of the current distance traveled to the total round target progress and applies continuous progress penalties to stagnant or slow-moving states, as shown below: in, Indicates a reward based on vehicle speed; Represents rewards based on round progression; Indicates the current vehicle speed; This indicates the vehicle speed limit in the current scenario; Indicates the distance the vehicle has traveled in the current round; This indicates the total distance the vehicle needs to travel in the current round; and These represent the weighting coefficients for speed rewards and penalties, respectively. The action guidance reward module consists of acceleration guidance rewards and deceleration guidance rewards, aiming to guide the agent to make efficient and safe decisions through dense rewards. Among them, acceleration guidance rewards are mainly determined by vehicle speed and TTC status. When the environment meets the preset safety threshold, this reward mechanism will incentivize the agent to accelerate appropriately while ensuring safety, thereby effectively avoiding getting stuck in a local optimum deadlock where it remains stationary for a long time. Deceleration guidance rewards focus on risk warning. When the TTC indicator triggers the danger threshold, it guides the agent to take timely braking and deceleration actions to avoid collision risks by enhancing negative feedback. in, This indicates an accelerated introductory reward; Indicates a deceleration-guided reward; Indicates the speed threshold; and These represent the acceleration and deceleration thresholds for TTC, respectively. Therefore, the overall reward function The expression is: in, , , , , and These represent the weight coefficients for safety, comfort, timeliness, and action-guided rewards, which are adjusted during training based on the training progress to guide the agent's policy learning process.

8. The method according to claim 7, characterized in that, In step S4, the intelligent decision-making process based on the security robust reinforcement learning decision-making framework is divided into a training phase and a deployment phase, specifically including: Step 1: Pre-training data collection; In the data preparation phase, a multi-vehicle conflict intersection scenario without signal control was first built based on the Carla high-fidelity simulation platform. The autonomous vehicle used the A-Star algorithm for path planning and integrated a path polygon conflict detection algorithm. By calculating the spatiotemporal overlap between the environmental vehicle bounding boxes and the autonomous vehicle's planned path, the potential collision risks were accurately identified. At the underlying execution level, a PID control algorithm was used as the autonomous vehicle's kinematic controller for path following control. Throughout the simulation, multi-step historical trajectory tuple data containing the state information of all environmental vehicles and the autonomous vehicle was collected cyclically at a preset frequency and stored in the historical data buffer. Step 2: Pre-training the feature extraction network; In the pre-training stage of the feature extraction network, the massive sequence data from the previous pre-training data acquisition stage is used to input the vehicle's historical trajectory sequence as the feature, and the corresponding future multi-step predicted trajectory is used as the supervision information. A forced teaching strategy is adopted to guide the feature fusion module to predict future states in a rolling manner, and a supervised learning task is constructed through an autoregressive mechanism. Under this framework, the maximum negative log-likelihood function is selected as the loss function. Its mathematical expression is as follows: in, Represents a logarithmic function; Represents the actual trajectory; This represents the predicted Gaussian distribution vector output by the vehicle temporal feature extraction module. Included predicted trajectories; It represents the likelihood function of the actual trajectory sample in the future relative to the trajectory predicted by the network; This represents the network parameters of the vehicle temporal feature extraction module; Indicates the environmental vehicle number; This indicates the total number of vehicles in the scene. After obtaining the loss values ​​of the batch data, the ADAM optimizer is used to perform stochastic gradient descent to iteratively update the weight parameters of the vehicle temporal feature extraction module. Step 3: Pre-train the policy network; After the feature extraction network is pre-trained, the system enters the pre-training stage of the policy network. To maintain the stability of the learned underlying representations, the system freezes the network parameters of the vehicle temporal feature extraction module and uses the behavior cloning method in imitation learning to achieve joint training of the feature extraction network and the policy network. Considering that the action space of the safety robust reinforcement learning decision framework is a continuous action space, and the action distribution output by the policy network follows a Gaussian distribution, the KL divergence between the action distribution generated by the pre-trained policy network and the real action distribution in the buffer data is selected as the core loss function, as shown in the following equation: in, and Let represent the mean and variance of the Gaussian distribution of the action output of the pre-trained policy network, respectively. Indicates the parameters of the pre-trained policy network; This indicates the action of collecting pre-training data; Indicates state; This represents the pre-trained dataset; After obtaining the loss value of the batch data, the ADAM optimizer is used to perform stochastic gradient descent, and the gradient is backpropagated to the feature fusion module and the pre-trained policy network. The weight parameters of both are updated iteratively to enhance the initial perception and decision-making ability for specific complex scenarios. During this period, the vehicle temporal feature extraction module is in a frozen state, and its internal parameters remain unchanged. Step 4: Training the secure and robust reinforcement learning decision-making framework; After completing all the above pre-training steps, the secure and robust reinforcement learning decision framework is trained. During the training startup phase, the initialization settings for each network parameter, optimizer data, and historical data buffer are first completed. Specifically, the network parameters of the pre-trained policy network are directly initialized for this policy network. Then, a sampling loop is entered, controlling the policy network to perform interactive sampling in the simulation environment, and the acquired sampled tuple data after being interfered with by the adversarial network is processed. Stored in the historical data buffer; in each training round, the system will monitor the current cumulative time steps in real time, and only when the number of sampling steps reaches the preset pre-sampling threshold will the subsequent parameter update process be triggered; After the update condition is triggered, a batch of data of a specified size is randomly selected from the historical data buffer to perform the network parameter learning process. First, the task value network is optimized. The next action and policy entropy are calculated through the policy network. Then, the action-state value of the next state is calculated using the target value network. Combining the two, the temporal difference target (TD Target) of the current state-action pair is obtained. Subsequently, based on this target value, the task value network loss function is constructed by calculating the temporal difference error (TD Error) between the predicted value and the target value. Finally, the parameters of the task value network are updated by gradient descent using the gradient backpropagation mechanism. Subsequently, the safety cost network is optimized; the target cost network is used to calculate the TDTarget of the action in the current state, and based on this target value, the TD Error between the predicted value and the target value is calculated to construct the safety cost network loss function. Finally, the parameters of the safety cost network are updated by gradient descent through the gradient backpropagation mechanism. After the task value network and security cost network are updated, the system performs the optimization steps of the policy network; calculates the loss function of the policy network, and updates the policy network parameters through the backpropagation mechanism to optimize the policy in the direction of satisfying the optimization objective; Next, update the adversarial network; select the corresponding adversarial network attack mode according to the actual situation: first, it is necessary to calculate the loss function of the adversarial network, and then update the parameters of the adversarial network to ensure that it can exert effective and robust attacks on the policy network and ensure that the attack can adapt to changes in policy and environmental state. Next, constraint adjustment and soft updates of the target value network and target cost network are performed. To balance the exploration and optimization strength of the strategy, the entropy regularization coefficient is dynamically adjusted by calculating the loss function of the entropy regularization coefficient, thus completing the update of this coefficient. Similarly, to balance the exploration strength of the optimal strategy and the strength of the security constraints, the system dynamically adjusts the safety constraint regularization coefficient by calculating the loss function of the cost constraint coefficient, thus completing the adjustment of the safety constraint strength. The loss functions of the two regularization coefficients are shown below: in, Represents the coefficient of the entropy regularization term; This indicates that the adversarial network is subject to noise interference with the state. Represents the coefficient of the regularization term for safety constraints; This represents the loss function with entropy regularization coefficients; This represents the loss function for the regularization coefficients of the safety constraints; The threshold representing policy entropy; Indicates the safety constraint threshold; This indicates the lower limit of the preset cost item coefficient; and This represents the adjustment factor for the cost item coefficient loss function; At the end of each update cycle, a soft update operation is performed on each target network, namely the target value network and the target cost network. By slowly tracking the parameter changes of the main networks, namely the task value network and the safety cost network, the numerical stability of the entire training process is ensured. The soft update process is as follows: in, Indicates the soft update coefficient; Step 5: Deployment of a secure and robust reinforcement learning decision-making framework; According to the above iterative update process, each network in the reinforcement learning decision-making method framework will converge to the optimal parameters one by one. Finally, after reaching the optimal policy or the upper limit of the number of iterations, the parameters of each network are frozen, and the policy network is taken as the decision module in the whole scheme. This decision module, namely the policy network, receives the high-dimensional feature vector processed by the feature extraction module in real time and calculates the optimal control action command to achieve precise control of the throttle and brake actuators of the autonomous vehicle.

Images