A safe automatic driving method based on a double alternative diffusion strategy

Abstract

The application discloses a kind of security automatic driving methods based on double alternative diffusion strategy, for the problem that traditional off-line reinforcement learning may cause unsafe behavior in automatic driving task due to extrapolation error, propose to use two diffusion models as mutually alternative strategy, and use integrated network containing multiple action value networks as policy evaluation network;By training two diffusion models and integrated network in offline environment, the uncertainty of the actions generated by the two strategies is evaluated using the integrated network during the deployment phase, and the strategy with lower uncertainty is selected as the final driving strategy, thereby ensuring the safety of the automatic driving task during the deployment phase.

Description

TECHNICAL FIELD

[0001] The present application relates to an offline reinforcement learning method, in particular to a safe automatic driving method. BACKGROUND

[0002] Reinforcement learning can effectively solve sequential decision-making problems through trial and error, and is widely used in automatic driving. However, traditional reinforcement learning needs to interact with the environment online, which is low in efficiency and has safety risks, limiting its development. Recent studies have shown that reinforcement learning can first learn an initial strategy from a static data set, and then optimize the strategy performance through limited online interaction. This method is called offline reinforcement learning (ORL). The development of ORL provides strong support for the application of reinforcement learning in automatic driving.

[0003] Automatic driving tasks are related to safety, and any unsafe behavior may cause serious consequences. In order to better apply ORL to automatic driving tasks, the primary challenge is to solve the extrapolation error problem. The extrapolation error is caused by the fact that ORL relies on static data sets for training, and these data sets are usually collected by a specific behavior strategy. Therefore, the state-action distribution in the data set is limited by the behavior strategy, resulting in inaccurate or even incorrect value estimates of the model in areas not covered by the data. This problem is particularly serious: if ORL is directly deployed to automatic driving tasks without adequate safety measures, the policy may make incorrect decisions due to extrapolation errors, which may lead to serious safety accidents. Methods to alleviate extrapolation errors usually focus on handling out-of-distribution (OOD) state-action pairs, which can be divided into the following two categories: 1) policy constraints: by constraining the learned policy to remain close to the behavior policy, the likelihood of sampling OOD actions during bootstrapping is reduced. 2) value function penalty: assign low values to OOD actions to punish the learned value function. By imposing constraints or penalties, the trained policy is explicitly guided to remain near the data set collected by the behavior policy. However, these methods require accurate evaluation of the behavior policy or precise sampling of OOD samples, both of which are difficult. In addition, these methods fail to fully utilize the generalization ability of the action value (Q-value) function, completely prohibiting the intelligent vehicle from exploring any OOD state-action pair without considering its potential value, resulting in a conservative policy. If OOD data points whose Q-values can be predicted with high confidence can be identified, it may be more effective to allow the intelligent vehicle to choose these data points.

[0004] Studies have shown that when using Q-value function ensemble networks to estimate the posterior distribution of Q-value, in areas with rich data, Q-value tends to converge, while in areas with sparse data, Q-value tends to diverge. This finding enables us to use Q-function ensemble networks to measure the uncertainty of actions and ensure that the learned policy does not deviate from the behavior policy by selecting actions with lower uncertainty. However, relying solely on uncertainty estimates may face three problems: 1) Lack of data diversity: In order for the Q-value function ensemble network to accurately capture the true uncertainty, the training data must have sufficient diversity and comprehensiveness; if the training data is limited or significantly biased, the ensemble method may not provide reliable uncertainty estimates. 2) Lack of independence: If the initialization, training data or optimization process of the Q-value network are highly similar, their outputs may be highly correlated, weakening the effect of the ensemble; in this case, the estimated uncertainty may not be accurate, failing to reflect the model's true confidence in actions. 3) Ineffectiveness for highly complex distributions: When the true source of uncertainty is complex, such as multi-modal distributions or environments with high heterogeneity, the numerical output of the Q-value function ensemble network may not adequately capture these complexities, limiting its effectiveness in representing uncertainty.

[0005] On the other hand, given the significant success of diffusion models in image generation tasks, recent studies have shown that diffusion models also exhibit high expressiveness in learning policies for reinforcement learning tasks, especially in generating highly diverse and robust policy or action distributions. Compared to traditional Gaussian distribution assumptions, diffusion models can more effectively handle multi-modal distributions, thus performing particularly well in complex scenarios involving multiple optimal policies. However, reinforcement learning based on diffusion models also faces some challenges, including training instability, sensitivity to hyperparameters, difficulty in reward guidance, and significant randomness in the sampling process; these challenges make it difficult for diffusion models to generate consistent policies. At the same time, the potential of diffusion models in automatic driving decision-making tasks has not been fully explored. The inherent reverse generation process of diffusion models has black-box characteristics, leading to a lack of interpretability. Therefore, the key to successfully applying diffusion models to automatic driving tasks lies in fully utilizing their high expressiveness while ensuring the safety of the decision-making process. SUMMARY

[0006] The purpose of the invention is to address the above-mentioned prior art, and to provide a safe automatic driving method based on double alternative diffusion strategies, which is a safe offline reinforcement learning method that does not require interaction with the environment during training, and can ensure the safety of the policy when deployed in an automatic driving environment.

[0007] Technical solution: A safe automatic driving method based on double alternative diffusion strategies, first, two diffusion models are constructed as policy networks, and an ensemble network containing multiple action value networks is used as the policy evaluation network;

[0008] Secondly, the two diffusion models and the integrated network are trained in an offline environment to obtain two trained diffusion models and a trained integrated network;

[0009] Then, in the deployment stage, the trained integrated network is used to evaluate the uncertainty of the actions generated by the two trained diffusion models;

[0010] Finally, the action with lower uncertainty is selected for execution.

[0011] Further, the safety automatic driving method based on the double alternative diffusion strategy comprises the following steps:

[0012] Step 1, constructing a strategy network and an integrated network, and initializing the network parameters of the strategy network and the integrated network;

[0013] The two diffusion models are diffusion model and diffusion model The two diffusion models constitute a strategy network;

[0014] An integrated network comprising K action value networks is established as the strategy evaluation network;

[0015] The diffusion model The diffusion model and the parameters of the K action value networks are represented by θ1, θ2 and φ k respectively;

[0016] Step 2, constructing the diffusion model diffusion model and the target strategy network of the K action value networks and the target integrated network wherein θ′1, θ′2 and φ′ k represent the network parameters of the target strategy network and the target integrated network respectively;

[0017] The initialization method of the parameters of the target network is to directly assign the parameters (θ1, θ2, φ k ) of the corresponding original network to the parameters (θ′1, θ′2, φ′ k ) of the target network;

[0018] Step 3, randomly extracting sample inputs into the integrated network, the strategy network and the target network from an offline data set, and training the diffusion model diffusion model train the diffusion model with K action-value networks, update the diffusion model diffusion model ensemble network and the parameters of its target network; obtain two trained diffusion models, and a trained ensemble network;

[0019] Step 4, in the deployment stage, the trained ensemble network and the two trained diffusion models are used for the deployment of the intelligent vehicle, the uncertainty of the actions generated by the two trained diffusion models is evaluated by using the trained ensemble network, and the action with lower uncertainty is selected for execution.

[0020] Further, step 3, randomly extract sample input into the ensemble network, policy network and target network from the offline data set, and update the diffusion model diffusion model train the diffusion model with K action-value networks, update the diffusion model diffusion model ensemble network and the parameters of its target network; obtain two trained diffusion models, and a trained ensemble network, the specific steps are as follows:

[0021] Step 3.1, randomly extract sample (s, a, r, s') from the offline data set ; wherein s in the sample (s, a, r, s') represents the current state of the intelligent vehicle, a represents the action executed by the intelligent vehicle through the policy network, r represents the immediate reward obtained by the intelligent vehicle, and s' represents the next time state of the intelligent vehicle;

[0022] input (s, a) in the sample into the ensemble network diffusion model and the diffusion model , input (r, s') into each action-value network in the target ensemble network , and input s' into the target policy network and respectively;

[0023] Step 3.2, update the network parameters of the ensemble network

[0024] First, input s' into the target policy network and respectively to generate actions

[0025] Then, the uncertainty values u'1, u'2 of the actions are calculated respectively by using the target network of the ensemble network ​

[0026] comparing the uncertainty value u'1 of the action with the uncertainty value u'2 of the action and selecting the target value according to the smaller u'

[0027]

[0028] represents an indicator function;

[0029] Finally, the parameters of each action value network are updated by minimizing the temporal difference error between the output value of each action value network in the ensemble network and the joint reward value of the target value, that is, the loss function is minimized as follows:

[0030]

[0031] wherein, represents the number of experience samples in a mini-batch, here when u'1≤u'2, otherwise

[0032] The parameters of the kth action value network are updated using gradient descent method, and the adjustment amount of the parameters is: k k

[0033]

[0034] wherein, l φ represents the learning rate in the gradient descent process;

[0035] Step 3.3, update the network parameters of the policy network and Specifically:

[0036] Minimize the loss function by gradient descent as follows:

[0037]

[0038] wherein, j={1,2}, represents the number of two independent diffusion models, η and σ are balance coefficients, represents the policy regularization loss, represents the policy improvement term, represents the uncertainty regularization term;

[0039] The parameters are updated using gradient descent method, and the adjustment amount of the parameters is: j j

[0040] ​​​​

[0041] wherein, l θ denotes the learning rate in the gradient descent process, denotes the derivation of the respective network parameters in the two independent diffusion models;

[0042] Step 3.4, updating the target network parameters;

[0043] First, calculate: ρθ1+ (1-ρ)θ'1, ρθ2+ (1-ρ)θ'2 and ρφ k + (1-ρ)φ' k respectively;

[0044] Then, assign the results of the above calculations to: θ'1, θ'2 and φ' respectively k ;

[0045] That is, θ'1= ρθ1+ (1-ρ)θ'1

[0046] θ'2= ρθ2+ (1-ρ)θ'2

[0047] φ' = ρφ k + (1-ρ)φ' k k

[0048] wherein, ρ denotes the target network update rate;

[0049] Step 3.5, repeat steps 3.1 to 3.4, constantly update the network parameters, get the trained integrated network and the two trained diffusion models.

[0050] Further, the target network using the integrated network calculates the uncertainty values u'1, u'2 of the actions respectively, as follows:

[0051]

[0052] wherein, denotes the kth action value function in the target integrated network , and denotes the average action value.

[0053] Further, the policy regularization loss in step 3.3 is equivalent to the behavior cloning loss, that is:

[0054]

[0055] wherein, ∈ denotes a standard Gaussian noise, denotes a noise prediction model,​ represents the cumulative value of a i ;

[0056] Then, the policy improvement item is:

[0057]

[0058] wherein, represents the action generated by the diffusion model according to the state s, that is

[0059] Finally, regarding guaranteeing the safety of the action generated by the policy, that is, the uncertainty regular term is:

[0060]

[0061] Further, in step 3.3, the balance coefficient η in the loss function is 0.01, and σ is 1.

[0062] The learning rate e in the gradient descent process θ = 0.003.

[0063] In step 3.4, the target network update rate ρ is 0.005.

[0064] Further, in step 4, the action with lower uncertainty is selected for execution, denoted as:

[0065]

[0066] a selected is the action finally selected by the intelligent vehicle for execution, s E is the real-time state obtained by the intelligent vehicle when interacting with the environment,

[0067] Further, the current state s of the intelligent vehicle includes the steering wheel value, the heading angle, the speed, the distance from the intelligent vehicle body to the two sides of the road, and the distance from the intelligent vehicle body to the surrounding obstacles.

[0068] The action a includes the throttle and brake values and the steering wheel value.

[0069] The throttle and brake values range from -1 to 1.

[0070] When the throttle and brake values are in the range [-1, 0), the intelligent vehicle is in the brake state, and when the throttle and brake values are equal to -1, the brake force is maximum.

[0071] When the throttle and brake values are in the range [0, 1], the intelligent vehicle is in the acceleration state, and when the throttle and brake values are equal to 1, the acceleration is maximum.

[0072] The steering wheel value ranges from [-1, 1], wherein [-1, 0) represents steering the steering wheel to the left, -1 represents steering the steering wheel to the left to the bottom, [0, 1] represents steering the steering wheel to the right, and 1 represents steering the steering wheel to the right to the bottom.

[0073] Further, the diffusion model and the action value network each contain 2 hidden layers, and the hidden layer neuron number is 256 multilayer perceptron structure.

[0074] Beneficial effects: The present application proposes a safe autonomous driving method based on double alternative diffusion strategy for the safety deployment problem in autonomous driving. The main advantages of the present application are: (1) This is a safe offline reinforcement learning method that does not need to interact with the environment during training. (2) This method can guarantee the safety of the strategy when deployed in the autonomous driving environment. (3) This method uses two diffusion models as alternative strategies, introduces an action value integration network to evaluate the alternative strategies, and selects actions with lower uncertainty, thereby significantly improving the safety of the strategy. (4) This method can achieve a high task success rate while ensuring the safety of autonomous driving. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is the structure diagram of the safe autonomous driving method based on double alternative diffusion strategy. DETAILED DESCRIPTION

[0076] The present application will be further explained below in combination with the drawings.

[0077] A safe autonomous driving method based on double alternative diffusion strategy, first, 2 diffusion models are used as alternative strategies, and an integration network containing multiple action value networks is used as a strategy evaluation network; second, the two diffusion models and the integration network are trained in an offline environment; then, the integration network is used to evaluate the uncertainty of the actions generated by the two diffusion models in the deployment stage; finally, the diffusion strategy with lower uncertainty is selected as the final driving strategy.

[0078] Comprise the following specific steps:

[0079] Step 1: Prepare an offline data set, establish a strategy network and an integration network, and initialize the network parameters.

[0080] Prepare an offline data set Establish two diffusion models And And Respectively represent diffusion model 1 and diffusion model 2, which are independent of each other and jointly constitute the strategy network; an integration network containing K action value networks is established Each action value network is independent of each other; θ1, θ2 and φ krespectively represent the parameters of diffusion model 1, diffusion model 2 and K action value networks;

[0081] policy network policy network Each of the policy network and the action value network is a multi-layer perceptron structure with 2 hidden layers and 256 hidden neurons, wherein: the input dimension of each diffusion model in the policy network is the state dimension, and the output dimension is the action dimension; the input dimension of each action value network in the ensemble network is the sum of the state dimension and the action dimension, and the output dimension is 1; the network parameters are initialized randomly.

[0082] Step 2, establish a target network and initialize the network parameters.

[0083] respectively establish the target network of diffusion model 1, diffusion model 2 and K action value networks and wherein: θ'1, θ'2 and φ' k respectively represent the parameters of the target network corresponding to diffusion model 1, diffusion model 2 and K action value networks; the target network structure is consistent with the original network structure, and the initialization method of the target network parameters is: the parameters (θ1, θ2, φ k ) of the corresponding original network are directly assigned to the parameters (θ'1, θ'2, φ' k ) of the target network.

[0084] Step 3, randomly extract samples from the offline data set and input them into the ensemble network, the policy network and the target network.

[0085] First, randomly extract samples (s, a, r, s') from the offline data set ; wherein, the sample (s, a, r, s') represents the current state s of the intelligent vehicle, the action a executed by the policy network, the immediate reward r, and the state s' converted to the next time;

[0086] In this specific embodiment, the current state s of the intelligent vehicle includes state information such as steering wheel value, heading angle, speed, distance from the intelligent vehicle body to the two sides of the road, distance from the intelligent vehicle body to the surrounding obstacles, etc., and the action a includes throttle and brake values, steering wheel turning angle.

[0087] Then, input (s, a) in the sample into the ensemble network and the policy network , and input (r, s') into the target ensemble network , input s' into the target policy network respectively. ​​

[0088] Step 4, updating the integrated network of network parameters.

[0089] By sampling mini-batch samples from the offline dataset, the parameters of each action value network in the integrated network are updated using gradient descent method to minimize the time difference error of the action value network, i.e., the square difference between the current Q value and the target Q value combined with the reward value, the specific steps are as follows:

[0090] First, according to the generation mechanism of the diffusion model:

[0091]

[0092] Where a i-1 |a i represents the action generation at one time step in the inverse diffusion process, i.e., generating the action a i at the previous time step according to the action a i-1 at the current time step a i , β i represents the variance adjustment scheme, which controls the noise level in the diffusion process, α i = 1-β i , ∈ θ represents the noise prediction model, which is composed of a neural network with 2 hidden layers, s represents the state input to the diffusion model, represents the cumulative value of α i , where i represents the current time step of the diffusion process, t represents a time step before the current time step i, represents a standard Gaussian noise;

[0093] s′ is input into the target policy network and to generate actions

[0094] Then, according to the definition of uncertainty u(s,a):

[0095]

[0096] Where Q k (s,a) is the kth action value function, is the average action value;

[0097] The uncertainty value of action is calculated using the target network of the integrated network :

[0098] ​

[0099] where, denotes the target integrated network the kth action value function, denotes the average action value comparison action the uncertainty value u'1 of action the uncertainty value u'2 of action

[0100]

[0101] denotes the target value finally determined according to the uncertainty value, denotes an indicator function;

[0102] Finally, by minimizing the time difference error of the output value of each action value network in the integrated network and the joint reward value of the target value, the parameters of each action value network are updated, that is, the loss function is minimized as follows:

[0103]

[0104] where, denotes the number of experience samples in a small batch, here when u'1≤u'2, otherwise

[0105] The parameters φk of the kth action value network are updated by gradient descent method, and the adjustment amount of the parameters φk is: k k

[0106]

[0107] where, e φ denotes the learning rate in the gradient descent process, here e φ =0.0003, denotes the derivative of the network parameters in the kth action value network.

[0108] Step 5, update the policy network parameters.

[0109] The update of the policy network parameters mainly optimizes three aspects of loss: policy regularization, policy improvement and uncertainty regularization, as follows:

[0110] First, the policy regularization loss is equivalent to the behavior cloning loss, that is:

[0111]

[0112] ​​Where j = {1, 2} represents the number of the two independent diffusion models, and ∈ represents standard Gaussian noise. Represents a noise prediction model. Indicates the proportion control of noise;

[0113] Then, regarding improving the quality of policy-generated actions, i.e., policy improvement items, are:

[0114]

[0115] in, This represents the action generated by the diffusion model based on state s, i.e.

[0116] Finally, regarding ensuring the safety of policy generation actions, the uncertainty regularization term is:

[0117]

[0118] Finally, the following loss function is minimized using gradient descent:

[0119]

[0120] Where η and σ are balance coefficients, η = 0.01 and σ = 1;

[0121] Using gradient descent to calculate parameter θ j Update the parameter θ j The adjustment amount is:

[0122]

[0123] Among them, l θ This represents the learning rate during the gradient descent process, where e θ =0.003, This indicates that the derivatives are taken with respect to the network parameters in the two independent diffusion models.

[0124] Step 6: Update the target network parameters.

[0125] First, calculate respectively: ρθ1+(1-ρ)θ′1, ρθ2+(1-ρ)θ′2 and ρφ k +(1-ρ)φ′ k ;

[0126] Then, the results of the above calculations are assigned to: θ′1, θ′2, and φ′ respectively. k ;

[0127] That is, θ′1=ρθ1+(1-ρ)θ′1

[0128] θ′2=ρθ2+(1-ρ)θ′2

[0129] φ' k = p phi k + (1-p) phi' k

[0130] wherein p represents the target network update rate, here p = 0.005.

[0131] Step 7, repeat steps 3 to 6, constantly update each network parameter, the number of updates is not less than 1x10 6 times, and the final updated policy network is used as the optimal policy for the deployment of the intelligent vehicle.

[0132] Step 8, use the uncertainty measure to select a relatively safe action to perform when deploying the algorithm, that is, by integrating the network to evaluate the actions given by the two diffusion models, then select the action with lower uncertainty to perform:

[0133]

[0134] a selected is the action finally selected to perform, s E is the real-time state obtained when the intelligent vehicle interacts with the environment,

[0135] The method of the present application is a safe offline reinforcement learning method that does not need to interact with the environment during training, which can ensure the safety of the policy when deployed in an autonomous driving environment. Specifically, to solve the problem that traditional offline reinforcement learning may lead to unsafe behavior in autonomous driving tasks due to extrapolation error, two diffusion models are proposed as alternative strategies, and an integrated network containing multiple action value networks is used as a policy evaluation network; by training the two diffusion models and the integrated network in an offline environment, the integrated network is used to evaluate the uncertainty of the actions generated by the two strategies during the deployment phase, and the strategy with lower uncertainty is selected as the final driving strategy, thereby ensuring the safety of the autonomous driving task during the deployment phase.

[0136] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, several improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A safe autonomous driving method based on a dual-alternative diffusion strategy, characterized in that, First, two diffusion models are constructed as policy networks, and an ensemble network containing multiple action value networks is used as the policy evaluation network. Second, the two diffusion models and the ensemble network are trained in an offline environment to obtain two trained diffusion models and a trained ensemble network. Then, during the deployment phase, the trained ensemble network is used to evaluate the uncertainty of the actions generated by the two trained diffusion models; finally, the action with lower uncertainty is selected for execution; the specific steps are as follows: Step 1: Construct the policy network and the integration network, and initialize the network parameters of the policy network and the integration network; The two diffusion models are diffusion models. and diffusion model The two diffusion models constitute a policy network; Establish an integrated network containing K action value networks. As the policy evaluation network; diffusion model diffusion model The parameters of the K action value network are respectively adopted , and express; Step 2, construct diffusion models respectively diffusion model And the target policy network of the K action value network , and target integration network ;in: , and Represent the target policy network respectively , and target integration network Network parameters; The initialization method for the target network parameters is as follows: initialize the parameters of the corresponding original network... Parameters directly assigned to the target network ; Step 3: Randomly sample samples from the offline dataset and input them into the ensemble network, policy network, and target network to test the diffusion model in an offline environment. diffusion model Train the diffusion model with K action value networks to update the diffusion model. diffusion model Integrated Network The parameters of the target network and the target network are obtained; two trained diffusion models and a trained ensemble network are obtained. Step 3.1, from the offline dataset Randomly select samples Among them, the sample In this context, s represents the current state of the intelligent vehicle, a represents the action performed by the intelligent vehicle through the policy network, r represents the immediate reward obtained by the intelligent vehicle, and s′ represents the next state of the intelligent vehicle. In the sample Input to the integrated network respectively diffusion model With diffusion model In the middle, put Input to target integration network In each action value network, The inputs are respectively fed into the target policy network. and middle; Step 3.2, Update the integrated network Network parameters first, The inputs are respectively fed into the target policy network. and Generate actions respectively , ; Then, the target network of the integrated network is utilized. Calculate the actions separately , Uncertainty value , ; Comparison of actions Uncertainty value With action Uncertainty value and according to the smaller Select target value ; Indicates an indicator function; Finally, the parameters of each action value network are updated by minimizing the time difference error between the output value and the joint reward value of each action value network in the ensemble network, i.e., minimizing the following loss function: in, This represents the number of empirical samples in the mini-batch. , ,when hour, ,otherwise ; Using gradient descent to evaluate the parameters of the value network for the k-th action Update parameters The adjustment amount is: in, This represents the learning rate during the gradient descent process; Step 3.3, Update the policy network and The network parameters are as follows: The following loss function is minimized using gradient descent: in, , indicating the numbers of the two independent diffusion models, and For balance coefficient, This represents the loss due to policy regularization. Indicates the strategy improvement item. Represents a regularization term indicating uncertainty; Using gradient descent to evaluate parameters Update parameters The adjustment amount is: in, This represents the learning rate during the gradient descent process. This indicates that the derivatives are taken with respect to the network parameters in two independent diffusion models. Step 3.4, update the target network parameters; First, calculate separately: , and ; Then, the results of the above calculations are assigned to: , and ; Right now, in, Indicates the target network update rate; Step 3.5: Repeat steps 3.1 to 3.4 to continuously update the network parameters to obtain the trained ensemble network and the two trained diffusion models. Step 4: In the deployment phase, the trained ensemble network and the two trained diffusion models are used for the deployment of the intelligent vehicle. The trained ensemble network is used to evaluate the uncertainty of the actions generated by the two trained diffusion models, and the actions with lower uncertainty are selected for execution.

2. The safe autonomous driving method based on a dual-alternative diffusion strategy according to claim 1, characterized in that, The target network utilizing integrated networks Calculate the actions separately , Uncertainty value , The details are as follows: in, Indicates target integration network The value function of the k-th action, This represents the average value of an action.

3. The safe autonomous driving method based on a dual-alternative diffusion strategy according to claim 1, characterized in that, Strategy regularization loss in step 3.3 Equivalent to behavioral cloning loss, i.e.: in, Indicates standard Gaussian noise. Represents a noise prediction model. ,express The cumulative value; Then, the strategy improvement items are: in, The diffusion model is based on the state. The generated action, i.e. ; Finally, regarding ensuring the safety of policy generation actions, the uncertainty regularization term is: 。 4. The safe autonomous driving method based on a dual-alternative diffusion strategy according to claim 1, characterized in that, In step 3.3, the balance coefficient in the loss function , ; Learning rate during gradient descent ; In step 3.4, the target network update rate .

5. The safe autonomous driving method based on a dual-alternative diffusion strategy according to claim 1, characterized in that, In step 4, the action with lower uncertainty is selected for execution, which is represented as: The action that the intelligent vehicle ultimately chooses to execute. This refers to the real-time status obtained when the intelligent vehicle interacts with its environment. , .

6. The safe autonomous driving method based on a dual-alternative diffusion strategy according to claim 1, characterized in that, Current status of intelligent vehicle This includes steering wheel values, heading angle, speed, distances from both sides of the intelligent vehicle to the sides of the road, and distances from the intelligent vehicle to surrounding obstacles; action This includes accelerator and brake settings, as well as steering wheel settings; The values ​​for the throttle and brake are in the range of [-1, 1]; When the accelerator and brake values ​​are in the range of [-1, 0), it indicates that the intelligent vehicle is in braking mode. When the accelerator and brake values ​​are equal to -1, it indicates that the braking force is at its maximum. When the accelerator and brake values ​​are [0,1], it indicates that the intelligent vehicle is in the accelerator mode; when the accelerator and brake values ​​are equal to 1, it indicates that the accelerator is at its maximum. The value range of the steering wheel is [-1, 1], where [-1, 0) represents turning the steering wheel to the left and -1 represents turning the steering wheel all the way to the left; [0, 1] represents turning the steering wheel to the right and 1 represents turning the steering wheel all the way to the right.

7. The safe autonomous driving method based on a dual-alternative diffusion strategy according to claim 1, characterized in that, Both the diffusion model and the action value network contain a multilayer perceptron structure with two hidden layers and 256 neurons in each hidden layer.

Images