A dynamic action-based stochastic decision transformation method

Abstract

The application discloses a random decision transformation method based on dynamic action, comprising the following steps of constructing an offline environment data set, a Critic network model, a model pi θ , a context length, a maximum iteration number, an expected loss parameter, a maximum decision action number candidate return value, and obtaining an optimal strategy through pre-training of a randomness network and a multi-action output model. The purpose of reaching the optimal driving strategy is to help the vehicle to evaluate the complexity of the current traffic environment in the automatic driving environment, to make the action space of the model as wide as possible, to select the optimal action, and to find the optimal driving strategy.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of information technology, and particularly relates to a random decision transformation method based on dynamic actions. BACKGROUND

[0002] Reinforcement learning is a method of learning an optimal policy by interacting with an environment, in which an agent adjusts its policy to maximize cumulative rewards by taking actions to obtain rewards. Common classical reinforcement learning methods include value-based methods, policy-based methods and Actor-Critic methods. However, in many fields such as autonomous driving, robot control, medical health and the like, the interaction with the environment requires high costs and a large amount of time, and online reinforcement learning becomes impractical. Offline reinforcement learning aims to learn the policy of an agent through pre-collected data without interaction, and typical algorithms include IQL, CQL and the like. However, offline reinforcement learning faces distribution shift problems, and the learned policy is often conservative, and performs poorly in handling long-term sequential dependencies and complex policies. SUMMARY

[0003] The application aims to solve the problems in the prior art that the interaction with the environment requires high costs and a large amount of time, online reinforcement learning becomes impractical, offline reinforcement learning faces distribution shift problems, and the learned policy is often conservative and performs poorly in handling long-term sequential dependencies and complex policies.

[0004] To solve the above problems, the application provides a random decision transformation method based on dynamic actions, comprising the following steps:

[0005] S1: creating input parameters;

[0006] S2: pre-training a randomness network;

[0007] S3: training a multi-action output model;

[0008] S4: the randomness network gives a standard Gaussian distribution of the next state and reward according to the current state and action, calculates the uncertainty by comparing the estimated value and the actual value, and transmits the uncertainty to the DecisionTransformer, and the estimated reward value is used as a penalty item of the reward value to fine-tune, and finally the decision tree model determines the number of generated actions by using the uncertainty, and the Critic network evaluates the final action to make an optimal selection;

[0009] S4-1: giving an initial state of an environment , initializing the number of actions , the loop is executed for each iteration i from 1 to n candidate rewards, a number of action pairs are sampled for each candidate reward, n iterations are ended, and the optimal action is obtained ;

[0010] S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , the environment is advanced to obtain the state of the next moment and reward , finally output the result as true to end the loop, use the trained model to process the new input offline environment data set D, and dynamically generate the optimal driving strategy;

[0011] Step S1 creates input parameters, which include: offline environment data set D, network Q function , randomly initialize the parameters of the Decision Transformer model , context length K, maximum iteration number T, expected loss parameter , maximum number of decision actions , n candidate reward values ;

[0012] The offline environment data set D includes: weather, road conditions and traffic participant behavior;

[0013] Weather includes: sunny, rainy, snowy, foggy, overcast;

[0014] Road conditions: highway, urban road, rural road, mountain road, construction area;

[0015] Traffic participant behavior: pedestrian crossing, bicycle riding, vehicle lane changing, vehicle overtaking, emergency braking;

[0016] Step S2 pre-trains the random network, which includes:

[0017] S2-1: data processing;

[0018] Convert weather, road conditions and traffic participant behavior data into state, action and reward value samples , which includes:

[0019] State value : generated according to the combination of weather, road conditions and traffic participant behavior, and the specific combination form is: sunny + urban road + pedestrian crossing;

[0020] Action value : the decision action of the vehicle in the current state, and the specific action is: deceleration, lane change, parking;

[0021] Reward Value : Evaluate the effectiveness of the action, specifically: safety (avoiding collisions) and efficiency (saving time);

[0022] S2-2: Construction, training, and optimization of stochastic network models;

[0023] For each iteration i from 1 to T, a batch of samples is sampled from the offline environment dataset D. According to the formula:

[0024]

[0025] Obtain the Gaussian distribution at the next time step. In the formula Representing state and actions The mean of the following, Representing state and actions The standard deviation below is used to minimize the loss function to optimize the stochastic network, and the formula is:

[0026]

[0027] The loss function is divided into two parts, the first part... This represents the mean of the model's predictions. The first part represents the true value; the second part is the regularization parameter, where... Indicates the number of samples. This indicates the number of dimensions in the output; each sample j has M features. Is the model for the first The predicted mean of the k-th feature of each sample, with parameter . It is the actual value, i.e., the target value; These are hyperparameters related to variance, representing the upper and lower bounds of variance, respectively; Represented by natural numbers The iteration ends by optimizing the stochastic network by minimizing the loss function, which is an exponential function with a base of 0.

[0028] In the preferred approach, the multi-action output model training in step S3 includes:

[0029] For each iteration i, which ranges from 1 to N, a batch of samples is sampled from the offline environment dataset D. , The reward value is the sum of the reward values ​​from the current moment until the end of the environment, expressed by the formula:

[0030]

[0031] renew network, wherein represents the Q-function updated in the k+1 iteration, the Q-function represents the expected return of taking action a in state s, represents the Bellman operator, the target Q-value is obtained based on the true return, represents the parameter that minimizes, refers to finding the parameter that makes a certain function reach a minimum value; represents the expectation over state s; by formula:

[0032]

[0033] obtain the predicted value of the next state and reward , wherein represents the transition function, represents the probability distribution of the system transitioning to the next state and obtaining the reward given the current state and action ; represents the normal distribution, the distribution of state transition and reward is normal given the state and action;

[0034] by formula:

[0035]

[0036] estimate the uncertainty of the environment , wherein represents the expected loss, the loss is weighted by adjusting the size of , by formula:

[0037]

[0038] obtain the dynamically adjusted number of actions , the outer min function ensures that the result does not exceed the maximum number of decision actions , the inner max function ensures that the result is greater than or equal to 1, represents the pre-set maximum uncertainty value, for each iteration i from 1 to K, the model is forward propagated in an autoregressive manner to obtain a number of action pairs, by formula:

[0039]

[0040] represents a condition, the action selected at time step is sampled from the SeaDT model with randomly initialized parameters, with the condition that the action is selected at time step The received reward value sequence, at time step to The state sequence, at time step to The selected action sequence and the number of actions , and finally end the K iterations;

[0041] By the formula:

[0042]

[0043] Minimize the loss function to optimize the strategy model, end N iterations, and return the optimized optimal model ; wherein, represents the loss function, and represents the optimization target of the model; represents an exponential function, represents a Softmax soft optimization, which ensures that the weights occupied by Q values add up to 1, is the expected loss, wherein represents the predicted action at time step i; represents the true action at time step i.

[0044] In the preferred mode, step S4 comprises:

[0045] S4-1: Given the initial state of the environment , initialize the number of actions , and loop for each iteration i from 1 to n, according to n candidate rewards, sample a number of action pairs for each candidate reward, and the formula is:

[0046]

[0047] End iterations, according to the formula:

[0048]

[0049] Obtain the optimal action ;

[0050] S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , and advances the environment to obtain the state and the reward at the next time Update the reward value, and the formula is:

[0051]

[0052] Finally, the output result is true to end the loop, and the trained model is used to process the new input offline environment data set D to dynamically generate the optimal driving strategy.

[0053] The beneficial effects of the present application are as follows: the next moment state of the environment is predicted based on the model-based network, and compared with the real state of the next sample, so as to obtain the uncertainty of the current moment of the model. In autonomous driving, this can help the vehicle to evaluate the complexity of the current traffic environment. The Critic network and the multi-action output decision converter improve the output dimension of the Decision Transformer, try to generate several action pairs from a single action, so that the action space of the model is as wide as possible, at the same time, the Critic network is trained at the same time, so that the model selects the optimal action from these action pairs, and skillfully combines the long sequence prediction ability of the Transformer model and the short-term prediction ability of the Critic network. The dynamic action output of the model, combined with the uncertainty estimation network, can estimate the uncertainty value of each moment, and the threshold of uncertainty and the maximum output number of action can be set, so that the number of model output can be linearly and dynamically controlled, and it is in line with intuition. In autonomous driving, when the uncertainty of the environment is high, the vehicle is expected to be conservative and choose the strategy that can effectively score as much as possible, and vice versa when the uncertainty is very low, the vehicle is encouraged to explore to find the optimal driving strategy. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 It is the overall architecture schematic diagram of the random dynamic action decision converter of the present application;

[0055] Figure 2 It is the model architecture schematic diagram of the random network of the present application;

[0056] Figure 3 It is the schematic diagram of the automatic driving scene of the random state transition of the present application. DETAILED DESCRIPTION

[0057] Embodiment 1:

[0058] A random decision transformation method based on dynamic action, comprising the steps of:

[0059] S1: creating input parameters;

[0060] S2: pre-training a random network;

[0061] S3: multi-action output model training;

[0062] S4: based on the above method, the specific work flow is explained;

[0063] S4-1: given the initial state of the environment , initialize the number of actions , loop for each iteration i from 1 to n, according to n candidate rewards, sample a number of action pairs for each candidate reward, end n iterations, and obtain the optimal action ;

[0064] S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , advance the environment to get the state of the next moment and reward , finally output the result as true end the loop, use the trained model to process the new input offline environment data set D, dynamically generate the optimal driving strategy.

[0065] Step S1 creates input parameters, specific methods include:

[0066] Offline environment data set D, Critic network Q function , randomly initialize the parameters of the DecisionTransformer model π θ , context length K, maximum iteration number T, expected loss parameter , maximum number of decision actions N max , n candidate reward values ;

[0067] The offline environment data set D includes: weather, road conditions and traffic participant behavior;

[0068] Weather includes: sunny, rainy, snowy, foggy, overcast;

[0069] Road conditions: highway, urban road, rural road, mountain road, construction area;

[0070] Traffic participant behavior: pedestrian crossing, bicycle riding, vehicle lane changing, vehicle overtaking, emergency braking.

[0071] Step S2 specifically includes:

[0072] S2-1: data processing;

[0073] Convert weather, road conditions and traffic participant behavior data into state, action and reward value samples , specific steps include:

[0074] State value : generated according to the combination of weather, road conditions and traffic participant behavior, specific combination form: sunny + urban road + pedestrian crossing;

[0075] Action value : the decision action of the vehicle in the current state, and the specific action is: deceleration, lane changing, parking;

[0076] : the reward value : evaluate the effect of the action, and the specific effect is: safety to avoid collision, efficiency to save time;

[0077] S2-2: random network model construction, training and optimization;

[0078] For each iteration i from 1 to T, a batch of sample sets is sampled from the offline environment data set D

[0079]

[0080] The Gaussian distribution of the next moment is obtained , wherein represents the mean value under the state and action , represents the standard deviation under the state and action , and the loss function is minimized to optimize the random network, and the formula is:

[0081]

[0082] The loss function is divided into two parts, the first part represents the mean value predicted by the model, represents the true value; the second part is the regularization parameter term, wherein N represents the number of samples, M represents the number of output dimensions, each sample j has M features, is the predicted mean value of the kth feature of the jth sample by the model, and the parameter is , is the true value, i.e. the target value; and are variance-related hyperparameters, representing the upper and lower limits of the variance, respectively; exp represents the exponential function with natural number e as the base, and finally the random network is optimized by minimizing the loss function, and the iteration is ended.

[0083] The multi-action output model training of step S3 includes:

[0084] For each iteration i from 1 to N, a batch of sample sets is sampled from the offline environment data set D represents the reward value, i.e. the sum of the reward values from the current moment to the end of the environment, and the formula is:

[0085]

[0086] updating the Critic network, wherein denotes the Q-function updated at iteration k+1, Q-function denotes the expected return of taking action a in state s, denotes the Bellman operator, is computed based on the true return for the target Q-value, arg denotes the parameter that minimizes or arg denotes the parameter that maximizes a certain function; denotes the expectation over state s; D denotes the experience replay pool or dataset storing historical state, action, and reward information; is obtained by:

[0087]

[0088] obtaining the predicted value of the next state and reward , wherein denotes the transition function, denotes the probability distribution of the system transitioning to the next state and obtaining the reward given the current state and action ; denotes the normal distribution, the distribution of state transition and reward is normal given the state and action;

[0089] is obtained by:

[0090]

[0091] estimating the uncertainty of the environment , wherein denotes the expected loss, the loss is weighted by adjusting the size of , is obtained by:

[0092]

[0093] obtaining the dynamically adjusted number of actions , the outer min function ensures that the result will not exceed N max , the inner max function ensures that the result is greater than or equal to 1, U max denotes the pre-set maximum uncertainty value, is obtained by:

[0094]

[0095] updating the reward value after the penalty, for each iteration i from 1 to k, the model is forward propagated in an autoregressive manner to obtain a number of action pairs, is obtained by:

[0096]

[0097] represents a condition that the time step selected action is sampled from the SeaDT model with randomly initialized parameters , conditioned on the sequence of reward values received at time steps t to t+i, the sequence of states at time steps t to t+i, and the sequence of actions selected at time steps t to t+i-1 and the number of actions , and finally end the K iterations; by the formula:

[0098]

[0099] minimizing the loss function to optimize the policy model, end the N iterations, and return the optimized optimal model π θ ; wherein L represents the loss function, and represents the optimization objective of the model; e represents the exponential function, and the summation in the first half of the formula represents the Softmax soft optimization, which ensures that the weights occupied by the Q values are summed to 1, and the second half of the formula is the expected loss, wherein represents the predicted action at time step i; represents the true action at time step i.

[0100] Step S4 includes:

[0101] S4-1: Given the initial state of the environment , initialize the number of actions , and loop for each iteration i from 1 to n, sample a number of action pairs for each candidate reward according to the n candidate rewards, and the formula is:

[0102]

[0103] end the n iterations, and according to the formula:

[0104]

[0105] obtain the optimal action ;

[0106] S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , and advances the environment to obtain the state at the next time and the reward update the return value, and the formula is:

[0107]

[0108] Finally, output the result as true to end the loop, and use the trained model to process the new input offline environment dataset D to dynamically generate the optimal driving strategy.

[0109] As Figure 1 shown, the stochastic network gives a standard Gaussian distribution of the next state and reward according to the current state and action. By comparing the estimated value and the actual value, the uncertainty is calculated, and this uncertainty is passed to the DecisionTransformer, while the estimated reward value is used as a penalty term for the reward value to fine-tune. Finally, the decision tree model determines the number of final actions generated by the uncertainty, and the Critic network evaluates the final action to make the optimal choice.

[0110] In autonomous driving applications, vehicles can make dynamic adjustments in complex traffic environments through this architecture to adapt to sudden road conditions and traffic signal changes.

[0111] As Figure 2 shown, Dropout layers and normalization layers are added to the integrated network to obtain more accurate uncertainty estimates. By performing K forward propagations on , the final Gaussian distribution can be obtained. A small autonomous driving stochastic environment is simulated to demonstrate the limitations of DT in splicing problems and stochastic problems, and to explain why these problems exist and how the model solves them. This example also demonstrates the essential causes of these problems. In this environment, the specific details are annotated in the diagram. The limitations of DT will be explained from the aspects of splicing problems and randomness.

[0112] As Figure 3 shown, the middle part is the training sample, and the right side is the result after the model is trained. In the figure, different numbers represent different states, and the available actions from each state are distinguished by the arrow shape of the line extending from the state. The circular arrow indicates that it will bring a positive reward value, and the pointing arrow indicates that it will bring a negative reward. Entering state 8 will bring a reward of 0, and the solid and dashed lines represent whether the environment is random. The dashed line indicates that performing this action will have a random result, which may transfer to one of several states. The density of the dashed line represents the probability of transfer, and the higher the density, the greater the probability. To illustrate extreme cases, a state 11 is introduced at the bottom right of the figure, which will bring a reward of 10 points, but the probability of occurrence is extremely low. And intentionally put it into the training set, hoping that the model will as much as possible to walk the circular trajectory to obtain the optimal solution.

[0113] Splicing problem: Take the first two in the training trajectory as an example to explain the splicing problem of DT. First, ignore the randomness in the environment, and can be in Figure 3As can be seen from the trajectory starting from state 0, the optimal state trajectory should be "0->1->4->7", but this trajectory is not contained in the training data of the DT, instead, the training trajectory contains a subset of the optimal trajectory, that is, "0->1->4" and "4->7", but unfortunately, due to the loss function design of the DT, even if the DT can generate according to the optimal trajectory in the first half, when it enters state 4, it will still tend to the behavior policy, resulting in a very low final return. Fortunately, the critic network solves this problem well. At each time, the critic network calculates the future return to correct the error of the DT, and the model framework can output multiple actions, and the critic network judges each action to finally select the optimal action.

[0114] Randomness problem: a randomness experiment is also designed, that is, the environment shows randomness at the beginning, sometimes performing a circular action and not necessarily entering state "1", but entering state "5". In this state, two representative data are also selected to illustrate the randomness problem, for example, Figure 2 As can be seen from the training trajectory, it can be seen that the training set exhibits differences after state 5. Both trajectories can represent the optimal trajectory, but due to the randomness of the environment, the DT does not know the probability of entering state "11". Even if there is a failure sample in the training sample that enters state "9", the reward of entering state "11" is still high, which leads the DT to blindly pursue high returns, which is counterproductive, resulting in a high probability of entering state "9" in the reasoning phase, and the reward becomes low. However, due to the existence of the critic network, when there are too many failure samples, the critic will naturally reduce the expectation of the action, and in the seaDT framework, the probability of naturally selecting the action will decrease, which is a dynamic adjustment strategy based on the critic.

[0115] Essential reasons: The main reason that SeaDT can solve the above problems lies in that it can effectively integrate the advantages of DT and critic, taking the advantages and making up the disadvantages. DT is good at processing long sequence data, so it can effectively integrate information and condense a better set of action space. The critic network can calculate the expected return of the action based on the above DT integration, so as to obtain the optimal solution. At the same time, the dynamic action output of SeaDT is also a key link for success. As shown in state "3", it has 4 optional actions, but if the training sample is insufficient, DT may be limited to the samples in the training set, and lack the possibility of exploration. When the uncertainty is high, the model is expected to be conservative and choose the strategy that can effectively score as much as possible. Conversely, when the uncertainty is very low, the model is encouraged to explore, so as to find the optimal action. In this way, SeaDT can effectively deal with randomness and splicing problems in complex autonomous driving environments, and improve the decision-making ability and safety of the agent. In the application of autonomous driving, the vehicle can make dynamic adjustments in the complex traffic environment through this architecture to adapt to sudden road conditions and traffic signal changes.

[0116] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A dynamic action based stochastic decision transformation method, characterized by, Comprise: S1: create input parameters; S2: pre-training randomness network; S3: multi-action output model training; S4: the randomness network gives the next state and the standard Gaussian distribution of the reward according to the current state and action, calculates the uncertainty by comparing the estimated value and the actual value, and transmits this uncertainty to the Decision Transformer, at the same time the estimated reward value will be used as a penalty item of the reward value to fine-tune, finally the decision tree model determines the final generated action quantity by using the uncertainty, and the Critic network evaluates the final action to make the optimal choice; S4-1: Given the initial state of the environment , initialize the number of actions , loop for each iteration i from 1 to n, for each candidate reward sample a number of action pairs, end n iterations, obtain the optimal action ; S4-2: Uncertainty of the computing environment , update the number of actions , the environment executes the optimal action , advance the environment to get the state of the next moment and reward , finally output the result as true to end the loop, and use the trained model to process the new input offline environment data set D to dynamically generate the optimal driving strategy; Step S1 creates an input parameter specific method including: an offline environment dataset D, network Q function , a Decision Transformer model with randomly initialized parameters , a context length K, a maximum number of iterations T, and an expected loss parameter , a maximum number of decision actions , n candidate return values ; The offline environment data set D includes: weather, road conditions and traffic participant behavior; Weather includes: sunny, rainy, snowy, foggy, overcast; Road conditions: highway, urban road, rural road, mountain road, construction area; Traffic participant behavior: pedestrian crossing, bicycle riding, vehicle lane changing, vehicle overtaking, emergency braking; The specific method of step S2 pre-training randomness network includes: S2-1: data processing; Converting weather, road condition, and traffic actor behavior data into state, action, reward value samples The specific steps include: State value : Generated according to the combination of weather, road conditions and traffic participant behavior, specific combination form: sunny + urban road + pedestrian crossing action value : decision action of the vehicle in the current state, the specific action is: deceleration, lane change, parking; reward value : evaluate the effect of the action, the specific effects are: safety avoid collision, efficiency save time; S2-2: randomness network model construction, training and optimization; For each iteration i from 1 to T, sample a batch set of samples from the offline environment dataset D , according to the formula: Obtain the Gaussian distribution at the next time step. In the formula Representing state and actions The mean of the following, Representing state and actions The standard deviation below is used to minimize the loss function to optimize the stochastic network, and the formula is: Wherein, the loss function is divided into two parts, the first part represents the mean value predicted by the model, represents the true value; the second part is the regularization parameter term, wherein, represents the number of samples, represents the number of dimensions of the output, each sample j has M features, is the predicted mean value of the kth feature of the model for the jth sample, is the parameter, is the true value, i.e. the target value; is a hyperparameter related to variance, representing the upper and lower limits of variance, respectively; represents the exponential function with base natural number , and finally the random network is optimized by minimizing the loss function, ending the iteration.

2. The dynamic action based stochastic decision transformation method of claim 1, wherein, The multi-action output model training of step S3 includes: For each iteration i from 1 to N, sample a batch set of samples from the offline environment dataset D , represents the return value, i.e., the sum of the reward values from the current time instant until the end of the environment, by the formula: updating network, wherein denotes the Q-function updated in the k+1 iteration, the Q-function denoting the expected return of taking action a in state s, denotes the Bellman operator, the target Q-value is obtained based on the true return, denotes the parameters that are to be minimized, referring to the parameters that are sought to bring a certain function to a minimum; denotes the expectation over state s; by the formula: obtaining a prediction of the next state and reward where denotes the transition function, denotes the probability distribution of the system transitioning to the next state and obtaining the reward given the current state and action ; denotes that the distribution of the state transition and reward is normal given the state and action; Through the formula: Estimating uncertainty of an environment wherein represents the expected loss, which is weighted to the loss by adjusting the size of the loss by the formula: Obtaining a dynamically adjusted number of actions The min function of the outer layer ensures that the result does not exceed the maximum number of decision actions The max function of the inner layer ensures that the result is greater than or equal to 1, denotes a pre-set maximum uncertainty value, for each iteration i from 1 to K, the model is forward propagated in an autoregressive manner to obtain a number of action pairs, by means of the formula: This represents a condition at time step. Selected action The SeaDT model with randomly initialized parameters Mid-sampling, condition is at time step arrive The sequence of received reward values, at time step arrive The state sequence, and at time step arrive Selected action sequence and number of actions Finally, the K iterations are completed. Through the formula: Minimizing the loss function optimizes the policy model, ends N iterations, and returns the optimized optimal model ; wherein, represents the loss function, representing the optimization objective of the model; represents an exponential function, represents a Softmax soft optimization, ensuring that the weights occupied by Q values add up to 1, is an expected loss, wherein represents the predicted action at time step i; represents the true action at time step i.

3. The dynamic action based stochastic decision transformation method of claim 1, wherein, Step S4 includes: S4-1: Given the initial state of the environment , initialize the number of actions , loop for each iteration i from 1 to n, according to n candidate rewards, sample a number of action pairs from each candidate reward, formula: End sub-iterations, according to the formula: Obtaining optimal actions ; S4-2: Uncertainty of the environment , number of update actions , environment performs optimal action , advance environment to next time step and reward update return value, formula: Finally, the output result is true to end the loop, and the trained model is used to process the new input offline environment data set D to dynamically generate the optimal driving strategy.

Images