Random decision transformation method based on dynamic action

Abstract

The invention discloses a random decision transformation method based on dynamic actions, which comprises the following steps of: constructing an offline environment data set, a Critic network model, a model pi theta, a context length, a maximum iteration number, an expected loss parameter and a maximum decision action quantity candidate return value, and training through a pre-training randomness network and a multi-action output model to obtain an optimal strategy. The purposes of helping the vehicle to evaluate the complexity of the current traffic environment in the automatic driving environment, enabling the motion space of the model to be as wide as possible, selecting the optimal motion and finding the optimal driving strategy are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and specifically relates to a random decision transformation method based on dynamic actions. Background Art

[0002] Reinforcement learning is a method of learning an optimal policy by interacting with the environment. In reinforcement learning, an agent takes actions to obtain rewards, and then adjusts its policy to maximize the cumulative reward. Common classical reinforcement learning methods include value-based methods, policy-based methods, and Actor-Critic methods. However, in many fields, such as autonomous driving, robot control, and healthcare, interacting with the environment often requires high costs and a large amount of time overhead, and online reinforcement learning often becomes impractical. Offline reinforcement learning aims to learn the agent's policy through pre-collected data without interaction. Typical algorithms include IQL, CQL, etc. However, offline reinforcement learning will face the problem of distribution shift, and the learned policy is often conservative, and it performs poorly in dealing with long-term temporal dependencies and complex policies. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the prior art that interacting with the environment requires high costs and a large amount of time overhead, online reinforcement learning becomes impractical, offline reinforcement learning will face distribution shift, the learned policy is often conservative, and it performs poorly in dealing with long-term temporal dependencies and complex policies.

[0004] To solve the above problems, the present invention proposes a random decision transformation method based on dynamic actions, including the steps of: S1: Create input parameters; S2: Pre-train a randomness network; S3: Train a multi-action output model; S4: Describe the specific working process based on the above method; S4-1: Given the initial state of the environment , initialize the number of actions , loop and execute for each iteration i from 1 to n, sample a number of action pairs according to n candidate rewards, end the n iterations, and obtain the optimal action ; S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , advance the environment to obtain the state at the next moment and the reward , finally, the output result is true to end the loop, and the trained model is used to process the newly input offline environment dataset D to dynamically generate the optimal driving strategy.

[0005] In the preferred mode, the specific method for creating the input parameters in step S1 includes: Offline environment dataset D, Critic network , the SeaDT model π with randomly initialized parameters θ , context length K, maximum number of iterations T, N, expected loss parameter , maximum number of decision-making actions N max , n candidate return values ; The offline environment dataset D includes: weather, road conditions, and traffic participant behaviors; Weather includes: sunny, rainy, snowy, foggy, cloudy; Road conditions: highway, urban road, rural road, mountain road, construction area; Traffic participant behaviors: pedestrians crossing the road, cycling, vehicle lane change, vehicle overtaking, emergency braking.

[0006] In the preferred mode, step S2 specifically includes: S2-1: Data processing; Convert weather, road condition, and traffic participant behavior data into state, action, and reward value samples , and the specific steps include: State value : Generated according to the combination of weather, road conditions, and traffic participant behaviors, and the specific combination form is: sunny + urban road + pedestrians crossing the road; Action value : The decision-making action of the vehicle in the current state, and the specific actions are: decelerate, change lanes, stop; Reward value : Evaluate the effect of the action, and the specific effects are: safety to avoid collisions, efficiency to save time; S2-2: Construction, training, and optimization of the stochastic network model; For each iteration i from 1 to T, sample a batch of sample sets from the offline environment dataset D , according to the formula: Obtain the Gaussian distribution at the next moment , where represents the mean at the fixed state and action , represents the fixed state and action The standard deviation below is used to minimize the loss function to optimize the stochastic network, with the formula: Among them, the loss function is divided into two parts. The first part represents the mean value predicted by the model, represents the true value; the second part is the regularization parameter term, where N represents the number of samples, M represents the number of output dimensions, and each sample j has M features. is the predicted mean value of the k-th feature of the j-th sample by the model, and the parameter is , is the true value, that is, the target value; and are hyperparameters related to variance, representing the upper and lower limits of variance respectively; exp represents the exponential function with the natural number e as the base. Finally, the stochastic network is optimized by minimizing the loss function to end the iteration.

[0007] In the preferred manner, the training of the multi-action output model in step S3 includes: For each iteration i from 1 to N, a batch of sample sets is sampled from the offline environment dataset D. represents the return value, that is, the sum of the reward values from the current moment until the end of the environment, through the formula: Update the Critic network, where represents the updated Q function in the k + 1-th iteration. The Q function represents the expected return of taking action a in state s. represents the Bellman operator. The target Q value is calculated based on the true return. arg represents the parameter that makes or minimizes, referring to finding the parameter that makes a certain function reach the minimum value; represents the expectation of state s; D represents the experience replay pool or dataset, storing historical state, action, and reward information; through the formula: Obtain the predicted values of the next state and reward , where represents the transition function, indicating that in the given current state and action , the system transfers to the next state and obtains the reward with a probability distribution; represents the normal distribution. Given the state and action, the distributions of state transition and reward are normal; Through the formula: Estimate the uncertainty of the environment , where represents the expected loss, and the loss is weighted by adjusting the size of, and the dynamic adjusted number of actions is obtained through the formula: Obtain the dynamically adjusted number of actions , and the outer min function ensures that the result does not exceed N max , and the inner max function ensures that the result is greater than or equal to 1, where U max represents the preset maximum uncertainty value, and is obtained through the formula: Update the reward value after punishment. For each iteration i from 1 to k, the model propagates forward in an autoregressive manner to obtain a number of action pairs through the formula: represents a condition, and at time step the selected action is sampled from the SeaDT model with randomly initialized parameters under the condition of the reward value sequence received from time step t to t + i, the state sequence from time step t to t + i, and the action sequence and the number of actions selected from time step t to t + i - 1 , and finally end the K iterations; through the formula: Minimize the loss function to optimize the policy model, end the N iterations, and return the optimized optimal model π θ ; where L represents the loss function, which represents the optimization objective of the model; e represents the exponential function, and the sum in the first half of the formula represents soft optimization through Softmax to ensure that the weights occupied by the Q values sum to 1, and the second half of the formula is the expected loss, where represents the predicted action at time step i; represents the true action at time step i.

[0008] In the preferred mode, step S4 includes: S4-1: Given the initial state of the environment , initialize the number of actions , and loop to execute for each iteration i from 1 to n. According to n candidate rewards, a number of action pairs are sampled from each candidate reward, and the formula is: End the n iterations, and according to the formula: Obtain the optimal action ; S4-2: Uncertainty of the computing environment , update the number of actions , the environment executes the optimal action , advance the environment to obtain the state at the next moment and reward Update the return value, the formula is: Finally, the output result is true to end the loop, and the trained model is used to process the newly input offline environment dataset D to dynamically generate the optimal driving strategy.

[0009] The beneficial effects of the present invention are as follows: By using a model-based network to predict the state of the environment at the next moment and comparing it with the true state of the next sample, the uncertainty of the model at the current moment can be obtained. In autonomous driving, this can help the vehicle evaluate the complexity of the current traffic environment. The Critic network and the multi-action output decision transformer improve the output dimension of the Decision Transformer, trying to change from a single action to generating several action pairs, so as to make the action space of the model as wide as possible. At the same time, the Critic network is added for simultaneous training to let the model select the optimal action from these action pairs, skillfully combining the long-sequence prediction ability of the Transformer model and the short-term prediction ability of the Critic network. The dynamic action output of the model, combined with the uncertainty estimation network, can estimate the uncertainty value at each moment, and at the same time, the threshold of uncertainty and the maximum number of action outputs can be set, so as to linearly and dynamically control the number of model outputs, which is in line with intuition. In autonomous driving, when the uncertainty of the environment is high, it is hoped that the vehicle can be conservative and choose a strategy that can effectively score as much as possible. On the contrary, when the uncertainty is very low, the vehicle is encouraged to explore to find the optimal driving strategy. Brief Description of the Drawings

[0010] Figure 1 is the overall architecture diagram of the stochastic dynamic action decision transformer of the present invention; Figure 2 is the model architecture diagram of the stochastic network of the present invention; Figure 3 is the schematic diagram of the autonomous driving scenario of the stochastic state transition of the present invention. Detailed Embodiments

[0011] A stochastic decision transformation method based on dynamic actions includes the steps of: S1: Create input parameters; S2: Pre-train the stochastic network; S3: Train the multi-action output model S4: Based on the above method, illustrate the specific work process; S4-1: Given the initial state of the environment , initialize the number of actions , loop and execute for each iteration i from 1 to n. According to n candidate rewards, sample several action pairs for each candidate reward. End the n iterations to obtain the optimal action ; S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , advance the environment to obtain the state at the next moment and reward , finally, if the output result is true, end the loop. Use the trained model to process the newly input offline environment dataset D and dynamically generate the optimal driving strategy.

[0012] The specific method for creating input parameters in step S1 includes: Offline environment dataset D, Critic network , SeaDT model π with randomly initialized parameters θ , context length K, maximum number of iterations T, N, expected loss parameter , maximum number of decision-making actions N max , n candidate return values ; The offline environment dataset D includes: weather, road conditions, and traffic participant behaviors; Weather includes: sunny, rainy, snowy, foggy, cloudy; Road conditions: highway, urban road, rural road, mountain road, construction area; Traffic participant behaviors: pedestrians crossing the road, cycling, vehicle lane change, vehicle overtaking, emergency braking.

[0013] Step S2 specifically includes: S2-1: Data processing; Convert weather, road condition, and traffic participant behavior data into state, action, and reward value samples , the specific steps include: State value : Generated according to the combination of weather, road conditions, and traffic participant behaviors. The specific combination form is: sunny + urban road + pedestrians crossing the road; Action value : The decision-making action of the vehicle in the current state. The specific actions are: decelerate, change lane, stop; Reward value : Evaluate the effects of actions, and the specific effects are: safety to avoid collisions, and efficiency to save time; S2-2: Construct, train, and optimize the stochastic network model; For each iteration i from 1 to T, sample a batch of sample sets from the offline environment dataset D , according to the formula: Obtain the Gaussian distribution at the next moment , where represents the stationary state and the action under the mean value, represents the stationary state and the action under the standard deviation, minimize the loss function to optimize the stochastic network, and the formula is: Among them, the loss function is divided into two parts. The first part represents the mean value predicted by the model, represents the true value; the second part is the regularization parameter term, where N represents the number of samples, M represents the number of output dimensions, each sample j has M features, is the predicted mean value of the k-th feature of the model for the j-th sample, and the parameter is , is the true value, that is, the target value; and are hyperparameters related to the variance, representing the upper and lower limits of the variance respectively; exp represents the exponential function with the natural number e as the base, and finally optimize the stochastic network by minimizing the loss function to end the iteration.

[0014] The training of the multi-action output model in step S3 includes: For each iteration i from 1 to N, sample a batch of sample sets from the offline environment dataset D , represents the return value, that is, the sum of the reward values from the current moment until the end of the environment, and through the formula: Update the Critic network, where represents the updated Q function in the (k + 1)-th iteration, and the Q function represents the expected return of taking the action a in the state s, represents the Bellman operator, is the target Q value calculated based on the true return, and arg represents the parameter that makes or makes the minimum, referring to finding the parameter that makes a certain function reach the minimum value; denotes the expectation for state s; D represents the experience replay pool or dataset, storing historical state, action, and reward information; through the formula: obtain the predicted values of the next state and reward , where represents the transition function, indicating that given the current state and action , the system transitions to the next state and the obtained reward probability distribution; represents the normal distribution. Given the state and action, the distributions of state transition and reward are normal; through the formula: estimate the uncertainty of the environment , where represents the expected loss. By adjusting the size to weight the loss, through the formula: obtain the dynamically adjusted number of actions , the outer min function ensures that the result does not exceed N max , and the inner max function ensures that the result is greater than or equal to 1, U max represents the pre-set maximum uncertainty value, through the formula: update the penalized reward value. For each iteration i from 1 to k, the model propagates forward in an autoregressive manner to obtain a number of action pairs, through the formula: represents a condition. At time step the selected action is sampled from the SeaDT model with randomly initialized parameters conditioned on the sequence of reward values received from time step t to t+i, the sequence of states from time step t to t+i, and the sequence of actions and the number of actions selected from time step t to t+i - 1 , and finally end the K iterations; through the formula: minimize the loss function to optimize the policy model, end the N iterations, and return the optimized optimal model π θ; where, L represents the loss function, which represents the optimization objective of the model; e represents the exponential function, and the summation in the first half of the formula represents soft optimization through Softmax to ensure that the sum of the weights occupied by the Q values is 1. The second half of the formula is the expected loss, where represents the predicted action at time step i; represents the true action at time step i.

[0015] Step S4 includes: S4-1: Given the initial state of the environment , initialize the number of actions , and loop to execute for each iteration i from 1 to n. According to n candidate rewards, a number of action pairs are sampled for each candidate reward. The formula is: End the n iterations. According to the formula: Obtain the optimal action ; S4-2: Calculate the uncertainty of the environment , update the number of actions , the environment executes the optimal action , advance the environment to obtain the state at the next moment and the reward Update the return value. The formula is: Finally, output the result as true to end the loop, and use the trained model to process the newly input offline environment dataset D to dynamically generate the optimal driving strategy.

[0016] As Figure 1 shown, the stochastic network gives the standard Gaussian distribution of the next state and reward according to the current state and action. The uncertainty is calculated by comparing the estimated value and the actual value, and this uncertainty is passed to the DecisionTransformer. At the same time, the estimated reward value is used as a penalty term for the return value for fine-tuning. Finally, the decision tree model determines the number of actions finally generated using the uncertainty, and the Critic network evaluates the final action to make the optimal choice.

[0017] In autonomous driving applications, the vehicle can make dynamic adjustments in complex traffic environments through this architecture to adapt to sudden road conditions and traffic signal changes.

[0018] As Figure 2 shown, a Dropout layer and a normalization layer are added on the basis of the integrated network to obtain a more accurate uncertainty estimate value. By Performing K forward propagations yields the final Gaussian distribution. This article simulates a small, randomized autonomous driving environment to demonstrate the limitations of DT with respect to splicing and randomness. It explains why these issues exist and how the model addresses them. This example also demonstrates the underlying causes of these issues. Using the environment in the figure as an example, the specific details of the environment are labeled in the diagram. The limitations of DT will be explained from the perspectives of splicing and randomness, respectively.

[0019] like Figure 3 The middle part is the training sample, and the right side is the result after model training. In the figure, different serial numbers represent different states, and the actions available in each state are distinguished by the shape of the arrow extending from the state. Circular arrows indicate positive rewards, directional arrows indicate negative rewards, and entering state 8 will bring 0 rewards. The solidity of the line represents whether the environment is random. A dotted line indicates that executing the action will have a random result and may transfer to one of multiple states. The density of the dotted line represents the probability of transfer, and the higher the density, the greater the probability. To illustrate the extreme case, a state number 11 is introduced, located in the lower right corner of the figure. This state will bring 10 points of reward, but the probability of occurrence is extremely low. It is also intentionally included in the training set, hoping that the model will follow a circular trajectory as much as possible to obtain the optimal solution.

[0020] Splicing problem: Take the first two training trajectories as an example to explain the splicing problem of DT. First, ignore the randomness in the environment. Figure 3 As we can see, the optimal state trajectory starting from state 0 should be "0->1->4->7", but this trajectory is not included in the training data. Instead, the training trajectory contains a subset of the optimal trajectory, namely "0->1->4" and "4->7". Unfortunately, due to the design of DT's loss function, even if DT can be generated according to the optimal trajectory in the first half, it will still tend to the behavioral strategy when entering state 4, resulting in a very low final return. Fortunately, the critic network solves this problem very well. At each moment, the critic network calculates the future benefits to correct DT's errors, and the model framework can output multiple actions. The critic network judges each action and finally selects the optimal action.

[0021] Randomness problem: We also designed a random experiment, that is, to show randomness at the beginning of the environment. Sometimes, when performing a circular action, it does not necessarily enter state "1", but instead enters state "5". In this state, we also picked out two more representative data to illustrate the randomness problem, such as Figure 2As shown in the training trajectory, it can be seen that differences start to emerge in the training set after state 5. Both trajectories can represent the optimal trajectory. However, due to the randomness of the environment, DT does not know the probability corresponding to entering state "11". Even though there are failed samples that enter state "9" in the training samples, the reward for entering 11 is still very high, which causes DT to blindly pursue high rewards, resulting in the opposite effect. In the inference stage, it enters state "9" with a high probability, but the reward becomes lower. However, due to the existence of the critic network, when there are too many failed samples, the critic will naturally reduce the expectation of this action. Correspondingly, in the seaDT framework, the probability of naturally selecting this action will decrease. This is a dynamic adjustment strategy based on the critic.

[0022] Essential reason: The main reason why SeaDT can solve the above problems is that it can effectively integrate the advantages of DT and critic, learning from each other's strengths and compensating for each other's weaknesses. DT is good at processing long sequence data, enabling it to effectively integrate information and condense a better set of action spaces. The critic network can calculate the expected reward of this action based on the actions integrated by DT above, so as to obtain the optimal solution. At the same time, the dynamic action output of SeaDT is also a crucial link to success. As shown in state "3", it has 4 optional actions. However, if the training samples are insufficient, for DT, it may only be limited to the samples in the training set and lack the possibility of exploration. When the uncertainty is high, it is hoped that the model can be conservative and choose strategies that can effectively score as much as possible. On the contrary, when the uncertainty is very low, the model is encouraged to explore to find the optimal action. In this way, SeaDT can effectively handle randomness and splicing problems in complex autonomous driving environments, improving the decision-making ability and safety of the intelligent agent. In autonomous driving applications, vehicles can make dynamic adjustments in complex traffic environments through this architecture to adapt to sudden road conditions and traffic signal changes.

[0023] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A random decision transformation method based on dynamic actions, characterized in that, Including the steps: S1: Create input parameters; S2: Pre-train the stochastic network; S3: Train the multi-action output model; S4: Illustrate the specific working process based on the above method; S4-1: Given the initial state of the environment , initialize the number of actions , loop and execute for each iteration i from 1 to n. According to n candidate rewards, sample a number of action pairs for each candidate reward. End the n iterations to obtain the optimal action ; S4-2: Uncertainty of the computing environment , update the number of actions , the environment executes the optimal action , advance the environment to obtain the state at the next moment and the reward , finally, if the output result is true, end the loop, and use the trained model to process the newly input offline environment dataset D to dynamically generate the optimal driving strategy.

2. The random decision transformation method based on dynamic actions according to claim 1, characterized in that The specific method for creating input parameters in step S1 includes: Offline environment dataset D, Critic network , SeaDT model π with randomly initialized parameters θ , context length K, maximum number of iterations T, N, expected loss parameter , maximum number of decision actions N max , n candidate return values ; The offline environment dataset D includes: weather, road conditions, and traffic participant behaviors; Weather includes: sunny, rainy, snowy, foggy, cloudy; Road conditions: highway, urban road, rural road, mountain road, construction area; Traffic participant behaviors: pedestrians crossing the road, cycling, vehicle lane change, vehicle overtaking, emergency braking.

3. The random decision transformation method based on dynamic actions according to claim 1, wherein The specific method for pre-training the stochastic network in step S2 includes: S2-1: Data processing; Convert weather, road condition, and traffic participant behavior data into state, action, and reward value samples , and the specific steps are as follows: Status value : Generated based on a combination of weather, road conditions, and the behavior of traffic participants. The specific combination form is: sunny + urban road + pedestrians crossing the road; Action value : The decision-making actions of the vehicle in the current state, and the specific actions are: decelerating, changing lanes, and stopping; Reward value : Evaluate the effect of the action, and the specific effects are: safety to avoid collisions and efficiency to save time; S2-2: Construction, training, and optimization of the stochastic network model; For each iteration \(i\) from \(1\) to \(T\), sample a batch of sample sets from the offline environment dataset \(D\). , according to the formula: Obtain the Gaussian distribution at the next moment , where represents the stationary state and the mean under the action , represents the stationary state and the standard deviation under the action . Minimize the loss function to optimize the stochastic network. The formula is: Among them, the loss function is divided into two parts. The first part represents the mean value predicted by the model, and represents the true value. The second part is the regularization parameter term, where N represents the number of samples, M represents the number of output dimensions, and each sample j has M features. is the predicted mean value of the k-th feature of the j-th sample by the model, and the parameter is , is the true value, that is, the target value; and are hyperparameters related to variance, representing the upper and lower limits of variance respectively; exp represents the exponential function with the natural number e as the base. Finally, the stochastic network is optimized by minimizing the loss function to end the iteration.

4. The random decision transformation method based on dynamic actions according to claim 1, characterized in that, The training of the multi-action output model in step S3 includes: For each iteration \(i\) from \(1\) to \(N\), sample a batch of sample sets from the offline environment dataset \(D\). , Denote the return value as the sum of the reward values from the current moment until the end of the environment, through the formula: Update the Critic network, where represents the Q-function updated in the (k + 1)-th iteration. The Q-function represents the expected return of taking action a in state s, represents the Bellman operator, is the target Q-value calculated based on the true return. arg represents the parameter that makes or minimizes. It refers to finding the parameter that minimizes a certain function; represents the expectation of state s; D represents the experience replay pool or dataset that stores historical state, action, and reward information; through the formula: Obtain the predicted values of the next state and the reward , where represents the transition function, indicating that given the current state and action , the system transitions to the next state and the obtained reward with a probability distribution; represents the normal distribution, and given the state and action, the distributions of state transition and reward are normal; Through the formula: Estimate the uncertainty of the environment , where represents the expected loss, and the loss is weighted by adjusting the magnitude of, through the formula: Obtain the dynamically adjusted number of actions , the outer min function ensures that the result does not exceed N max , the inner max function ensures that the result is greater than or equal to 1, U max represents the pre-set maximum uncertainty value, through the formula: Update the penalized reward value. For each iteration i from 1 to k, the model propagates forward in an autoregressive manner to obtain several action pairs. Through the formula: Represents a condition at time step The selected action is sampled from the SeaDT model with randomly initialized parameters conditioned on the sequence of reward values received from time step t to t+i, the sequence of states from time step t to t+i, and the sequence of actions and the number of actions selected from time step t to t+i-1 , and finally ends after K iterations; through the formula: Minimize the loss function to optimize the policy model, end N iterations, and return the optimized optimal model π θ ; where L represents the loss function, which represents the optimization objective of the model; e represents the exponential function, and the summation in the first half of the formula indicates soft optimization through Softmax to ensure that the sum of the weights occupied by the Q value is 1. The second half of the formula is the expected loss, where represents the predicted action at time step i; represents the true action at time step i.

5. The random decision transformation method based on dynamic actions according to claim 1, wherein Step S4 includes: S4-1: Given the initial state of the environment , initialize the number of actions , and loop to execute for each iteration i from 1 to n. According to n candidate rewards, a number of action pairs are sampled for each candidate reward. The formula is: End the n iterations. According to the formula: Obtain the optimal action ; S4-2: Uncertainty of the computing environment , update the number of actions , the environment executes the optimal action , advance the environment to obtain the state at the next moment and the reward Update the return value, and the formula is: Finally, when the output result is true, end the loop. Use the trained model to process the newly input offline environment dataset D and dynamically generate the optimal driving strategy.

Images