A trajectory planning method based on pessimistic critic decision transformer

Abstract

The present invention discloses a trajectory planning method based on a pessimistic critic decision converter, which is specifically as follows: 1: collecting historical trajectory data of an industrial robot; 2: constructing a decision converter network, a maximum value decision converter strategy network, and a pessimistic value network 1 and a pessimistic value network 2; and initializing these network parameters; 3: establishing a maximum value decision converter strategy network and a target network corresponding to two independent pessimistic value networks; and initializing these target networks; 4: updating decision converter network parameters; 5: updating pessimistic value network parameters; 6: updating maximum value decision converter strategy network parameters; 7: updating target network parameters; 8: repeating steps 4 to 7 until the maximum number of iterations is reached; 9: guiding the maximum value decision converter strategy network to generate the optimal trajectory at the next moment through the value of the pessimistic value network output action finally obtained. The present invention improves the efficiency and safety of task completion.

Description

Technical Field

[0001] The invention belongs to the field of robot technology, and in particular relates to a trajectory planning method based on a pessimistic critic decision converter. Background Art

[0002] Training a robot to walk in a real environment may take weeks, and human intervention is required to recover after each fall. Reinforcement learning can effectively solve sequential decision-making problems through trial and error, and has achieved remarkable success. However, reinforcement learning relies on online interaction with the environment, which limits its application in the real world. In some scenarios, online interaction is inefficient and may pose safety risks. In contrast, offline reinforcement learning does not require real-time interaction with the environment, but only uses a static experience data set to learn strategies, which provides more possibilities for the application of reinforcement learning in real-world scenarios.

[0003] The decision transformer proposes a new supervised learning paradigm to solve offline reinforcement learning problems through conditional sequence modeling technology. The decision transformer uses autoregression and attention mechanisms to maximize the probability of conditional actions based on historical information, thereby realizing the transition from offline reinforcement learning to supervised sequence modeling. By considering diverse historical information, the conditional sequence modeling method can predict actions and relax the strict Markov assumption in traditional dynamic programming. Therefore, it has the potential to solve suboptimal strategies, long-view problems, and sparse reward problems, and also provides a method for application in industrial scenarios.

[0004] However, existing conditional sequence modeling methods only associate reward-driven token values ​​with a single trajectory, focusing on actions that obtain higher rewards, ignoring the potential contribution of other trajectories to the optimal trajectory, and thus lack the ability to splice trajectories. This limitation may result in the inability to effectively splice the optimal trajectory in actual industrial applications. Therefore, enhancing the trajectory splicing ability of conditional sequence modeling methods has become a key challenge that needs to be addressed. Summary of the invention

[0005] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides a trajectory planning method based on a pessimistic critic decision converter.

[0006] Technical solution: The present invention discloses a trajectory planning method based on a pessimistic critic decision converter, which specifically includes the following steps:

[0007] Step 1: Collect historical trajectory data of industrial robots to form a trajectory dataset τ t , the trajectory dataset τ t The expression is: Where t represents the time step, K represents the sequence length, and s irepresents the i-th state, a i represents the i-th action, represents the i-th driving return;

[0008] Step 2: Build the decision converter network β ω 、Maximum value decision converter strategy network π θ , Pessimistic Value Network and pessimistic value network And initialize these network parameters, ω is the parameter of the decision converter network, θ is the parameter of the maximum value decision converter strategy network, φ1 is the parameter of the pessimistic value network 1, and φ2 is the parameter of the pessimistic value network 2;

[0009] Step 3: Establish the maximum value decision converter strategy network and the target network corresponding to the two pessimistic value networks and And initialize these target networks; is the parameter of the target network corresponding to the maximum value decision converter strategy network, is the parameter of the target network corresponding to the pessimistic value network 1, are the parameters of the target network corresponding to the pessimistic value network 2;

[0010] Step 4: Update the decision converter network parameters;

[0011] Step 5: Update the pessimistic value network parameters;

[0012] Step 6: Update the maximum value decision converter strategy network parameters;

[0013] Step 7: Update target network parameters;

[0014] Step 8: Repeat steps 4 to 7 until the maximum number of iterations is reached;

[0015] Step 9: The value of the action is output by the pessimistic value network, which guides the maximum value decision converter strategy network to generate the optimal trajectory of the industrial robot at the current moment according to the robot's current state and previous trajectory information.

[0016] Furthermore, the decision converter network and the maximum value decision converter strategy network are both generated pre-trained converter structures, and the two pessimistic value networks are both multi-layer perceptron structures with two hidden layers and 256 neurons in the hidden layers.

[0017] Furthermore, the target network parameters are initialized by directly assigning the original parameters of the network corresponding to the target network to the target network.

[0018] Furthermore, in step 4, according to the following loss function Update the decision transformer network parameters:

[0019]

[0020] Among them, ~ represents the sampling operation, represents the action of the decision converter output, β ω (τ t ) represents the set of actions output by the decision transformer, Indicates the expectation symbol.

[0021] Furthermore, in step 5, the pessimistic value network parameters are updated by the following loss function;

[0022]

[0023] in, represents the loss function of the pessimistic value network 1, represents the loss function of the pessimistic value network 2, represents the action of the decision converter output, βx(τ t ) represents the set of actions output by the decision transformer, Indicates from The action sampled in represents the action set of the target network output corresponding to the maximum value decision transformer policy network, π θ (τ t ) represents the set of actions output by the maximum value decision transformer policy network, Indicates that from π θ (τ t ) in the sampled action, r m Indicates (s m ,a m ) corresponding to the reward value, Indicates that the input is (s m ,a m ) when the pessimistic value network outputs Q value, u = 1 or 2; Indicates that the input is The Q value of the target network output corresponding to the pessimistic value network, Indicates that the input is The Q value output by the pessimistic value network is Indicates that the input is The Q value output by the pessimistic value network; γ is a hyperparameter, r j is the reward value under the corresponding action state, ~ represents sampling, represents the importance sampling ratio, represents the penalty constant, α is the temperature coefficient, Indicates the expectation symbol.

[0024] Furthermore, in step 6, according to the following loss function Update the maximum value decision converter strategy network parameters:

[0025]

[0026] Among them, ~ represents sampling, π θ (τ t ) represents the set of actions output by the maximum value decision transformer policy network, Indicates that from π θ (τ t ) in the sampled action, Indicates that the input is When , the Q value output by a random pessimistic value network in the two pessimistic value networks is, Indicates the expectation symbol.

[0027] Furthermore, in step seven, the target network parameters are updated, specifically:

[0028]

[0029] Where k represents the kth iteration, Indicates the target network update rate.

[0030] Beneficial effects: The present invention can solve the problem of trajectory splicing caused by sparse offline data in robot trajectory planning. It is suitable for industrial scenarios that require high-precision and efficient task execution. Especially when offline data is insufficient, it can still generate stable optimal trajectory planning, thereby improving the efficiency and safety of task completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Architecture diagram of the offline reinforcement learning method based on the pessimistic critic decision transformer. DETAILED DESCRIPTION

[0032] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0033] The trajectory planning method based on the pessimistic critic decision converter of the present invention has the following process: Figure 1As shown. The method proposed in this embodiment is mainly applied to the field of industrial robot trajectory planning. Aiming at the limitations of the existing methods in the insufficient trajectory splicing ability and the selection of a single high-reward action in complex task scenarios, an improvement scheme is proposed. Specifically, the present invention first collects the motion data of the industrial robot performing tasks in actual operation, including state information such as joint angles, speeds, and positions, as well as feedback on task results. Through the pre-trained decision converter network, the model can express the robot's historical behavior sequence, and calculate the importance sampling ratio of each action based on this. This ratio is used to regularize the critic network, so that it can more accurately evaluate the action value of the robot in different task scenarios. By regularizing the critic network, the excessive reliance on a single high-reward action in the existing offline reinforcement learning method is overcome, and the diversity and flexibility of action decisions are improved. Subsequently, the maximum value decision converter strategy network is updated to generate the optimal trajectory sequence for the trajectory planning and execution of industrial robots in different tasks. The present invention can solve the problem of trajectory splicing caused by sparse offline data in robot trajectory planning, and is suitable for industrial scenarios that require high-precision and efficient execution of tasks. In particular, in the case of insufficient offline data, it can still generate a stable optimal trajectory planning, which improves the efficiency and safety of task completion.

[0034] The method of the present invention comprises the following specific steps:

[0035] Step 1: Decision converter network, maximum value decision converter policy network and pessimistic value network, and initialize network parameters.

[0036] Step 2: Establish the target network and initialize the network parameters.

[0037] Step 3: Update the decision converter network parameters.

[0038] Step 4: Update the pessimistic value network parameters.

[0039] Step 5: Update the maximum value decision converter strategy network parameters.

[0040] Step 6: Update the target network parameters.

[0041] Step 7: Repeat steps 3 to 6 until the optimal strategy is obtained.

[0042] In step 1, a decision converter network β is established. ω and the maximum value decision converter strategy network π θ , establish two independent value networks and and They represent pessimistic value network 1 and pessimistic value network 2 respectively; initialize network parameters; network parameters include: decision converter network parameter ω, maximum value decision converter strategy network parameter θ, pessimistic value network 1 parameter φ1 and pessimistic value network 2 parameter φ2; decision converter network and maximum value decision converter strategy network are used to generate pre-trained converter structures; pessimistic value networks are both multilayer perceptron structures with two hidden layers and 256 neurons in the hidden layer; network parameters are initialized by random initialization.

[0043] In step 2, the maximum value decision converter strategy network and the target network corresponding to the two independent pessimistic value networks are respectively established. and and They represent the target network parameters corresponding to the maximum value decision converter strategy network, pessimistic value network 1, and pessimistic value network 2 respectively; the target network parameters are initialized by directly assigning the corresponding original network parameters (θ, φ1, φ2) to the target network parameters

[0044] In step 3, using the experience pool The trajectory samples in τ t , Where t represents the time step, K represents the sequence length, and s i represents the i-th state, a i represents the i-th action, represents the i-th driving return; r i Indicates that with s i , a i The corresponding reward value; the decision converter network parameter ω is updated by minimizing the following loss function through the gradient descent method:

[0045]

[0046] Among them, ~ represents the sampling operation, represents the action of the decision converter output, β ω (τ t ) represents the set of actions output by the decision transformer, Indicates the expectation symbol.

[0047] In step 4, using the experience pool The empirical trajectory τ in t The pessimistic value network parameters φ1 and φ2 are updated by minimizing the following loss function through gradient descent:

[0048]

[0049] in, represents the loss function of the pessimistic value network 1, represents the loss function of the pessimistic value network 2, Indicates from The action sampled in represents the action set of the target network output corresponding to the maximum value decision transformer policy network, π θ (τ t ) represents the set of actions output by the maximum value decision transformer policy network, Indicates that from π θ (τ t ) in the sampled action, r m Indicates (s m ,a m ) corresponding to the reward value, Indicates that the input is (s m ,a m ) when the pessimistic value network outputs Q value, u = 1 or 2; Indicates that the input is The Q value of the target network output corresponding to the pessimistic value network, Indicates that the input is The Q value output by the pessimistic value network is Indicates that the input is The Q value output by the pessimistic value network; γ is a hyperparameter, r j is the reward value under the corresponding action state, represents the importance sampling ratio, represents the penalty constant and α is the temperature coefficient.

[0050] In step 5, using the experience pool The trajectory samples in update the maximum value decision converter policy network parameters θ by minimizing the following loss function through gradient descent:

[0051]

[0052] Among them, π θ (τ t ) represents the set of actions output by the maximum value decision transformer policy network, Indicates that from π θ (τ t ) in the sampled action, Indicates that the input is When the Q value of a random pessimistic value network outputs, in step 6, the following update method is used to update the parameters of the target network:

[0053] First, calculate and Then, assign the results of the above calculations to: and Where k represents the kth iteration, Indicates the target network update rate. In this embodiment

[0054] Repeat steps 3 to 6 to update the parameters of each network and the target network 1×10 5 times, and outputs the value of the action through the updated pessimistic value network, thereby guiding the maximum value decision converter policy network to generate the action with the highest value.

[0055] The present invention can use the value-guided decision converter model to generate optimal actions. Specifically, first, considering that the decision converter method of existing conditional sequence modeling lacks trajectory splicing capabilities and tends to select a single high-driving reward action, a critic network is introduced to approximate the value; then, the decision converter network is pre-trained to express the behavior sequence, thereby calculating the sequence importance sampling ratio to regularize the critic network; finally, the maximum value decision converter policy network is updated to obtain the action sequence with the highest value.

[0056] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A trajectory planning method based on a pessimistic critic decision transformer, characterized in that: The specific steps include: Step 1: Collect historical trajectory data of industrial robots to form a trajectory dataset τ t , the trajectory dataset τ t The expression is: Where t represents the time step, K represents the sequence length, and s i represents the i-th state, a i represents the i-th action, represents the i-th driving return; Step 2: Build the decision converter network β ω 、Maximum value decision converter strategy network π θ 、Pessimistic value network 1 and pessimistic value network 2 And initialize these network parameters, ω is the parameter of the decision converter network, θ is the parameter of the maximum value decision converter strategy network, φ1 is the parameter of the pessimistic value network 1, and φ2 is the parameter of the pessimistic value network 2; Step 3: Establish the maximum value decision converter strategy network and the target network corresponding to the two pessimistic value networks and And initialize these target networks; is the parameter of the target network corresponding to the maximum value decision converter strategy network, is the parameter of the target network corresponding to the pessimistic value network 1, are the parameters of the target network corresponding to the pessimistic value network 2; Step 4: Update the decision converter network parameters; Step 5: Update the pessimistic value network parameters; Step 6: Update the maximum value decision converter strategy network parameters; Step 7: Update target network parameters; Step 8: Repeat steps 4 to 7 until the maximum number of iterations is reached; Step 9: The value of the action is output by the pessimistic value network, which guides the maximum value decision converter strategy network to generate the optimal trajectory of the industrial robot at the current moment according to the robot's current state and previous trajectory information.

2. The trajectory planning method based on the pessimistic critic decision transformer according to claim 1 is characterized in that: The decision converter network and the maximum value decision converter strategy network are both generated pre-trained converter structures, and the two pessimistic value networks are multi-layer perceptron structures with two hidden layers and 256 neurons in the hidden layer.

3. The trajectory planning method based on the pessimistic critic decision transformer according to claim 1 is characterized in that: The method for initializing the target network parameters is: directly assigning the original parameters of the network corresponding to the target network to the target network.

4. The trajectory planning method based on the pessimistic critic decision transformer according to claim 1 is characterized in that: In step 4, the following loss function is used Update the decision transformer network parameters: Among them, ~ represents the sampling operation, represents the action of the decision converter output, β ω (τ t ) represents the set of actions output by the decision transformer, Indicates the expectation symbol.

5. The trajectory planning method based on the pessimistic critic decision transformer according to claim 1 is characterized in that: In the step 5, the pessimistic value network parameters are updated by the following loss function; in, represents the loss function of the pessimistic value network 1, represents the loss function of the pessimistic value network 2, represents the action of the decision converter output, β ω (τ t ) represents the set of actions output by the decision transformer, Indicates from The action sampled in represents the action set of the target network output corresponding to the maximum value decision transformer policy network, π θ (τ t ) represents the set of actions output by the maximum value decision transformer policy network, Indicates that from π θ (τ t ) in the sampled action, r m Indicates (s m ,a m ) corresponding to the reward value, Indicates that the input is (s m ,a m ) when the pessimistic value network outputs Q value, u = 1 or 2; Indicates that the input is The Q value of the target network output corresponding to the pessimistic value network, Indicates that the input is The Q value output by the pessimistic value network is Indicates that the input is The Q value output by the pessimistic value network; γ is a hyperparameter, r j is the reward value under the corresponding action state, ~ represents sampling, represents the importance sampling ratio, represents the penalty constant, α is the temperature coefficient, Indicates the expectation symbol.

6. The trajectory planning method based on pessimistic critic decision transformer according to claim 1, characterized in that: In step 6, the following loss function is used Update the maximum value decision converter strategy network parameters: Among them, ~ represents sampling, π θ (τ t ) represents the set of actions output by the maximum value decision transformer policy network, Indicates that from π θ (τ t ) in the sampled action, Indicates that the input is When , the Q value output by a random pessimistic value network in the two pessimistic value networks is, Indicates the expectation symbol.

7. The offline reinforcement learning method based on pessimistic critic decision transformer according to claim 1, characterized in that: In step 7, the target network parameters are updated, specifically: Where k represents the kth iteration, Indicates the target network update rate.

Images