A robot double-layer control method based on a conditional diffusion model

Abstract

The application discloses a robot double-layer control method for selecting skills based on a conditional diffusion model. The long-time-span reinforcement learning track is constructed by a double-layer strategy, and is abstracted into a short-time-span task for skill execution of an upper-layer strategy selection. In the strategy pre-training stage, an additional condition is introduced to balance and bind the robot pose state and the selected skill, thereby improving the exploration rate of the robot in the reinforcement learning environment, enabling the robot agent to combine different skills to complete the task in a shorter time. In the strategy fine-tuning stage, the reinforcement learning PPO algorithm is introduced to maximize the cumulative reward as the target, and the skill selection strategy of the upper-layer skill selection strategy is continuously fine-tuned until the model converges. The application can reduce the skill exploration space to speed up the exploration by binding the correlation between different pose states and their skills in the original small skill space under the action space of a long-time-span robot task.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of reinforcement learning classification, and particularly relates to a robot double-layer control method for skill selection based on a conditional diffusion model. BACKGROUND

[0002] Reinforcement learning is driving the development of different technologies in the current machine learning field, especially in the field of robot motion planning and control. End-to-end reinforcement learning has shown remarkable results, whether from a bionic robot or a bionic arm. However, the training process of reinforcement learning requires the robot to continuously explore in a simulation environment and use existing trajectory data to find the optimal strategy. However, this also introduces a large number of problems, such as high interaction costs when the robot interacts in a real-world scenario, and the final performance of the robot is directly affected by the exploration strategy.

[0003] In actual applications, when the robot is required to complete a task that takes a long time to obtain a reward, the problem of sparse rewards will be encountered, which will result in the control strategy of the robot being unable to obtain optimal feedback in a timely manner, and the excessively sparse rewards will make most of the actions in the trajectory sequence unable to obtain reasonable feedback information, thereby greatly hindering the improvement of the strategy performance of the robot agent.

[0004] To address these challenges, the application provides a robot double-layer control method for skill selection based on a conditional diffusion model, which compresses the original long-time span trajectory sequence into a shorter time sequence trajectory by constructing a double-layer control strategy, thereby solving the problem of unobvious model feedback under the problem of sparse rewards. At the same time, the diffusion model is introduced to bind the current state with the skills that can be selected, thereby discarding the open set skills and improving the exploration efficiency. SUMMARY

[0005] The application aims to improve the exploration efficiency of the robot interaction and solve the problem of slow strategy updating under the problem of sparse rewards, and provides a robot double-layer control method for skill selection based on a conditional diffusion model.

[0006] Technical scheme: The robot double-layer control method for skill selection based on a conditional diffusion model provided by the application comprises a strategy pre-training stage and a strategy fine-tuning stage.

[0007] The strategy pre-training stage is as follows:

[0008] Given a set of training data sets with N robot pose trajectories S and action sequences A, initialize the robot bottom-layer control strategy and upper-layer control strategy model components;

[0009] Randomly select B samples from the data set, and randomly slice the length T pose trajectory B in the pose trajectory T ;

[0010] Skill z is obtained by using LSTM network as an encoder to encode the pose trajectory slice B T in S and A, and using LSTM-based decoder as the bottom-level control strategy π lo of the double-layer control, according to z and the pose state S in B l , the corresponding action sequence A is restored, and the action imitation loss function is calculated

[0011] Initialize the robot upper skill selection strategy π hi based on the diffusion model, bind the pose state S in B T and the skill z through the diffusion model, and normalize the action sequence A as the additional condition c of the conditional diffusion model, to realize the upper skill selection strategy π hi Different skill vectors z are sampled through different additional conditions c, and the training process of the diffusion model is realized by the loss function ;

[0012] Determine the total loss function for training the bottom-level control strategy and the upper skill selection strategy;

[0013] The policy fine-tuning stage is:

[0014] Initialize the conditional selection model component, and the robot uses the conditional selection model π c to generate the additional condition c, and the upper skill selection strategy π hi samples the skill z corresponding to the additional condition c and hands it over to the bottom-level control strategy π lo to restore the action, and uses the PPO algorithm loss and state value evaluation network loss to find the optimal skill for different pose states to run the robot, and constantly fine-tune the skill selection strategy of the upper skill selection strategy, iteratively optimize the model, until the robot bottom-level control strategy and upper skill selection strategy model converge.

[0015] Further, the training data set D = {τ n = (s0, a0, s1, a1, …, s T , a T ) n |n = 1, …, N}, wherein τ represents a trajectory sequence containing pose state S and robot action A, T is the length of the trajectory sequence, s is the pose state of the robot, and a is the action taken by the robot at the current pose state.

[0016] Further, the initialization of the robot bottom-level control strategy and upper skill selection strategy model component includes:

[0017] LSTM-based trajectory sequence encoder E(·), LSTM-based robot low-level control policy lo (·), upper-level skill selection policy based on diffusion model hi (·); wherein E(·) is responsible for extracting trajectory features, and lo (·) is responsible for restoring the robot action a under the current pose state s given the skill z, and hi (·) is responsible for restoring the available skill z corresponding to the current condition and pose state under the premise of given diffusion model condition.

[0018] Further, the process of restoring the corresponding action sequence A is as follows:

[0019] Select B sets of length T robot pose sequences s from the trajectory data pool 0:T and action sequences a 0:T to form trajectory sequences [s 0:T , a 0:T ]; first encode the trajectory sequence by the LSTM-based encoder E to obtain its skill z; and reconstruct the action sequence by the LSTM-based decoder for the existing pose sequence and skill z, so that lo has the ability to restore the robot action sequence A under the condition of known state s and skill z.

[0020] Further, the action imitation loss function is:

[0021]

[0022] wherein s 0:T , a 0:T is the pose state and action sequence that the encoder needs to encode into the hidden space, z is the skill space vector after s 0:T , a 0:T is encoded, and β is a variable that constrains the size of the regularization term, KL is used to calculate the KL divergence size between two different distributions, and p(z) represents N(0, I) as a standard Gaussian distribution.

[0023] Further, the loss is implemented as follows:

[0024] Introduce a diffusion model to bind the pose state s and the skill z obtained after encoding, limit the skill that the robot should have when exploring the current pose state during exploration; according to the existing trajectory sequence [s 0:T , a 0:T ] and the skill encoding z corresponding to the trajectory data, introduce a conditional diffusion model, and restore the original action sequence a 0:THard-coded as an additional condition c, defined as follows:

[0025]

[0026] The additional condition c and the initial state s0 are used to generate the corresponding skills, and the loss

[0027]

[0028] Further, the total loss function for training the underlying control policy and the upper skill selection policy is:

[0029]

[0030] wherein, is the action imitation loss function, is the loss function of the diffusion model training process.

[0031] Further, the policy fine-tuning stage is implemented as follows:

[0032] In the policy fine-tuning stage, the model components are initialized as the conditional selection policy π c , the model components π hi and π lo are frozen, and the state value evaluation network V is initialized; the robot generates the optimal skill z under the current state s by selecting the optimal additional condition c, and generates a trajectory data [(s t , c t , r t )|t=0,1,…,T'] with reward return by interacting with the environment; where T' represents the length of time the robot interacts with the environment, s t represents the pose state, c t represents the additional condition selected by the robot to select the skill, and r t represents the reward obtained by taking the skill under the current state; after collecting the trajectory data D, the PPO algorithm is used to update the policy π c , and the state evaluation network V is learned, the PPO loss and the state evaluation loss are as follows:

[0033]

[0034] wherein A t is the advantage value function of the additional condition c t selected under the current state s t , and the higher the advantage, the more likely that the additional condition will generate excellent skill actions.

[0035] Beneficial effects: compared with the prior art, the beneficial effects of the present application: the present application realizes the reduction of time span with skill time length as the denominator by abstracting the originally long-time-span reinforcement learning trajectory into a shorter-time-span task of skill selection and skill execution through double-layer strategy construction, thereby significantly solving the problem of sparse reward in policy training, and achieving good results on long-time-span tasks; at the same time, the present application binds the robot pose state and the selected skill by introducing an additional condition, thereby improving the exploration rate of the robot in the reinforcement learning environment, so that the robot agent can combine different skills to complete the task in a shorter time; the present application uses an iterative optimization algorithm, uses mean square error loss in the policy pre-training stage, and introduces reinforcement learning PPO algorithm in the policy fine-tuning stage to maximize the cumulative reward, which is simple and efficient, easy to implement and apply; the present application can apply the initial policy obtained by pre-training to different downstream reinforcement learning tasks, and can have very high performance in different downstream task environments, the pre-trained model improves the generalization ability of reinforcement learning in different scene tasks, and changes the original from-scratch training agent paradigm to model fine-tuning paradigm, improving the application scenarios and reusability of the model; the present application shows significant superiority in robot arm control and bionic operation, and can be widely applied to robots that need to complete long-time-span tasks, and has important practical value. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of the present application;

[0037] Figure 2 is a curve graph of the success rate of the present application in several robot control scenarios changing with model learning rounds;

[0038] Figure 3 is a curve graph of the total reward of the present application in several robot control scenarios changing with model learning rounds. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with the drawings.

[0040] As shown in Figure 1 , the present application proposes a robot double-layer control method based on condition diffusion model skill selection, designs a model pre-training stage and a fine-tuning stage, and significantly improves the reusability and generalization of the model algorithm, and the specific implementation process is as follows:

[0041] Steps 1 to 4 are the policy pre-training stage; step 5 is the policy fine-tuning stage.

[0042] Step 1, given a set of training data set with N robot pose trajectories S and action sequences A, initialize the model components.

[0043] Training data set D = {τ n = (s0, a0, s1, a1, …, s T , a T ) n |n = 1, …, N}, where τ represents a sequence of trajectories containing pose states S and robot actions A, T is the length of the trajectory sequence, s is the pose state of the robot, and a is the action taken by the robot at the current pose state; the initialized model components include: an LSTM-based trajectory sequence encoder E(·), an LSTM-based robot bottom control policy π lo (·), and a diffusion model-based upper skill selection policy π hi (·). Where E(·) is responsible for extracting trajectory features, π lo (·) is responsible for restoring the robot action a under the given skill z at the current pose state s, and π hi (·) is responsible for restoring the available skill z corresponding to the current condition and pose state under the condition of the given diffusion model.

[0044] In this embodiment, the robot arm control is taken as the task scenario, and in the pre-training stage, the robot is allowed to randomly operate the arm to obtain different skill trajectories τ = (s0, a0, s1, a1, …, s T , a T ), and the skill trajectory has a time length of T.

[0045] Step 2, randomly select B samples from the data set, and randomly slice the length-T pose trajectories B T from these pose trajectories.

[0046] In this stage, a batch of data is randomly taken from the set D each time as training data, where the entire time sequence trajectory is taken as an abstract skill, and it is defined as z in the abstract space.

[0047] Step 3, use the LSTM network as an encoder to encode the S and A in the pose trajectory slice B T to obtain the skill vector z, and use the LSTM-based decoder as the bottom control policy π lo in the double-layer control to restore the corresponding action sequence A according to z and the pose state S in B l , and calculate the action imitation loss function

[0048] Select B groups of length-T robot pose sequences s 0:T and action sequences a 0:T from the trajectory data pool to form a trajectory sequence [s 0:T , a 0:T]. First, the trajectory sequence is encoded by the LSTM-based encoder E to obtain its skill z. And the action sequence is reconstructed by the LSTM-based decoder for the existing pose sequence and skill z, so that lo The robot action under the skill is restored under the condition of known state s and skill z.

[0049] The reconstruction loss is used, and the loss function for reconstructing the robot action is considered according to the characteristics of the LSTM model is:

[0050]

[0051] Where s 0:T ,a 0:T is the pose state and action sequence that the encoder needs to encode into the hidden space, z is the skill space vector after encoding s 0:T ,a 0:T , β is a variable that restricts the size of the regularization term, KL is used to calculate the KL divergence size between two different distributions, p(z) represents N(0, I) as a standard Gaussian distribution, the purpose is to make the hidden space features encoded by the encoder as a whole as the same as the standard Gaussian distribution.

[0052] Step 4, initialize the robot upper skill selection strategy π based on the diffusion model hi , the pose state S in B T and the skill z obtained in step 3 are bound by the diffusion model, and the action sequence A is normalized as the condition c of the conditional diffusion model, so that a single pose state can sample different skills through different additional conditions c, and this goal is achieved by the loss function .

[0053] Considering that the skills available to the robot at a certain pose state s are limited, a diffusion model is introduced to bind the pose state s and the encoded skill z, so as to limit the skills that the robot should have when exploring the current pose state during exploration. According to the existing trajectory sequence [s 0:T ,a 0:T ], and the skill z corresponding to the trajectory data, we introduce a conditional diffusion model to hard-code the original action sequence a 0:T into an additional condition c, which is defined as follows:

[0054]

[0055] The additional condition c and the initial state s0 are used as the condition for generating the corresponding skill, and the loss

[0056]

[0057] Step 5, initialize the condition selection model component, the robot adopts the condition selection model c Generate additional conditions c, and select the strategy hi Sample the skill z corresponding to the additional condition c and hand it over to the bottom-level control strategy lo Restore the action, use the PPO algorithm loss And state value evaluation network loss Find the optimal skill for the robot to run in different pose states, and continuously fine-tune the skill selection strategy of the upper-level policy, iteratively optimize the model until the model converges.

[0058] In the policy fine-tuning stage, initialize the model component as the condition selection policy c , freeze the model component hi and lo , the state value evaluation network V. The robot generates the optimal skill z in the current state s by selecting the optimal additional condition c, and generates a trajectory data [(s t , c t , r t )|t=0,1,…,T'] by interacting with the environment, where T' represents the length of time the robot interacts with the environment, s t represents the pose state, c t represents the additional condition selected by the robot to select the skill, and r t represents the reward obtained by taking the skill in the current state. After collecting the trajectory data D, use the PPO algorithm to update the policy c , and learn the state evaluation network V, the PPO loss and the state evaluation loss are as follows:

[0059]

[0060] Where A t is the advantage value function of the current state s t selecting additional conditions c t , the higher the advantage, the more likely the additional condition will produce excellent skill actions.

[0061] So far, the pre-training and fine-tuning process of the robot's bottom-level control policy and upper-level skill selection policy model is complete. If the robot's bottom-level control policy and upper-level control policy model has converged, stop training and output the required model modules for classification: the bottom-level control policy lo (·), the upper-level skill selection policy hi (·) and the condition selection policy c(·) else, go to step 5 to continue fine-tuning training. The output pre-trained model can be directly used as the initial policy in the robot arm operating environment, and the output fine-tuned model can be directly used for the specific robot arm control task. In the embodiment, only the current pose state s of the robot needs to be input into the policy c (·) generate an additional condition c, and input the pose state s and the additional condition c into the skill selector hi (·) generate a skill z, and finally input the skill z and the pose state s into the lower-level controller lo (·) and output the optimal action of the robot.

[0062] The present application is particularly suitable for the field of robot action planning and control. The traditional reinforcement learning task does not consider the problem of sparse rewards in a long time span, which makes it difficult to solve long time span tasks. At the same time, the existing double-layer control algorithm does not consider the problem of exploration efficiency, and often explores from all skills or uneven skills, which reduces the speed of the robot to obtain the optimal skill. The present application introduces an additional condition, hierarchical reinforcement learning and diffusion model, and proposes a new double-layer control method, which solves the problem of sparse rewards and effectively improves the exploration efficiency of the robot.

[0063] To verify the effectiveness of the present application, the effects of the proposed method on three robot arm control tasks: Table-Cleanup, Slippery-Push and Pyramid-Stack are given in the embodiment, and are compared with the most advanced method.

[0064] The evaluation index of the comparative experiment is the success rate in different downstream task test environments (Success Rate), which measures the success rate in different tasks. And the reward return in different downstream task test environments, which measures the reward return in different tasks. The higher the reward, the faster the completion speed.

[0065] All methods use the same diffusion model architecture as the upper-layer skill selection policy hi (·) as the basic architecture, and the diffusion time step is 100 steps. For the condition selector c (·), a three-layer multi-layer perceptron (MLP) is selected as the basic architecture, the input dimension is the pose state dimension, the hidden layer dimension is 128, and the output dimension is the action dimension. For the lower-level policy lo (·), an LSTM is used as the backbone network, the input dimension is the sum of the pose dimension and the skill dimension, and the output dimension is also the action dimension. For the skill encoding network E, an LSTM is also used as the backbone network, the input dimension is the sum of the pose state dimension and the action dimension, and the output dimension is the skill dimension. The final effect is as followsFigure 2 , Figure 3 As shown in FIG. 13, it can be seen that the present application effectively improves the effect of the robot bottom layer control strategy and the upper layer skill selection strategy model in the final test environment, and has a faster convergence speed compared with other methods, and our method can obtain a higher reward.

[0066] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A robot double-layer control method for selecting skills based on a conditional diffusion model, characterized by, The policy pre-training stage and the policy fine-tuning stage are included. The policy pre-training stage is as follows: Given a set of training data sets with N robot pose trajectories S and action sequences A, initialize the robot bottom control policy and upper control policy model components; B samples are randomly selected from the dataset, and the pose trajectory is randomly sliced in the pose trajectory B with a length of T T ; Using LSTM network as encoder to encode S and A in pose trajectory slice B T to get skill z, and using LSTM-based decoder as bottom-level control policy π lo in double-layer control, according to z and pose state S in B l to restore the corresponding action sequence A, and calculate action imitation loss function Initialization of a robot upper skill selection strategy π based on a diffusion model hi , the pose state S and the skill z in B T are bound through the diffusion model, and the action sequence A is normalized as an additional condition c of the conditional diffusion model, so as to realize the upper skill selection strategy π hi By sampling different skill vectors z through different additional conditions c, the training process of the diffusion model is realized by a loss function ; Determine the total loss function for training the bottom control policy and the upper skill selection policy; The policy fine-tuning stage is as follows: Initialize the conditional selection model component, the robot adopts the conditional selection model π c Generate additional conditions c, and select the strategy π by the upper skill selection policy hi Sample the skill z corresponding to the additional condition c and hand it over to the bottom control policy π lo Restore the action, use the PPO algorithm loss And state value evaluation network loss Find the optimal skill for different pose states to run the robot, and continuously fine-tune the skill selection policy of the upper skill selection policy. Iterative optimization model until the robot bottom control policy and upper skill selection policy model converge. 2.The robot double-layer control method based on the conditional diffusion model skill selection method according to claim 1, wherein, The training data set D = {τ n = (s0, a0, s1, a1, …, s T , a T ) n |n = 1, …, N}, wherein τ represents a sequence of trajectories containing a pose state S and a robot action A, T is the length of the trajectory sequence, s is the pose state of the robot, and a is the action taken by the robot at the current pose state. 3.The robot double-layer control method based on the conditional diffusion model skill selection method according to claim 1, wherein, The initialization of the robot bottom control policy and the upper skill selection policy model components includes: an LSTM-based trajectory sequence encoder E(·), an LSTM-based robot low-level control policy π lo an LSTM-based trajectory sequence encoder E(·), an LSTM-based robot low-level control policy π hi an LSTM-based trajectory sequence encoder E(·), an LSTM-based robot low-level control policy π lo an LSTM-based trajectory sequence encoder E(·), an LSTM-based robot low-level control policy π hi an LSTM-based trajectory sequence encoder E(·), an LSTM-based robot low-level control policy π 4. The robot double-layer control method for selecting skills based on a conditional diffusion model according to claim 1, wherein, The corresponding action sequence A is restored as follows: selecting a sequence of robot poses s of length T from the trajectory data pool 0:T and action sequence a 0:T composing a trajectory sequence [s 0:T , a 0:T ]; first encoding the trajectory sequence by an LSTM-based encoder E to obtain its skill z; and reconstructing the action sequence from the existing pose sequence and skill z by an LSTM-based decoder, so that π lo has the ability to restore the robot action sequence A under the skill z given the known state s.

5. The robot double-layer control method for selecting skills based on a conditional diffusion model according to claim 1, wherein, The action imitation loss function is: Among them, s 0:T ,a 0:t is the pose state and action sequence that the encoder needs to encode into the latent space, z is s 0:T ,a 0:T The skill space vector after encoding, β is the variable that constrains the size of the regularization term, KL is used to calculate the KL divergence between two different distributions, and p(z) represents N(0,I) as the standard Gaussian distribution.

6. The robot double-layer control method for selecting skills based on a conditional diffusion model according to claim 1, wherein, the loss The implementation process is as follows: A diffusion model is introduced to bind the pose state s and the encoded skill z, limiting the skill that the robot should have when exploring the current pose state during exploration. According to the existing trajectory sequence [s 0:T ,a 0:T ], and the skill code z corresponding to the trajectory data, a conditional diffusion model is introduced, and the original action sequence a 0:T is hard-coded into an additional condition c, which is defined as follows: The additional condition c is taken together with the initial state s0 as a condition for generating the corresponding skill, and a loss is designed 7. The robot double-layer control method for selecting skills based on a conditional diffusion model according to claim 1, wherein, The total loss function for training the bottom control policy and the upper skill selection policy is as follows: wherein, is the action imitation loss function, is the loss function for the training process of the diffusion model. 8.The robot double-layer control method for selecting skills based on a conditional diffusion model according to claim 1, wherein, The policy fine-tuning stage is implemented as follows: In the policy fine-tuning stage, the model component is initialized as the conditional selection policy c , the model component hi is frozen, and lo , the state value evaluation network V; the robot generates an optimal skill z under the current state s by selecting the optimal additional condition c, and generates a trajectory data [(s t ,c t ,r t )|t=0,1,…,T'] with reward return by interacting with the environment; wherein T' represents the length of time that the robot interacts with the environment, s t represents the pose state, c t represents the additional condition selected by the robot for selecting the skill, and r t represents the reward obtained by taking the skill under the current state; after collecting the trajectory data D, the PPO algorithm is used to update the policy c , and the state evaluation network V is learned, and the PPO loss and the state evaluation loss are as follows: where A t is the current state s t selects an additional condition c t with a value function of advantage, where higher advantage values represent that the additional condition will result in an excellent skill action.

Images