Robot double-layer control method based on conditional diffusion model selection skills

Abstract

The invention discloses a robot double-layer control method based on conditional diffusion model selection skills, which comprises the following steps: constructing a reinforcement learning track with a long time span through a double-layer strategy, and abstracting the reinforcement learning track into a task with a short time span executed by an upper-layer strategy selection skill; in the strategy pre-training stage, additional conditions are introduced to bind the pose state and the selected skills of the robot in a balanced manner, so that the exploration rate of the robot in a reinforcement learning environment is improved, and a robot agent can combine different skills in a shorter time to complete a task; in the strategy fine tuning stage, a reinforcement learning PPO algorithm is introduced to maximize the accumulated reward as a target, and the skill selection strategy of the upper skill selection strategy is continuously fine-tuned until the model converges. According to the method, a smaller skill space can be sampled from an original action space in a long-time-span robot task, and the correlation between different pose states and skills of the robot can be bound, so that the skill exploration space is reduced to accelerate exploration.

Description

Technical Field

[0001] The invention belongs to the field of reinforcement learning classification, and in particular relates to a robot double-layer control method for selecting skills based on a conditional diffusion model. Background Art

[0002] Reinforcement learning is driving the development of different technologies in the current field of machine learning, especially in the field of robot motion planning and control. End-to-end reinforcement learning has shown remarkable results, whether it is from bionic robots to bionic arms. However, the training process of reinforcement learning requires the robot to continuously explore in the simulation environment and use the existing trajectory data to find the optimal strategy. However, this also introduces a lot of problems. For example, when the robot interacts in real scenes, it will generate high interaction costs, and the final performance of the robot is directly affected by the exploration strategy.

[0003] In practical applications, when a robot is required to complete a task that takes a long time to obtain a reward, it will also face the sparse reward problem. This problem will cause the robot's control strategy to be unable to obtain the optimal feedback in a timely manner, and too sparse rewards will make most actions in the trajectory sequence unable to obtain reasonable feedback information, which will greatly hinder the improvement of the robot's intelligent agent strategy performance.

[0004] In order to meet these challenges, this paper proposes a two-layer control method for robots based on conditional diffusion model to select skills. By constructing a two-layer control strategy, the original long-span trajectory sequence is compressed into a shorter time series trajectory, thereby solving the problem of unclear model feedback under sparse reward problems. At the same time, the diffusion model is introduced to bind the current state with the selectable skills, thereby discarding open set skills and improving exploration efficiency. Summary of the invention

[0005] Purpose of the invention: In order to improve the efficiency of robot interactive exploration and solve the problem of slow strategy update under sparse reward problems, the present invention provides a robot two-layer control method for selecting skills based on a conditional diffusion model.

[0006] Technical solution: The present invention discloses a dual-layer control method for a robot based on a conditional diffusion model to select skills, including a strategy pre-training stage and a strategy fine-tuning stage;

[0007] The strategy pre-training stage is:

[0008] Given a training data set with N robot pose trajectories S and action sequences A, initialize the robot's underlying control strategy and upper-level control strategy model components;

[0009] Randomly select B samples from the dataset and randomly slice out a pose trajectory B of length T in the pose trajectory T ;

[0010] Use LSTM network as encoder to slice pose trajectory B T The S and A in are encoded to obtain the skill z, and the LSTM-based decoder is used as the underlying control strategy π in the two-layer control lo , according to z and B l The pose state S in the image is used to restore the corresponding action sequence A and calculate the action imitation loss function

[0011] Initialize the robot upper-level skill selection strategy π based on the diffusion model hi , B T The posture state S and skill z in are bound through the diffusion model, and the action sequence A is normalized as an additional condition c of the conditional diffusion model to realize the upper-level skill selection strategy π hi By sampling different skill vectors z with different additional conditions c, the training process of the diffusion model is determined by the loss function accomplish;

[0012] Determine the total loss function used to train the underlying control strategy and the upper-level skill selection strategy;

[0013] The strategy fine-tuning phase is:

[0014] Initialize the conditional selection model component, the robot uses the conditional selection model π c Generate additional conditions c, and select strategy π from the upper-level skills hi Sample the skill z corresponding to the additional condition c and pass it to the underlying control strategy π lo Restore action, using PPO algorithm loss and state value to evaluate network loss Find the optimal skills for different posture states to run the robot, and continuously fine-tune the skill selection strategy of the upper-level skill selection strategy, iteratively optimize the model until the robot's underlying control strategy converges with the upper-level skill selection strategy model.

[0015] Furthermore, the training data set D = {τ n =(s0,a0,s1,a1,……,s T ,a T ) n |n=1,…,N}, where τ represents a trajectory sequence containing posture state S and robot action A, T is the length of the trajectory sequence, s is the robot posture state, and a is the action taken by the robot in the current posture state.

[0016] Furthermore, the initialization robot bottom-level control strategy and upper-level skill selection strategy model components include:

[0017] LSTM-based trajectory sequence encoder E(·), LSTM-based robot underlying control strategy π lo (·), upper skill selection strategy π based on diffusion model hi (·); where E(·) is responsible for extracting trajectory features, π lo (·) is responsible for restoring the robot action a under the current posture state s when given skill z, π hi (·) is responsible for restoring the available skill z corresponding to the current conditions and posture state under the given diffusion model conditions.

[0018] Furthermore, the process of restoring the corresponding action sequence A is as follows:

[0019] Select B groups of robot pose sequences s with a length of T from the trajectory data pool 0:T and action sequence a 0:T Composition trajectory sequence [s 0:T ,a 0:T ]; first, the LSTM-based encoder E encodes the trajectory sequence to obtain its skill z; and the LSTM-based decoder reconstructs the existing posture sequence and skill z to perform the action sequence, so that π lo It is capable of restoring the robot action sequence A under the condition of known state s and skill z.

[0020] Furthermore, the action mimics the loss function for:

[0021]

[0022] Among them, s 0:T ,a 0:T is the pose state and action sequence that the encoder needs to encode into the latent space, z is s 0:T ,a 0:T The skill space vector after encoding, β is the variable that constrains the size of the regularization term, KL is used to calculate the KL divergence between two different distributions, and p(z) represents N(0,I) as the standard Gaussian distribution.

[0023] Furthermore, the loss The implementation process is as follows:

[0024] The diffusion model is introduced to bind the posture state s and the skill z obtained after encoding, limiting the robot to explore the skills that should be possessed by the current posture state during the exploration process; according to the existing trajectory sequence [s 0:T ,a 0:T ], and the skill code z corresponding to the trajectory data, the conditional diffusion model is introduced to transform the original action sequence a 0:THard-coded into additional condition c, the specific definition is as follows:

[0025]

[0026] The additional condition c and the initial state s0 are used as the conditions for generating the corresponding skills, and the loss is designed

[0027]

[0028] Furthermore, the total loss function for training the underlying control strategy and the upper-level skill selection strategy is:

[0029]

[0030] in, is the action imitation loss function, is the loss function of the diffusion model training process.

[0031] Furthermore, the specific implementation process of the strategy fine-tuning stage is as follows:

[0032] In the policy fine-tuning phase, the model components are initialized to the conditional selection strategy π c , freeze the model component π hi and π lo , state value evaluation network V; the robot generates the optimal skill z under the current state s by selecting the optimal additional condition c, and generates a trajectory data with reward return by interacting with the environment [(s t ,c t ,r t )|t=0,1,…,T']; where T' represents the duration of the robot's interaction with the environment, s t represents the posture state, c t Represents the additional conditions for the robot to select skills, r t Represents the reward for taking the skill in the current state; after collecting the trajectory data D, use the PPO algorithm to c Update the strategy and learn the state evaluation network V at the same time, PPO loss and state assessment losses as follows:

[0033]

[0034] Among them, A t is the current state t Select additional conditions c t The advantage value function, the higher the advantage, the more likely the additional condition will become an excellent skill action.

[0035] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: the present invention constructs the original reinforcement learning trajectory with a long time span through a two-layer strategy, abstracts it into a task with a shorter time span executed by the upper-layer strategy selecting skills, thereby reducing the time span with the skill time length as the denominator, thereby significantly solving the problem of difficult strategy training for sparse rewards, and achieving better results on long-time span tasks; at the same time, the present invention introduces additional conditions to balance the robot's posture state and selected skills, thereby improving the robot's exploration rate in the reinforcement learning environment, allowing the robot agent to combine different skills in a shorter time to complete the task; the present invention adopts an iterative optimization algorithm, in the strategy pre-training stage The mean square error loss is used in the first stage, and the reinforcement learning PPO algorithm is introduced in the strategy fine-tuning stage to maximize the cumulative reward. The process is simple and efficient, easy to implement and apply; the present invention can apply the initial strategy obtained by pre-training to different downstream reinforcement learning tasks, and can have very high performance in different downstream task environments. The pre-training model improves the generalization ability of reinforcement learning in different scenario tasks, and transforms the original paradigm of training intelligent agents from scratch into a paradigm of model fine-tuning, which improves the application scenarios and reusability of the model; the present invention shows significant superiority in robot arm control and bionic operation, and can be widely used in robot tasks that need to complete tasks with a long time span, and has important practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the present invention;

[0037] Figure 2 It is a curve diagram showing the change of the success rate of the present invention in several robot control scenarios with the number of model learning rounds;

[0038] Figure 3 This is a curve chart showing how the total reward of the present invention in several robot control scenarios changes with the number of model learning rounds. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, the present invention proposes a robot double-layer control method for selecting skills based on a conditional diffusion model, designs a model pre-training stage and a fine-tuning stage, and significantly improves the reusability and generalization of the model algorithm. The specific implementation process is as follows:

[0041] Steps 1 to 4 are the strategy pre-training stage; step 5 is the strategy fine-tuning stage.

[0042] Step 1: Given a set of training data with N robot pose trajectories S and action sequences A, initialize the model components.

[0043] Training data set D = {τ n =(s0,a0,s1,a1,……,s T ,a T ) n |n=1,…,N}, where τ represents a trajectory sequence containing posture state S and robot action A, T is the trajectory sequence length, s is the robot posture state, and a is the action taken by the robot in the current posture state; the initialization model components include: LSTM-based trajectory sequence encoder E(·), LSTM-based robot underlying control strategy π lo (·), upper skill selection strategy π based on diffusion model hi (·). E(·) is responsible for extracting trajectory features, π lo (·) is responsible for restoring the robot action a under the current posture state s when given skill z, π hi (·) is responsible for restoring the available skill z corresponding to the current conditions and posture state under the given diffusion model conditions.

[0044] In this embodiment, the robot arm control is used as the task scenario. In the pre-training stage, the robot is allowed to randomly operate its arm to obtain different skill trajectories τ = (s0, a0, s1, a1, ..., s T ,a T ), the skill trajectory time length is T.

[0045] Step 2: Randomly select B samples from the data set and randomly slice out a pose trajectory B of length T from these pose trajectories. T .

[0046] In this stage, a batch of data is randomly taken from the set D as training data each time, where the entire time series trajectory is regarded as the abstract skill and defined as z in the abstract space.

[0047] Step 3: Use the LSTM network as an encoder to slice the pose trajectory B T The skill vector z is obtained by encoding S and A in , and the LSTM-based decoder is used as the underlying control strategy π in the two-layer control lo , according to z and B l The pose state S in the image is used to restore the corresponding action sequence A and calculate the action imitation loss function

[0048] Select B groups of robot pose sequences s with a length of T from the trajectory data pool 0:T and action sequence a 0:T Composition trajectory sequence [s 0:T ,a 0:T]. First, the LSTM-based encoder E encodes the trajectory sequence to obtain its skill z. Then, the LSTM-based decoder reconstructs the existing posture sequence and skill z into an action sequence, so that π lo It is capable of restoring the robot's actions under the condition of known state s and skill z.

[0049] Using reconstruction loss and considering the characteristics of the LSTM model, the loss function used to reconstruct the robot's actions for:

[0050]

[0051] Among them, s 0:T ,a 0:T is the pose state and action sequence that the encoder needs to encode into the latent space, z is s 0:T ,a 0:T The skill space vector after encoding, β is the variable that constrains the size of the regularization term, KL is used to calculate the KL divergence between two different distributions, p(z) represents N(0,I) is a standard Gaussian distribution, the purpose is to make the overall latent space features encoded by the encoder the same as those from the standard Gaussian distribution.

[0052] Step 4: Initialize the robot upper skill selection strategy π based on the diffusion model hi , B T The posture state S in step 1 and the skill z obtained in step 3 are bound through the diffusion model, and the action sequence A is normalized as the condition c of the conditional diffusion model, so that a single posture state can sample different skills through different additional conditions c. This goal is determined by the loss function accomplish.

[0053] Considering that the skills that the robot can use in a certain posture state s are limited, a diffusion model is introduced to bind the posture state s and the encoded skill z to limit the robot to explore the skills that should be available in the current posture state during the exploration process. 0:T ,a 0:T ], and the skill z corresponding to the trajectory data, we introduce the conditional diffusion model to transform the original action sequence a 0:T Hard-coded into additional condition c, the specific definition is as follows:

[0054]

[0055] The additional condition c and the initial state s0 are used as the conditions for generating the corresponding skills, and the loss is designed

[0056]

[0057] Step 5: Initialize the conditional selection model component. The robot uses the conditional selection model π c Generate additional conditions c, and select strategy π from the upper-level skills hi Sample the skill z corresponding to the additional condition c and pass it to the underlying control strategy π lo Restore action, using PPO algorithm loss and state value to evaluate network loss Find the optimal skills for different posture states to run the robot, continuously fine-tune the skill selection strategy of the upper-level strategy, and iteratively optimize the model until the model converges.

[0058] In the policy fine-tuning phase, the model components are initialized to the conditional selection strategy π c , freeze the model component π hi and π lo , state value evaluation network V. The robot generates the optimal skill z under the current state s by selecting the optimal additional condition c, and generates a trajectory data [(s t ,c t ,r t )|t=0,1,…,T'], where T' represents the duration of the robot's interaction with the environment, s t represents the posture state, c t Represents the additional conditions for the robot to select skills, r t Represents the reward obtained by taking the skill in the current state. After collecting the trajectory data D, use the PPO algorithm to c Update the strategy and learn the state evaluation network V at the same time, PPO loss and state assessment losses as follows:

[0059]

[0060] Among them, A t is the current state t Select additional conditions c t The advantage value function, the higher the advantage, the more likely the additional condition will become an excellent skill action.

[0061] At this point, the pre-training and fine-tuning training process of the robot's underlying control strategy and upper-level skill selection strategy model is completed. If the robot's underlying control strategy and upper-level control strategy models have converged, stop training and output the model module required for classification: underlying control strategy π lo (·), upper-level skill selection strategy π hi (·) and the conditional selection strategy π c(·); otherwise, return to step 5 to continue fine-tuning training. The output pre-trained model can be directly used as the initial strategy in the robot arm operation environment, and the output fine-tuned model can be directly applied to a specific robot arm to control a character. In this embodiment, it is only necessary to input the current robot posture state s into π c (·) Generate additional condition c, and input the posture state s and additional condition c into the skill selector π hi (·) Generate skill z, and finally input skill z and posture state s into the lower controller π lo (·) can output the optimal action of the robot.

[0062] The present invention is particularly suitable for the fields of robot motion planning and control. Traditional reinforcement learning tasks do not take into account the sparse reward problem in long time spans, which makes it difficult to solve long time span tasks. At the same time, the existing two-layer control algorithm does not take into account the exploration efficiency problem, and often explores from all skills or uneven skills, which reduces the speed at which the robot acquires the optimal skills. The invention introduces additional conditions, hierarchical reinforcement learning and diffusion models, and proposes a new two-layer control method, which solves the sparse reward problem and effectively improves the robot's exploration efficiency.

[0063] To verify the effectiveness of the present invention, the effects of the proposed method on three robot arm control tasks are given in this embodiment: Table-Cleanup, Slippery-Push and Pyramid-Stack, and compared with the most advanced methods.

[0064] The evaluation indicators of the comparative experiment are the success rate in the test environment of different downstream tasks, which measures the success rate of different tasks, and the reward return in the test environment of different downstream tasks, which measures the reward return of different tasks. The higher the reward, the faster the completion speed.

[0065] All methods use the same diffusion model architecture as the upper-level skill selection strategy π hi (·) with a diffusion time step of 100. For the conditional selector π c (·), select a three-layer multi-layer perceptron MLP as its basic architecture, the input dimension is the posture state dimension, the hidden layer dimension is 128, and the output dimension is the action dimension. For the lower layer strategy π lo (·), LSTM is used as the backbone network, the input dimension is the sum of the posture dimension and the skill dimension, and the output dimension is also the action dimension. For the skill encoding network E, LSTM is also used as the backbone network, the input dimension is the sum of the posture state dimension and the action dimension, and the output dimension is the skill dimension. The final effect is as follows Figure 2 , Figure 3 As shown, it can be seen that the present invention effectively improves the effects of the robot's underlying control strategy and the upper-level skill selection strategy model in the final test environment, and has a faster convergence speed than other methods. At the same time, our method can obtain higher rewards.

[0066] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A robot double-layer control method for selecting skills based on a conditional diffusion model, characterized in that: Includes strategy pre-training stage and strategy fine-tuning stage; The strategy pre-training stage is: Given a training data set with N robot pose trajectories S and action sequences A, initialize the robot's underlying control strategy and upper-level control strategy model components; Randomly select B samples from the dataset and randomly slice out a pose trajectory B of length T in the pose trajectory T ; Use LSTM network as encoder to slice pose trajectory B T The S and A in are encoded to obtain the skill z, and the LSTM-based decoder is used as the underlying control strategy π in the two-layer control lo , according to z and B l The pose state S in the image is used to restore the corresponding action sequence A and calculate the action imitation loss function Initialize the robot upper-level skill selection strategy π based on the diffusion model hi , B T The posture state S and skill z in are bound through the diffusion model, and the action sequence A is normalized as an additional condition c of the conditional diffusion model to realize the upper-level skill selection strategy π hi By sampling different skill vectors z with different additional conditions c, the training process of the diffusion model is determined by the loss function accomplish; Determine the total loss function used to train the underlying control strategy and the upper-level skill selection strategy; The strategy fine-tuning phase is: Initialize the conditional selection model component, the robot uses the conditional selection model π c Generate additional conditions c, and select strategy π from the upper-level skills hi Sample the skill z corresponding to the additional condition c and pass it to the underlying control strategy π lo Restore action, using PPO algorithm loss and state value to evaluate network loss Find the optimal skills for different posture states to run the robot, and continuously fine-tune the skill selection strategy of the upper-level skill selection strategy, iteratively optimize the model until the robot's underlying control strategy converges with the upper-level skill selection strategy model.

2. A robot double-layer control method for selecting skills based on a conditional diffusion model according to claim 1, characterized in that: The training data set D = {τ n =(s0,a0,s1,a1,……,s T ,a T ) n |n=1,…,N}, where τ represents a trajectory sequence containing posture state S and robot action A, T is the length of the trajectory sequence, s is the robot posture state, and a is the action taken by the robot in the current posture state.

3. The robot double-layer control method based on conditional diffusion model selection skill according to claim 1 is characterized in that: The initialization robot bottom-level control strategy and upper-level skill selection strategy model components include: LSTM-based trajectory sequence encoder E(·), LSTM-based robot underlying control strategy π lo (·), upper skill selection strategy π based on diffusion model hi (·); where E(·) is responsible for extracting trajectory features, π lo (·) is responsible for restoring the robot action a under the current posture state s when given skill z, π hi (·) is responsible for restoring the available skill z corresponding to the current conditions and posture state under the given diffusion model conditions.

4. The robot double-layer control method for selecting skills based on conditional diffusion model according to claim 1 is characterized in that: The process of restoring the corresponding action sequence A is as follows: Select B groups of robot pose sequences s with a length of T from the trajectory data pool 0:T and action sequence a 0:T Composition trajectory sequence [s 0:T ,a 0:T ]; first, the LSTM-based encoder E encodes the trajectory sequence to obtain its skill z; and the LSTM-based decoder reconstructs the existing posture sequence and skill z to perform the action sequence, so that π lo It is capable of restoring the robot action sequence A under the condition of known state s and skill z.

5. The robot double-layer control method for selecting skills based on conditional diffusion model according to claim 1 is characterized in that: The action imitation loss function for: Among them, s 0:T ,a 0:t is the pose state and action sequence that the encoder needs to encode into the latent space, z is s 0:T ,a 0:T The skill space vector after encoding, β is the variable that constrains the size of the regularization term, KL is used to calculate the KL divergence between two different distributions, and p(z) represents N(0,I) as the standard Gaussian distribution.

6. The robot double-layer control method for selecting skills based on conditional diffusion model according to claim 1 is characterized in that: The loss The implementation process is as follows: The diffusion model is introduced to bind the posture state s and the skill z obtained after encoding, limiting the robot to explore the skills that should be possessed by the current posture state during the exploration process; according to the existing trajectory sequence [s 0:T ,a 0:T ], and the skill code z corresponding to the trajectory data, the conditional diffusion model is introduced to transform the original action sequence a 0:T Hard-coded into additional condition c, the specific definition is as follows: The additional condition c and the initial state s0 are used as the conditions for generating the corresponding skills, and the loss is designed 7. The robot double-layer control method for selecting skills based on conditional diffusion model according to claim 1 is characterized in that: The total loss function used to train the underlying control strategy and the upper-level skill selection strategy is: in, is the action imitation loss function, is the loss function of the diffusion model training process.

8. The robot double-layer control method for selecting skills based on conditional diffusion model according to claim 1 is characterized in that: The specific implementation process of the strategy fine-tuning stage is as follows: In the policy fine-tuning phase, the model components are initialized to the conditional selection strategy π c , freeze the model component π hi and π lo , state value evaluation network V; the robot generates the optimal skill z under the current state s by selecting the optimal additional condition c, and generates a trajectory data with reward return by interacting with the environment [(s t ,c t ,r t )|t=0,1,…,T']; where T' represents the duration of the robot's interaction with the environment, s t represents the posture state, c t Represents the additional conditions for the robot to select skills, r t Represents the reward for taking the skill in the current state; after collecting the trajectory data D, use the PPO algorithm to c Update the strategy and learn the state evaluation network V at the same time, PPO loss and state assessment losses as follows: Among them, A t is the current state t Select additional conditions c t The advantage value function, the higher the advantage, the more likely the additional condition will become an excellent skill action.

Images