A robot arm control method based on imitation and parametric two-stage reinforcement learning

Abstract

The application provides a mechanical arm control method based on two-stage reinforcement learning of imitation and parameter variation, comprising the following steps: 1) obtaining an expert motion trajectory of a manually controlled mechanical arm for completing a specified task, and performing normalization processing to obtain an imitation learning data set; 2) training a mechanical arm imitation learning strategy close to the expert strategy by using the obtained data set; 3) designing a parameter efficient fine-tuning strategy based on the imitation learning strategy to interact with the environment for training, and using a value network to guide strategy learning to obtain a final reinforcement learning strategy; and 4) outputting a control action sequence of the mechanical arm by using the reinforcement learning strategy to complete various tasks. The application trains and learns the strategy quickly under the premise of ensuring safety and stability by using a two-stage reinforcement learning method and a parameter efficient fine-tuning strategy, and controls the mechanical arm to complete the specified task by using a small amount of expert demonstration information and a small number of environment interactions.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, specifically to a robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters. Background Technology

[0002] As a form of embodied intelligence, robotic arms are used in fields such as medical collaboration, industrial manufacturing, and aerospace to optimize processes, reduce manual labor, and improve operational precision. However, their complex dynamics and nonlinear characteristics pose challenges to the control of robotic arms.

[0003] Reinforcement learning and imitation learning have proven to be effective methods for intelligently controlling robotic arms to complete challenging tasks. Typical reinforcement learning algorithms design an agent to learn a series of actions in the environment through trial and error to maximize expected reward. However, this usually requires a large amount of sampling interactions, leading to low training efficiency. Furthermore, acquiring data in real-world environments is very expensive and time-consuming, and may also encounter safety issues such as collisions. Imitation learning, on the other hand, learns the actions required for the task by observing expert demonstrations. However, this requires a large amount of high-quality expert demonstration data for training, and the resulting policy can only imitate and approximate the expert policy, not surpass it, and has poor generalization ability. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters, which reduces the risk of catastrophic forgetting and improves the stability of training and the safety of environmental interaction.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A robotic arm control method based on two-stage reinforcement learning involving imitation and variable parameters includes the following steps:

[0007] 1) Obtain the expert motion trajectory of the manually controlled robotic arm to complete the specified task, and normalize it to obtain the imitation learning dataset;

[0008] 2) Based on the dataset obtained in step 1), train a robotic arm imitation learning strategy that approximates the expert strategy;

[0009] 3) Based on the imitation learning strategy in step 2), design a parameter-efficient fine-tuning strategy to interact with the environment for training, and use a value network to guide the policy gradient learning to obtain the final reinforcement learning strategy.

[0010] 4) Utilize the reinforcement learning strategy from step 3) to output the control action sequence of the robotic arm and execute the specified task;

[0011] As a preferred option, the acquired expert motion trajectory dataset It is a series of current state-action-reward-next state tuples , where the state This includes visual information about the task scene and the robotic arm's body information, including its movements. This includes the movements of the robotic arm's end effector and its pose changes in three-dimensional space. The robotic arm performs actions in the state The state changes and a new state is obtained. ,award A reward function designed by the user based on a specified task. get, ...

[0012] As a preferred approach, the first stage is imitation learning, In As input, through imitation learning strategy Get output and with tag action Calculate the mean squared error loss (MSE), and the optimization objective of the strategy is:

[0013]

[0014] in The policy weight parameters include those for feature extraction. Feature modulation and action generation The feature extraction module is a model with a convolutional neural network structure, which can extract features from... Extracting low-dimensional feature vectors The feature modulation module is a fully connected neural network that can perform... Make adjustments The action generation module is a generative model with an encoder-decoder structure that can modulate initial random noise into features. Generation under conditions and Actions with the same shape.

[0015] As a preferred option, the second stage is reinforcement learning, specifically a heterogeneous policy algorithm based on deep deterministic policy gradients, known as the parameter-efficient fine-tuning policy, which includes a policy network. Value network With expert playback buffer pool ,in Initialize to , For a fully connected neural network, according to Output a scalar To evaluate The value of the output action; For a series Includes offline buffer pool and online buffer pool Two parts, For dataset B obtained in the first phase, Acquired in the second phase, for strategy Control the robotic arm to complete the designated task along its trajectory.

[0016] As a preferred option, the designed policy network incorporates a parameter fine-tuning module. Embedded , with feature modulation module Efficient parameter fine-tuning method Combine to obtain new weights ,

[0017]

[0018] The weights obtained in the first stage , and Everything remains frozen and unchanged.

[0019] As a preferred approach, design a reinforcement learning algorithm policy network. The optimization objective is:

[0020]

[0021] Among them It is a behavior cloning regularization term that constrains the update magnitude of the policy. Hyperparameters used to adjust the size of constraints.

[0022] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters.

[0023] This invention has the following characteristics and beneficial effects:

[0024] This invention employs a two-stage reinforcement learning method: first, imitation learning pre-training, and then efficient parameter fine-tuning. Compared to typical imitation learning methods, this approach requires only a small number of expert demonstrations and does not require the demonstration trajectories to be optimal, while the policy can continuously optimize and surpass expert policies. Furthermore, compared to typical reinforcement learning methods, it requires only a small amount of environmental interaction to optimize the policy. Current reinforcement learning methods utilizing demonstration information either directly fine-tune the pre-trained policy or train an additional policy network superimposed on the pre-trained policy, leading to instability in the training process. This invention refines the pre-trained policy into modules such as feature extraction, feature modulation, and action generation. It also designs an efficient parameter fine-tuning strategy that combines with the feature modulation module to freeze the parameters of other modules, reducing the number of trainable parameters and compensating for the huge computational cost of full parameter fine-tuning while maintaining performance comparable to full fine-tuning. Moreover, it retains the knowledge captured by the action generation module while constraining the magnitude of fine-tuning updates, reducing the risk of catastrophic forgetting and improving training stability and the safety of environmental interaction. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the present invention.

[0027] Figure 2 This is a framework diagram of the two-stage reinforcement learning algorithm according to an embodiment of the present invention.

[0028] Figure 3 This is a framework diagram of the action generation module according to an embodiment of the present invention.

[0029] Figure 4 This is a framework diagram of the parameter fine-tuning module in an embodiment of the present invention. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0032] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0033] This invention provides a robotic arm control method based on two-stage reinforcement learning involving imitation and variable parameters, such as... Figure 1 As shown, firstly, the motion trajectories of expert-controlled robotic arms to complete a specified task are acquired and normalized to obtain an imitation learning dataset. Then, using the acquired dataset, a robotic arm imitation learning strategy based on a diffusion model that closely approximates the expert policy is trained. Further, based on the imitation learning strategy, a parameter-efficient policy network is designed to interact with the environment for training, and a value network is designed to guide the policy gradient learning, resulting in the final reinforcement learning strategy. Finally, the control action sequence of the robotic arm is output according to the reinforcement learning strategy to complete the task.

[0034] The embodiments of the present invention are described in detail below.

[0035] Figure 2 This is a framework diagram of the two-stage reinforcement learning algorithm according to an embodiment of the present invention.

[0036] First, obtain the expert demonstration dataset for the specified task. A series of current state-action-reward-next state tuples are defined as follows: , where state Includes visual information about the task scene: this could be a three-dimensional array of RGB images taken by an eye-to-hand external camera. And the robot arm's body information: this could be a one-dimensional vector of the robot arm's end effector's pose in three-dimensional space. ,action It is also a one-dimensional vector, including the motion of the robotic arm's end effector and its pose changes in three-dimensional space. The robotic arm performs actions in the state The state changes and a new state is obtained. ,award A reward function designed by the user based on a specified task. get, .

[0037] In this embodiment, the first stage is imitation learning, and this example uses a diffusion strategy as a specific imitation learning algorithm for illustration.

[0038] Specifically, design an imitation learning strategy. ,Will In As input, through imitation learning strategy Get output and with tag action Calculate the mean squared error loss; the optimization objective of the strategy is:

[0039]

[0040] in, Network weight parameters, including those extracted from features. Feature modulation and action generation The feature extraction module uses a deep residual network, ResNet, which can extract features from... Extracting low-dimensional feature vectors :

[0041]

[0042] The `cat` function can concatenate two vectors along a specified dimension.

[0043] The feature modulation module uses the Feature Linear Modulation Network (FiLM), which can... Adjustments will be made:

[0044]

[0045] MLP is a fully connected neural network. Perform the transformation.

[0046] Finally, the action generation module takes an initial random noise and modulates it using an encoder-decoder structured generative network to generate an action with the same shape as the action corresponding to the state in the dataset.

[0047] The action generation module is a Diffusion generation model with a UNet architecture that can take random action inputs. go through Next iteration:

[0048]

[0049] Denoising the action ,in For a UNet-structured denoising network, the structure is as follows: Figure 3 As shown, It is a hyperparameter for controlling the level of noise. It is Gaussian noise.

[0050] The training process of the denoising network is to first start from Mid-sampling No need in the first phase Then randomly select an iteration round for the iteration process. Gaussian noise was obtained through random sampling. Add to action Online The optimized loss function is:

[0051]

[0052] Optimized As an action generation network.

[0053] The imitation learning strategy obtained from the first stage of pre-training:

[0054]

[0055] Furthermore, the second stage is reinforcement learning, which is a heterogeneous policy algorithm based on deep deterministic policy gradients, called parameter-efficient fine-tuning policy, and includes a policy network. Value network With expert playback buffer pool ,in, Initialize to , For a fully connected neural network, the input Output a scalar To evaluate The value of the output action; For a series Includes offline buffer pool and online buffer pool ; For dataset B obtained in the first phase, Acquired in the second phase, for strategy The trajectory for controlling the robotic arm to complete the specified task is initialized to empty.

[0056] Then, design a module for efficient parameter fine-tuning.

[0057] Figure 4 This is a framework diagram of the parameter fine-tuning module in an embodiment of the present invention.

[0058] This example uses low-rank adaptive LoRA as the specific parameter fine-tuning module. Medium feature modulation module Low-rank adaptation module include Two parameter matrices, where The hyperparameter specifies the rank size, and the dimension reduction matrix... Initialized with a random Gaussian distribution, the matrix is ​​increased in dimensionality. Initialized with a zero matrix. The combination of the feature modulation module and the low-rank adaptation module yields...

[0059]

[0060] in The freeze remains unchanged in the second phase, and in the initial phase... The features obtained by feature modulation become:

[0061]

[0062] Will Embedded The policy network for reinforcement learning was then obtained. Only the weight parameters of the feature modulation module were changed. Without affecting the feature extraction and action generation modules, it allows for interaction with the environment. Obtain the trajectory Placed into the online buffer pool , and will and Merging .

[0063] Each training time step, from Medium sampling batch size tuples .

[0064] Policy Network The loss function is:

[0065]

[0066] Value Network The loss function is:

[0067]

[0068] in For timing difference error, a discount factor is used. :

[0069]

[0070] The pseudocode for the efficient parameter fine-tuning strategy is as follows:

[0071]

[0072] Finally, the obtained reinforcement learning strategy is used. Real-time based on status Calculate the action It drives the robotic arm to complete the designated task.

[0073] This embodiment also provides a computer-readable storage medium storing a computer program, characterized in that when the program is executed by a processor, it implements the above-described robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters.

[0074] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters, characterized in that, Includes the following steps: Step 1: Obtain the expert motion trajectory of the manually controlled robotic arm to complete the specified task, and normalize it to obtain the imitation learning dataset. The imitation learning dataset includes the current state-action-reward-next state tuple. Step 2: Train a robotic arm imitation learning strategy that is close to the expert strategy. The imitation learning strategy is as follows: take the state in the imitation learning dataset as input, extract low-dimensional feature vectors through the feature extraction module, then adjust the low-dimensional feature vectors through a feature modulation module to obtain modulated features, and finally generate an action with the same action shape as the state in the dataset by taking an initial random noise under the modulation feature condition through an encoder-decoder structure generator network. Step 3: Construct a parameter-efficient fine-tuning strategy and train it interactively with the environment. Use a value network to guide the gradient learning of the robotic arm's imitation learning strategy to obtain the final reinforcement learning strategy. The parameter fine-tuning strategy includes a policy network. Value Network With expert playback buffer pool ,in, Initialize to , For a fully connected neural network, the input Output a scalar To evaluate The value of the output action; For a series Includes offline buffer pool and online buffer pool Two parts, For the dataset obtained in step 2, The dataset obtained in step 3 is used as the policy. Control the robotic arm to complete the designated task along its trajectory; Policy network The method for fine-tuning parameters is as follows: adjust the weight parameters of the parameter fine-tuning module. Embedded , with feature modulation weight parameters Efficient parameter fine-tuning method Combine to obtain new weights , Among them, weight parameters , and All remain frozen and unchanged; Step 4: Use the final reinforcement learning strategy to output the control action sequence of the robotic arm and execute the specified task.

2. The robotic arm control method based on two-stage reinforcement learning using imitation and variable parameters as described in claim 1, characterized in that, In step 1, the imitation learning dataset is defined as ,Will A series of current state-action-reward-next state tuples are defined as follows: , where state This includes visual information about the task scene and the robotic arm's body information, including its movements. This includes the movements of the robotic arm's end effector and its pose changes in three-dimensional space. The robotic arm performs actions in the state The state changes and a new state is obtained. ,award A reward function designed by the user based on a specified task. get, .

3. The robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters according to claim 2, characterized in that, In step 2, the training method is as follows: Will In As input, through imitation learning strategy Get output and with tag action Calculate the mean squared error loss; the optimization objective of the strategy is: in, For the strategy weight parameters, the Including feature extraction weight parameters Feature modulation weight parameters Action generation weight parameters .

4. The robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters according to claim 3, characterized in that, In step 2, the imitation learning dataset will be used. The state in As input, a low-dimensional feature vector is extracted using a deep residual network, ResNet. Then, the low-dimensional feature vector is processed by the Feature Linear Modulation Network (FiLM). Adjustments are made to obtain modulation characteristics. Finally, a Diffusion generator network with a UNet structure modulates an initial random noise into its modulated features. Generate actions corresponding to the states in the dataset under certain conditions. Actions with the same shape.

5. The robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters according to claim 4, characterized in that, In step 3, the parameter-efficient fine-tuning strategy is a heterogeneous strategy algorithm based on the parameter-efficient fine-tuning model and the gradient of the deep deterministic strategy.

6. The robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters according to claim 5, characterized in that, In step 3, the reinforcement learning training phase only optimizes the weight parameters for efficient fine-tuning of the parameters. ,Right now: 。 7. The robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters according to claim 5, characterized in that, The parameters are weight parameters that are efficiently fine-tuned. The optimization method is as follows: Among them It is a behavior cloning regularization term that constrains the update magnitude of the policy. Hyperparameters used to adjust the size of constraints.

8. A computer-readable storage medium storing a computer program, characterized in that... When the program is executed by the processor, it implements the robotic arm control method based on two-stage reinforcement learning of imitation and variable parameters as described in any one of claims 1 to 7.

Images