A Robot Assembly Method and System Based on Feature Adaptive Transfer Reinforcement Learning

Abstract

This invention provides a robot assembly method and system based on feature adaptive transfer reinforcement learning. This invention combines source domain expert data information, filters data based on the distance between features of the source domain and the target domain, and obtains the assembly strategy of the target domain more quickly through the transfer of prior knowledge between the source domain and the target domain.

Description

Technical Field

[0001] This invention belongs to the field of robot assembly technology, and particularly relates to a robot assembly method and system based on feature adaptive transfer reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The diversification of assembly tasks and assembly objects is one of the urgent problems that needs to be solved to improve the flexible manipulation skills of robots. In the assembly of shafts and holes and complex parts, due to the diversification of assembly objects and the complexity of assembly models, strategies trained on one assembly task often fail to achieve ideal assembly results when directly transferred to another assembly task. Moreover, new assembly tasks often lack sufficient interactive data due to the fragility of parts.

[0004] Therefore, how to make the robot's assembly skills more generalized so that it can handle the assembly of various complex objects, and reduce the interaction time and iteration number on new tasks by utilizing prior knowledge from the original task, thereby reducing interaction costs, is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a robot assembly method and system based on feature adaptive transfer reinforcement learning. This invention combines source domain expert data information, utilizes prior knowledge from the original task, filters data based on the distance between features of the source and target domains, and obtains the assembly strategy for the target domain more quickly through the transfer of prior knowledge between the source and target domains.

[0006] To achieve the above objectives, a first aspect of the present invention provides a robot assembly method based on feature adaptive transfer reinforcement learning, comprising:

[0007] Acquire robotic arm feature data in the source and target domains;

[0008] After processing the robotic arm feature data in the source and target domains, the MMD distance between them is calculated, and the robotic arm feature data in the source domain with an MMD distance less than a set threshold is stored in the dataset.

[0009] Establish a transfer reinforcement learning model and pre-train it using data from the dataset.

[0010] The pre-trained transfer reinforcement learning model is used to continue training on the target domain until a well-trained transfer reinforcement learning model is obtained that can input assembly tasks on the target domain and output assembly actions.

[0011] A second aspect of the present invention provides a robot assembly system based on feature adaptive transfer reinforcement learning, comprising:

[0012] Data acquisition module: used to acquire robotic arm feature data in the source and target domains;

[0013] Distance metric module: This module processes the robotic arm feature data in the source and target domains, calculates the MMD distance between them, and stores the robotic arm feature data in the source domain whose MMD distance is less than a set threshold into the dataset.

[0014] Model building and pre-training module: used to build transfer reinforcement learning models and pre-train them using data within the dataset;

[0015] The retraining module uses the pre-trained transfer reinforcement learning model to continue training on the target domain until a trained transfer reinforcement learning model is obtained that can input the assembly task on the target domain and output assembly actions.

[0016] A third aspect of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps described in the above method.

[0017] A fourth aspect of the present invention provides an electronic device including a memory and a processor, and computation stored in the memory and running on the processor.

[0018] The beneficial effects of this invention are:

[0019] This invention combines source domain expert data information and filters data based on the distance between the features of the source domain and the target domain. By transferring prior knowledge between the source domain and the target domain, the assembly strategy of the target domain can be obtained more quickly. Compared with exploring directly in the target domain, it can reduce the interaction with the target domain environment and save manpower costs.

[0020] In this invention, the update method of the model network is improved by selecting the interactive data closest to the target trajectory to update the network, making the training process of the network more stable. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a schematic diagram of the robot assembly process in Embodiment 1 of the present invention;

[0023] Figure 2 This is a schematic diagram of the overall process of transfer reinforcement learning in Embodiment 1 of the present invention;

[0024] Figure 3 This is a schematic diagram of the robot assembly model network flow in Embodiment 1 of the present invention;

[0025] Figure 4 This is a schematic diagram of the MMD distance data filtering process in Embodiment 1 of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] Example 1

[0029] like Figure 1 As shown, the robot assembly method based on feature adaptive transfer reinforcement learning includes:

[0030] Step 1: Obtain the robotic arm feature data in the source and target domains;

[0031] Step 2: After processing the robotic arm feature data in the source and target domains, calculate the MMD distance between them, and store the robotic arm feature data in the source domain with an MMD distance less than a set threshold into the dataset.

[0032] Step 3: Build a transfer reinforcement learning model and pre-train the model using data from the dataset;

[0033] Step 4: Continue training the pre-trained transfer reinforcement learning model on the target domain until a well-trained transfer reinforcement learning model is obtained that can input the assembly task into the target domain and output assembly actions.

[0034] In step 1 of this embodiment, six-dimensional force information and image information are acquired using a force sensor and a depth camera to initialize the robotic arm and obtain test data of the robotic arm in the source domain. An assembly round is represented as: Collect a small amount of teaching data from the robotic arm in the target domain. s t =(s p ,s τ ), where s p = [x,y,z,α,β,γ], representing the pose of the end effector of the robotic arm, s τ =[F x ,F y ,F z M x My M z ] represents the contact force / torque at the end of the robotic arm, a t =[Δx,Δy,Δz,Δα,Δβ,Δγ] represents the next assembly action of the robotic arm.

[0035] like Figure 4 As shown, in step 2 of this embodiment, the robotic arm feature data D in the source domain is... s and robotic arm feature data D in the target domain t Normalization is performed, and the MMD distance between the source domain robotic arm feature data and the target domain robotic arm feature data is calculated using formula (1).

[0036]

[0037] Where, ξ i and ζ i φ(·) represents an assembly round in the source domain and an assembly round in the target domain, respectively. φ(·) is a mapping used to map the original variables to the reproducing kernel Hilbert space (RKHS).

[0038] like Figures 2-3 As shown, in step 3 of this embodiment, the transfer reinforcement learning model includes the old policy network Actor. old New Strategy Network Actor new And evaluation network Critic;

[0039] The new strategy network takes the current assembly state of the robotic arm as input and outputs the next assembly action of the robotic arm. The next assembly state of the robotic arm is obtained by interacting with the environment.

[0040] The evaluation network is used to calculate the value corresponding to the final step of the robotic arm assembly state output by the new strategy network;

[0041] The old strategy network takes the current assembly state of the robotic arm as input and outputs the next assembly action of the robotic arm. The weights of the old strategy network are updated by the weights of the new strategy network.

[0042] The new policy network, the old policy network, and the policy network all consist of an input layer, several intermediate layers, and an output layer. The output layer is activated by an activation function to scale the output data to a specified range. The new policy network and the old policy network have the same structure, but the number of network layers and neurons in the evaluation network and the policy network are not exactly the same.

[0043] During the pre-training phase, the inputs to the new policy network and the old policy network are the assembly state of the robotic arm in the source domain; during the retraining phase, the inputs to the new policy network and the old policy network are the assembly state of the robotic arm in the target domain.

[0044] Step 3-1: Actor new The input is the current robot assembly state s. t The network outputs the next assembly action of the robotic arm, and outputs μ2 and σ2. A normal distribution is constructed using μ2 and σ2. μ2 and σ2 are the mean and standard deviation of a normal distribution, respectively. The next assembly action 'a' of the robotic arm is obtained by sampling from N2. t The next state s is obtained by interacting with the environment. t+1 The reward value r is calculated by using a reward function designed based on assembly force, assembly depth, and assembly steps. t Calculate the MMD distance between features and target domain features in each assembly round of transfer reinforcement learning, and exclude data whose MMD distance is less than a set threshold. t ,a t ,r t ,s t+1 Stored in the experience recycling pool.

[0045] The reward function is expressed as follows:

[0046] r = λ1r1 + λ2r2 + λ3r3 + λ4r4

[0047] in:

[0048] r1 = -step / step_max

[0049] r2=|Z min -Z| / 0.05

[0050]

[0051] r4=|Δz| / Δz max

[0052] Where λ1, λ2, λ3, and λ4 are the weights of each term in the reward function, which can be adjusted appropriately according to the training effect; step_max is the maximum allowed assembly steps, set to 30; Z min Z and f represent the maximum assembly depth and the current assembly depth, respectively. i / τ i and f max / τ max Force / torque and maximum assembly force / torque in the X, Y, and Z directions, respectively; Δz and Δz maxThese represent the single-step assembly depth and the maximum single-step assembly depth, respectively. The coefficients in the formula can be adjusted based on the convergence of the assembly process and must satisfy... λ 31 +λ 32 =1.

[0053] Step 3-2: Input the state s from the last step in the experience recycling pool into the Critic network to obtain the v of that state. t The value is used to calculate the discount reward using formula (2):

[0054] R t =r t +γr t+1 +γ 2 r t+2 +…+γ T-t+1 r T-1 +γ T-t v t (2)

[0055] Where γ is the discount factor, r t These are the reward functions for each step.

[0056] The last batch of states s selected from the experience pool (where batch is a value chosen based on network training performance, typically a power of 2) is input into the Critic network to obtain the v of all states. t Value, and calculate A t :

[0057] A t =R t -v t (3)

[0058] Where R t v is the discount reward value. t The value corresponding to the state.

[0059] The loss function for the Critic network is:

[0060] c loss =mean(square(A) t (4)

[0061] Using c loss Backpropagation updates the Critic network.

[0062] Step 3-3: Input the combination of the last stored batch group states s into the Actor. old And Actor new In a network, a normal distribution is constructed using the network output values. and And store the last batch group a t Input into normal distributions N1 and N2, a t Using the horizontal axis of the normal distribution, we obtain the corresponding probabilities prob1 and prob2. Then, we divide prob2 by prob1 to obtain the importance weight ratio.

[0063] Steps 3-4:

[0064] Calculate the loss function al of the Actor network oss :

[0065]

[0066] Using loss function a loss Update Actor new Network weight.

[0067] Step 3-5: Repeat steps 3-3 to 3-4, looping for several steps, then use the Actor. new Network weights to update Actors old Weights.

[0068] Steps 3-6: Save the pre-trained network model and use the pre-trained model as an Actor. old Actor new And the initialization network of Critic.

[0069] Step 3-7: Repeat steps 3-1 to 3-5 to retrain the model in the target domain until the model converges and achieves the expected results.

[0070] Example 2

[0071] This embodiment provides a robot assembly system based on feature adaptive transfer reinforcement learning, including:

[0072] Data acquisition module: used to acquire robotic arm feature data in the source and target domains;

[0073] Distance Measurement Module: This module processes the robotic arm feature data in the source and target domains, calculates the MMD distance between them, and stores the robotic arm feature data in the source domain whose MMD distance is less than a set threshold into the data experience pool.

[0074] Model building and pre-training module: used to build transfer reinforcement learning models and pre-train them using data from the data experience pool;

[0075] The retraining module uses the pre-trained transfer reinforcement learning model to continue training on the target domain until a trained transfer reinforcement learning model is obtained that can input the assembly task on the target domain and output assembly actions.

[0076] Example 3

[0077] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0078] Example 4

[0079] The purpose of this embodiment is to provide a computer-readable storage medium.

[0080] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0081] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0082] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0083] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robot assembly method based on feature adaptive transfer reinforcement learning, characterized in that, include: Acquire robotic arm feature data in the source and target domains; After processing the robotic arm feature data in the source and target domains, the MMD distance between them is calculated, and the robotic arm feature data in the source domain with an MMD distance less than a set threshold is stored in the dataset. A transfer reinforcement learning model is established, and the model is pre-trained using data from the dataset. The transfer reinforcement learning model includes an old policy network, a new policy network, and an evaluation network. The new strategy network takes the current assembly state of the robotic arm as input and outputs the next assembly action of the robotic arm. The next assembly state of the robotic arm is obtained by interacting with the environment. The evaluation network is used to calculate the value corresponding to the final step of the robotic arm assembly state output by the new strategy network; The input to the old policy network is the current assembly state of the robotic arm, and the output is the next assembly action of the robotic arm. Its weights are updated by the weights of the new policy network. Calculate the MMD distance between the features and the target domain features in each assembly round of transfer reinforcement learning, and store data with an MMD distance less than a set threshold into the experience recycling pool; Based on the output values ​​of the new policy network and the old policy network, a first normal distribution and a second normal distribution are constructed. The last robotic arm action put into the experience recycling pool is output to the first normal distribution and the second normal distribution to obtain the corresponding first probability and second probability. The importance weights are obtained based on the first probability and the second probability. The loss function of the new policy network is constructed based on the importance weights. The weights of the new policy network are updated using the loss function. The pre-trained transfer reinforcement learning model is used to continue training on the target domain until a well-trained transfer reinforcement learning model is obtained that can input assembly tasks on the target domain and output assembly actions.

2. The robot assembly method based on feature adaptive transfer reinforcement learning as described in claim 1, characterized in that, After normalizing the robotic arm feature data in the source and target domains, the difference between the two is measured using MMD distance.

3. The robot assembly method based on feature adaptive transfer reinforcement learning as described in claim 1, characterized in that, The discounted reward is calculated based on the value corresponding to the robotic arm assembly state output by the evaluation network. Based on the value and discounted reward corresponding to the robotic arm assembly state, the loss function of the evaluation network is determined, and the evaluation network is updated based on the loss function of the evaluation network.

4. The robot assembly method based on feature adaptive transfer reinforcement learning as described in claim 1, characterized in that, The feature data includes the robot arm's pose, contact torque, and the robot arm's next assembly action.

5. A robot assembly system based on feature adaptive transfer reinforcement learning, characterized in that, include: Data acquisition module: used to acquire robotic arm feature data in the source and target domains; Distance metric module: This module processes the robotic arm feature data in the source and target domains, calculates the MMD distance between them, and stores the robotic arm feature data in the source domain whose MMD distance is less than a set threshold into the dataset. Model building and pre-training module: used to build a transfer reinforcement learning model and pre-train the transfer reinforcement learning model using data within the dataset; the transfer reinforcement learning model includes an old policy network, a new policy network, and an evaluation network; The new strategy network takes the current assembly state of the robotic arm as input and outputs the next assembly action of the robotic arm. The next assembly state of the robotic arm is obtained by interacting with the environment. The evaluation network is used to calculate the value corresponding to the final step of the robotic arm assembly state output by the new strategy network; The input to the old policy network is the current assembly state of the robotic arm, and the output is the next assembly action of the robotic arm. Its weights are updated by the weights of the new policy network. Calculate the MMD distance between the features and the target domain features in each assembly round of transfer reinforcement learning, and store data with an MMD distance less than a set threshold into the experience recycling pool; Based on the output values ​​of the new policy network and the old policy network, a first normal distribution and a second normal distribution are constructed. The last robotic arm action put into the experience recycling pool is output to the first normal distribution and the second normal distribution to obtain the corresponding first probability and second probability. The importance weights are obtained based on the first probability and the second probability. The loss function of the new policy network is constructed based on the importance weights. The weights of the new policy network are updated using the loss function. The retraining module uses the pre-trained transfer reinforcement learning model to continue training on the target domain until a trained transfer reinforcement learning model is obtained that can input the assembly task on the target domain and output assembly actions.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the robot assembly method based on feature adaptive transfer reinforcement learning as described in any one of claims 1-4.

7. A processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the robot assembly method based on feature adaptive transfer reinforcement learning as described in any one of claims 1-4.

Images