A robot assembly method and system based on metric learning and meta-reinforcement learning

Abstract

The application provides a robot assembly method and system based on metric learning and meta-reinforcement learning, learns a meta-strategy from a source domain, measures distances between different samples by using metric learning, and then generalizes a universal strategy to a strategy suitable for a specific task (a new task) in a target domain, compared with the need for retraining for each new task, the method can quickly generalize the universal strategy to different new tasks, improve the learning effect, and improve the intelligent level of robot assembly.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology for robots, and particularly relates to a robot assembly method and system based on metric learning and meta-reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, when faced with more personalized and complex assembly tasks, especially when there are significant differences in material properties and shapes, the difficulty in acquiring interactive data and the low efficiency of sample training have become challenges for strategy training. The general applicability of strategies across different tasks has also constrained the further development of intelligent assembly.

[0004] Therefore, the diversity and complexity of assembly objects make it difficult to design general assembly strategies, and how to improve the generalization ability of robot assembly skills is an urgent problem to be solved. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a robot assembly method and system based on metric learning and meta-reinforcement learning. The proposed method can generalize to different new tasks more quickly based on a general policy, improve learning performance, and enhance the intelligence level of robot assembly.

[0006] To achieve the above objectives, a first aspect of the present invention provides a robot assembly method based on metric learning and meta-reinforcement learning, comprising:

[0007] Obtain multiple training sample sets of the robotic arm assembly in the source domain, each training sample set including multiple training samples of the same task;

[0008] Based on the acquired multiple training sample sets, the reinforcement learning network is trained until the network converges, thus obtaining the target reinforcement learning network. During the reinforcement learning network training process, the historical training data and the current training task data are compared based on the meta-learning network to obtain the data used for reinforcement learning network training.

[0009] Metric learning is used to calculate the distance between the new task in the target domain and the test sample in the source domain, which represents the similarity between the test sample in the source domain and the new task in the target domain.

[0010] Based on similarity calculations, test samples are determined for training the new task policy. The target reinforcement learning network is then trained and updated using reinforcement learning on the new task until the network converges, thus obtaining the new task-specific policy.

[0011] A second aspect of the present invention provides a robot assembly system based on metric learning and meta-reinforcement learning, comprising:

[0012] The acquisition module is used to acquire multiple training sample sets of the robotic arm assembly in the source domain. Each training sample set includes multiple training samples of the same task.

[0013] The first training module is used to train the reinforcement learning network based on multiple training sample sets until the network converges and the target reinforcement learning network is obtained. During the training process of the reinforcement learning network, the historical training data and the current training task data are compared based on the meta-learning network to obtain the data used for training the reinforcement learning network.

[0014] The computation module is used to calculate the distance between the new task in the target domain and the test sample in the source domain based on metric learning, so as to characterize the similarity between the test sample in the source domain and the new task in the target domain.

[0015] The second training module is used to determine test samples for training the new task strategy based on similarity calculation, and to perform reinforcement learning update training on the target reinforcement learning network on the new task until the network converges, thereby obtaining the new task assembly strategy.

[0016] A third aspect of the present invention provides a computer device comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a robot assembly method based on metric learning and meta-reinforcement learning is performed.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a robot assembly method based on metric learning and meta-reinforcement learning.

[0018] The above one or more technical solutions have the following beneficial effects:

[0019] This invention learns a meta-policy from the source domain through meta-reinforcement learning, uses metric learning to measure the distance between different samples, and then generalizes the general policy into a policy applicable to a specific task (new task) in the target domain more quickly. Compared with the need to retrain for each new task, the method proposed in this invention can generalize to different new tasks more quickly based on the general policy, improve the learning effect, and enhance the intelligence level of robot assembly.

[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is an assembly block diagram of the meta-reinforcement learning robot based on deep metric learning in Embodiment 1 of the present invention;

[0023] Figure 2 This is a diagram of the SNAIL network structure in Embodiment 1 of the present invention;

[0024] Figure 3 This is the PPO network update process in Embodiment 1 of the present invention;

[0025] Figure 4 This is the deep metric learning network in Embodiment 1 of the present invention. Detailed Implementation

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0029] Example 1

[0030] This embodiment discloses a robot assembly method based on metric learning and meta-reinforcement learning, including:

[0031] Obtain multiple training sample sets of the robotic arm assembly in the source domain, each training sample set including multiple training samples of the same task;

[0032] Based on the acquired multiple training sample sets, the reinforcement learning network is trained until the network converges, thus obtaining the target reinforcement learning network. During the reinforcement learning network training process, the historical training data and the current training task data are compared based on the meta-learning network to obtain the data used for reinforcement learning network training.

[0033] Metric learning is used to calculate the distance between the new task in the target domain and the test sample in the source domain, which represents the similarity between the test sample in the source domain and the new task in the target domain.

[0034] Based on similarity calculations, test samples are determined for training the new task policy. The target reinforcement learning network is then trained and updated using reinforcement learning on the new task until the network converges, thus obtaining the new task-specific policy.

[0035] This embodiment enables the robot to learn by means of a meta-reinforcement learning method. It learns meta-policies from the source domain and uses deep metric learning to measure the distance between different samples. Then, it can more quickly generalize the general policy into a policy applicable to a specific task in the target domain. Compared with the need to retrain for each new task, the method proposed in this embodiment can generalize to different new tasks more quickly based on the general policy, improve the learning effect, and improve the intelligence level of robot assembly.

[0036] Reinforcement learning iterates until it converges to obtain the final model. During training, many models are generated, but these are not convergent. If the initial model is randomly generated, it might take ten thousand steps to converge. Therefore, this embodiment uses the model obtained through meta-training as the initial model. This model is the result of convergence on multiple previous historical tasks. It is directly applied to a completely new task and will not converge immediately; it needs to be trained again. However, this retraining is much faster than random training on a new task, potentially converging in just a thousand steps.

[0037] This embodiment combines SNAIL and PPO networks. SNAIL analyzes historical data to filter data and guide PPO learning and training. The combination of SNAIL and PPO networks is mainly reflected in the fact that the SNAIL network provides effective cross-task data for the PPO network training. It can compare features of historically trained data with the currently trained task data to filter out effective data to assist the PPO network training. Specifically, during the PPO network update process, SNAIL generates the action distribution function in the PPO Actor network, thereby selecting the robotic arm action 'a' according to probability. t In the Critic network of PPO, the state value v is obtained through SNAIL. t This process updates the network, and iterates until the PPO network converges.

[0038] like Figure 1The diagram shown is a block diagram of the system built in this embodiment, including a robotic arm, a force sensor, and a depth sensor. It also constructs a task space for assembly objects with different material properties and shapes. Position information, force information, and assembly object attribute information are used as inputs to train the network, and the output is the robotic arm's movements. Corresponding sub-policies are trained for multiple tasks, and then the meta-policy parameters θ are updated. The meta-policy then adjusts the sub-policies until the training converges. When facing a new task, the test data and a small number of prior samples are used for metric learning. Data with higher similarity is selected to pre-train the meta-policy, thereby completing the new assembly task.

[0039] The robot assembly method proposed in this embodiment, based on deep metric learning and meta-reinforcement learning, uses domain randomization to set parameters for different tasks when constructing the source domain task space. Through temporal convolution and attention mechanisms, it learns assembly strategies for multiple different but similar tasks in the source domain task space, including search and insertion stages, to obtain meta-policies for similar tasks. Through metric learning, it obtains the implicit distance between a small number of samples in the target domain and test samples in the source domain, and adjusts the general strategy to an assembly strategy suitable for new tasks.

[0040] The following is combined with Figures 1-4 The robot assembly method based on metric learning and meta-reinforcement learning proposed in this embodiment will be described in detail below:

[0041] Initialize the robotic arm and use a depth sensor to acquire the point cloud of the hole part template as the source point cloud P. s Randomize the pose of the hole part to obtain the target point cloud P. t After filtering, it is compared with the source point cloud P. s Registration is performed to obtain the registration matrix [R,t]. The robotic arm is then controlled to adjust its pose so that the axis reaches the hole, and the small-range search and insertion stage begins.

[0042] Figure 2 The inputs and outputs in this context refer to the overall training process. This way of describing inputs and outputs is to illustrate that a reinforcement learning task is being applied here.

[0043] The input state of the PPO network during subtask training is s t =(s p ,s τ ,s ω ), where s p = [x,y,z,α,β,γ], representing the six-dimensional pose of the end effector of the robotic arm; s τ =[F x ,F y ,F z M x M y M z] represents the contact force and torque generated at the end of the robotic arm; s ω =λ1E+λ2ν, representing the material properties of the assembled parts, where λ1+λ2=1, E represents the elastic modulus, and ν represents Poisson's ratio. The output is a. t = [Δx, Δy, Δz, Δα, Δβ, Δγ], Δx, Δy, Δz, Δα, Δβ, Δγ respectively represent the offset of the robotic arm in the x, y, z, rx, ry, rz directions.

[0044] Meta-reinforcement learning employs the SNAIL algorithm, based on temporal convolution and attention mechanisms, with the network structure as follows: Figure 2 As shown.

[0045] The meta-learning network is based on two temporal convolutional layers (TC layers) and one attention mechanism layer (CA layer), and neurons within all layers are connected only to neurons representing past values. The TC layer starts from the historical state trajectory (s t ,a t-1 ,r t-1 Feature information is collected in ), where r t-1 To represent the reward, the CA layer extracts keywords (K), values ​​(V), and query indexes (Q) from the information collected by the TC layer, stores them in the memory module, and calculates the weighted information value using the following formula:

[0046]

[0047] Where, d k Indicates the dimension of K. The similarity between Q and K is represented by the similarity weights, which are normalized by the softmax function and then multiplied by the value matrix V. This helps the reinforcement learning network to filter from historical training data, quickly refer to and absorb relevant experience from historical training data, learn the meta-policy, and adjust the meta-policy parameters θ to further guide the formation of sub-policies during the reinforcement learning training process of the task. This process is repeated until the meta-reinforcement learning network converges and learns the optimal meta-policy.

[0048] In the SNAIL network, the combination of temporal convolution and attention mechanisms allows the meta-learner to gather contextual information from historical experience and retain causal relationships, thereby enabling data filtering. Furthermore, the training process of meta-reinforcement learning involves multiple sub-tasks, making the data information relatively more complex. Through temporal convolution and attention layers, specific information can be located, and corresponding data that is more conducive to training can be extracted, playing an auxiliary role in the training of each sub-task.

[0049] The PPO algorithm is used to train each task in the meta-training task space. The same policy is used to learn the search and insertion processes. Therefore, an adaptive reward function is set, and a reward function is designed specifically for the search and insertion processes. The weights are changed according to the relative position of the shaft and hole and the force sensor information. The network update process is as follows:

[0050] Initialize Actor old Actor new After the Critic network, the robotic arm state s t As input, the Actor network obtains feature logits through SNAIL, and uses a Categorical function to obtain the action probability density distribution, sampling from it according to the probability to obtain the robotic arm action 'a'. t The Critic network obtains its state value v through SNAIL. t And calculate the discount reward R t =r t +κr t+1 +κ 2 r t+2 +κ 3 r t+3 +…, where κ represents the discount parameter for reward decay, yielding the advantage function A. t =R t -v t In practical applications, the estimated value is used. Instead, calculate the loss using the mean squared error function (MSE). Backpropagation updates the Critic network; the ratio of the probability density of the old and new Actor networks is r. t (ω), where ω represents the policy parameters of the PPO algorithm, the loss is calculated according to the following formula, and the Actor is updated. new network:

[0051]

[0052] Here, ε is a hyperparameter that works in conjunction with the clip function to control the range of policy updates.

[0053] The reward function is set as follows:

[0054]

[0055] Where, r search The characteristic reward function r represents the search phase. insert The characteristic reward function for the insertion phase is represented by ι, which is a function of the shaft-hole center distance deviation Δ and the force F. The weights are adjusted according to the changes in the state during the assembly process. The search phase r search Greater importance, insertion phase r insertIt is of greater importance. The reward functions for the two phases are set as follows:

[0056]

[0057] in, These represent the two weights of the search reward, which sum to 1, k se , These represent the current search step count and the set maximum search step count, respectively. x ,d y ) represents the average distance between the current axis's x and y coordinates and the hole center's x and y coordinates. These represent the three weights of the insertion reward, which sum to 1, k in , Δ represents the current insertion step and the set maximum insertion step, respectively. z , These represent the insertion depth at the current step and the set maximum insertion depth, respectively, in μ. i The weights of the torque ratios corresponding to the x, y, and z directions are Γ(i), Γ(i), and Γ(i) are summed to 1. max (i) represents the force / torque in the current corresponding direction and the maximum force / torque allowed in that direction, respectively.

[0058] Construct an implicit feature distance calculation network based on deep metric learning, such as... Figure 4 As shown, the network is used for the data selection stage of new task generalization. Its input consists of two parts: a large amount of test data obtained after testing the meta-policy on historical tasks, and a small amount of data obtained by training on new tasks. The neural network sorts the similarity between the test data and the small amount of sample data and outputs the test data in descending order of similarity.

[0059] By fully utilizing the supervised information in the test data as constraints, an objective function is constructed to learn the distance metric between the test data and a small number of prior samples in the new task. The optimized objective function is as follows:

[0060]

[0061] in, Indicates class inner spacing, The interval between classes is represented by N, where N represents the number of samples and P represents the distance between classes. ij Represents the intra-class distance, when sample x i It is x j When the class has k1 nearest neighbors, P ij =1, otherwise 0, d 2 (x i ,x j ) represents sample x i With x jThe square of the Mahalanobis distance between them; similarly, Q ij Represents the inter-class distance, when sample x i It is x j When the k2 nearest neighbors of the class are inter-class, Q ij =1, otherwise 0. D(X) s ,X t ) represents the explicit feature distance, X s Let X represent the set of sample points in the source domain. t This represents the set of sample points in the target domain. Deep learning sorts the test data by similarity, and selects the n most similar sets as training data for the new task policy. The purpose is to provide a large amount of data for the reinforcement learning training network of the new task, helping it to train and obtain the assembly policy. The initial policy for the new task training is the meta-policy obtained through meta-learning. The training network is a PPO network. The initial policy, i.e., the initial model, is updated and trained through the PPO network, which allows the network to converge more quickly and obtain the new task assembly policy.

[0062] The constructed optimization objective function is used as part of the objective function of the neural network, which is used to train an implicit feature distance model. During the training of this model, the explicit distance can be used as part of the objective function, but the explicit feature distance does not play a decisive role. This is better than using only the explicit feature distance. Moreover, the data dimension in robotic arm operation tasks is relatively high, and neural networks are inherently more advantageous in processing high-dimensional data. Therefore, deep metric learning is used to process the sample data, rather than directly calculating it using only the explicit feature distance.

[0063] This embodiment combines meta-reinforcement learning, considering the material and shape attributes of the assembly objects, and proposes a scheme to enhance the robot's assembly generalization ability. It accelerates the training process by training the overall strategy in the search and insertion stages through an adaptive reward function; and uses deep metric learning to rank the data for similarity and update the meta-strategy, enabling the robotic arm to adapt to new assembly tasks more quickly.

[0064] Example 2

[0065] The purpose of this embodiment is to provide a robot assembly system based on metric learning and meta-reinforcement learning, including:

[0066] The acquisition module is used to acquire multiple training sample sets of the robotic arm assembly in the source domain. Each training sample set includes multiple training samples of the same task.

[0067] The first training module is used to train the meta-learning network based on multiple acquired training sample sets, so that the meta-learning network learns meta-policies for similar tasks.

[0068] The computation module is used to calculate the distance between the new task in the target domain and the test sample in the source domain based on metric learning, so as to characterize the similarity between the test sample in the source domain and the new task in the target domain.

[0069] The second training module is used to determine test samples for training the new task strategy based on similarity calculation, and to train the meta-learning network with the meta-policy as the initial policy until the network converges to obtain the new task assembly policy.

[0070] Example 3

[0071] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0072] Example 4

[0073] The purpose of this embodiment is to provide a computer-readable storage medium.

[0074] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0075] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0076] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0077] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robot assembly method based on metric learning and meta-reinforcement learning, characterized in that, include: Obtain multiple training sample sets of the robotic arm assembly in the source domain, each training sample set including multiple training samples of the same task; Based on the acquired training sample sets, the reinforcement learning network is trained until the network converges, resulting in the target reinforcement learning network. During the reinforcement learning network training process, a meta-learning network is used to compare historically trained data with the current training task data to obtain training data for the reinforcement learning network. Each task in the task space is trained using a PPO network. In the PPO's Actor network, the meta-learning network generates an action distribution function, selecting robotic arm actions according to probability. In the PPO's Critic network, the meta-learning network obtains state values, updating the PPO network until it converges. The meta-learning network includes a temporal convolutional layer and an attention mechanism layer. The temporal convolutional layer is used to extract feature data from the historical trajectory of the robotic arm, and the attention mechanism layer is used to calculate a weighted information value based on the extracted feature data. The distance between the new task in the target domain and the test sample in the source domain is calculated based on metric learning to characterize the similarity between the test sample in the source domain and the new task in the target domain. Specifically, an optimization objective function is constructed using the distance metric between training samples; the constructed optimization objective function is used as the objective function of the neural network to train the neural network and obtain an implicit feature distance model; and test data for reinforcement learning training of the new task is obtained based on the implicit feature distance model. Based on similarity calculations, test samples are determined for training the new task policy. The target reinforcement learning network is then trained and updated using reinforcement learning on the new task until the network converges, thus obtaining the new task-specific policy.

2. The robot assembly method based on metric learning and meta-reinforcement learning as described in claim 1, characterized in that, The PPO network is trained based on the acquired training sample sets, specifically as follows: Initialize the action network and evaluation network; The robotic arm state data is used as input to the motion network and evaluation network, respectively, to obtain the mean and variance of the motion probability density function, as well as the state value. Calculate the discount reward, and obtain the advantage function based on the calculated discount reward and state value; The first loss is calculated using the mean squared error function based on the calculated advantage function, and the evaluation network is updated in reverse based on the calculated loss. The second loss is calculated based on the mean and variance of the action probability density function, and the action network is updated based on the calculated second loss.

3. A robot assembly method based on metric learning and meta-reinforcement learning as described in any one of claims 2, characterized in that, The action network includes a new action network and an old action network. The second loss is calculated based on the ratio of the action probability density output by the new action network and the old action network, respectively.

4. A robot assembly method based on metric learning and meta-reinforcement learning as described in any one of claims 1, characterized in that, The reward function includes a characteristic reward function for the search phase and a characteristic reward function for the insertion phase. The weights of the characteristic reward functions for the search phase and the insertion phase are adjusted according to the changes in the state during the assembly process.

5. A robot assembly system based on metric learning and meta-reinforcement learning, characterized in that, include: The acquisition module is used to acquire multiple training sample sets of the robotic arm assembly in the source domain. Each training sample set includes multiple training samples of the same task. The first training module is used to train the reinforcement learning network based on multiple acquired training sample sets until the network converges, thus obtaining the target reinforcement learning network. During the reinforcement learning network training process, a meta-learning network is used to compare historically trained data with the current training task data to obtain training data for the reinforcement learning network. Each task in the task space is trained using a PPO network. In the PPO's Actor network, the meta-learning network generates an action distribution function, selecting robotic arm actions according to probability. In the PPO's Critic network, the meta-learning network obtains state values, updating the PPO network until it converges. The meta-learning network includes a temporal convolutional layer and an attention mechanism layer. The temporal convolutional layer is used to extract feature data from the historical trajectory of the robotic arm, and the attention mechanism layer is used to calculate a weighted information value based on the extracted feature data. The computation module is used to calculate the distance between the new task in the target domain and the test sample in the source domain based on metric learning, so as to characterize the similarity between the test sample in the source domain and the new task in the target domain; specifically, it constructs an optimization objective function using the distance metric between training samples; it trains the neural network based on the constructed optimization objective function as the objective function of the neural network to obtain an implicit feature distance model; and it obtains test data for reinforcement learning training of the new task based on the implicit feature distance model. The second training module is used to determine test samples for training the new task strategy based on similarity calculation, and to perform reinforcement learning update training on the target reinforcement learning network on the new task until the network converges, thereby obtaining the new task assembly strategy.

6. A computer device, characterized in that, include: The system includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform a robot assembly method based on metric learning and meta-reinforcement learning as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor as described in any one of claims 1 to 4, a robot assembly method based on metric learning and meta-reinforcement learning.

Images