A method for training a robotic arm to complete cloth folding tasks using curriculum learning

Through course learning and reinforcement learning algorithm optimization, the problems of slow and low success rate of robotic arm cloth folding training in virtual environments have been solved, and efficient and stable cloth folding training has been achieved.

CN116061190BActive Publication Date: 2024-11-15ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310243570.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-14
Publication Date
2024-11-15
Estimated Expiration
2043-03-14

AI Technical Summary

Technical Problem

Existing technologies for training robotic arms to fold fabric in virtual environments suffer from slow training and low task success rates due to unreasonable physical parameter settings.

Method used

The course-based learning approach is adopted, a virtual environment is built using physical simulation technology, the HER algorithm and SAC algorithm are used to optimize the folding strategy of the robotic arm, the virtual environment parameters and reward mechanism are gradually adjusted, and the cloth folding task is trained by combining the spring-mass model.

Benefits of technology

It accelerates the stability and success rate of robotic arm cloth folding training, avoids the gradient vanishing problem in low-friction environments, and improves training efficiency and effectiveness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116061190B_ABST
    Figure CN116061190B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for training a robot arm to complete a cloth folding task by using curriculum learning, and belongs to the field of artificial intelligence. Traditional reinforcement learning algorithms for training robots to complete tasks rarely carefully consider the impact of physical environment parameters on task difficulty, and most of them use domain randomization methods to overcome the problem of transferring from virtual environment to real environment. When faced with cloth folding tasks, environmental parameters will significantly affect the decision-making of the robot arm, thereby affecting the training process of the robot arm. The present invention proposes a curriculum learning method, which arranges cloth folding tasks for the robot from easy to difficult, thereby accelerating training and improving training stability. Specifically, the present invention will randomize the friction between the cloth and the contact table during the training process, provide some tasks with low difficulty, and use the dynamic friction factor as the observation quantity of the robot arm to accelerate the training process. Under the training arrangement of the present invention, compared with ordinary training methods, the present invention can significantly shorten the training time and improve the training stability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence, and in particular relates to a method for using course learning to train a robotic arm to complete a cloth folding task. Background Art

[0002] In the field of intelligent robots, it is a very common demand for robots to assist humans in completing various daily household tasks, among which folding clothes is a widely studied task. Now, based on the development of physical simulation and reinforcement learning technology, robot training can be transferred to a virtual environment, thereby accelerating training efficiency and reducing costs. For tasks such as folding cloth that involve the operation of deformable objects, the setting of physical parameters in the simulation is very sensitive. On the one hand, it needs to correspond to the physical performance of the real world. On the other hand, the physical parameters are related to the success rate of task completion, which affects the training process. Unreasonable physical parameter settings will cause problems such as slow training and low task success rate. Summary of the invention

[0003] In view of the above problems, the present invention provides a method for training a robotic arm to complete cloth folding tasks by using curriculum learning. The method adopts the curriculum learning method to arrange cloth folding tasks for the robot from easy to difficult, thereby accelerating training and improving training stability.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A method for training a robotic arm to complete a cloth folding task using curriculum learning, comprising:

[0006] S1, build a virtual environment using physical simulation technology to ensure that the robotic arm can interact with the virtual environment to fold fabric;

[0007] S2, let the robot arm interact in the virtual environment for multiple task cycles. At the beginning of each cycle, the parameter sampler randomly adjusts the virtual environment parameters. After each step of interaction, the interaction data and reward value of the robot arm in the virtual environment are collected as experience data;

[0008] S3, after completing a number of task cycles, the HER algorithm is used to enhance the collected experience data;

[0009] S4, based on the original experience data and enhanced experience data, the SAC algorithm is used to optimize the robot arm strategy;

[0010] S5, repeat steps S2-S4 until the SAC algorithm converges and obtains the cloth folding strategy of the robot arm.

[0011] Furthermore, the step S1 is specifically as follows:

[0012] S11, representing the cloth as a spring mass point model, first discretizing the cloth into mass points in three dimensions, and the mass points are connected by springs;

[0013] S12, by numerically solving ordinary differential equations Perform physical simulation, where x is the mass point position, t is the time, k is the spring constant, F ext For external forces;

[0014] S13, for the robot arm simulation, it is assumed that the robot arm has grasped a specific end point of the cloth before the task starts, and the movement of the robot arm is equivalent to applying an external force to the point to displace it.

[0015] Furthermore, the step S2 is specifically as follows:

[0016] S21, for the robot arm folding cloth task, the observation space is defined as That is, n feature points are selected on the cloth, and their three-dimensional Euclidean coordinates are used as the observation quantity of the robot arm; the action space is That is, at each time step, the robot performs a small displacement in the operating space; the target space is That is, the state that the user wants the m feature points on the fabric to achieve, and the state of the m feature points on the current fabric;

[0017] S22, based on the definition of observation space, action space, and target space in the task of folding cloth by the robot arm, for each time step, the robot arm will receive the following reward:

[0018]

[0019] Among them, g d is the posture that the user wants the fabric to achieve, g o is the current cloth posture, δ is the error limit, ‖·‖ is the modulo operation, and r(·) is the reward value;

[0020] S23, when the number of interaction steps exceeds the limit or the task is completed, the states of the cloth and the robot arm are reset to the initial state;

[0021] S24, the environmental parameter sampler samples new g according to the course strategy d and the new kinetic friction factor μ;

[0022] S25, for each task cycle, the virtual environment and the robot are initialized. Then the robot performs action a according to the strategy π based on the current observation state s and obtains the corresponding reward r. After completing an interaction, the six-tuple (s t ,s t+1 ,a,r,g o ,g d) into the replay cache for use in the training step for several task cycles.

[0023] Furthermore, in step S23, if || g d -g o If ||≤δ exceeds 200 consecutive time steps, the task is considered completed.

[0024] Furthermore, the environment sampler samples the dynamic friction factor μ according to the Gaussian distribution, and adjusts the mean of μ according to the mean reward obtained by the current strategy. When the mean reward increases, the mean of μ will be gradually reduced.

[0025] Furthermore, the step S3 is specifically as follows: for the experience tuple (s t i ,s t +1 i ,a i ,r i ,g i o ,g d ), the HER algorithm is used to randomly select the achieved target g in a certain step of the task cycle j o ,j≥i, and according to g j o ,g d Calculate the new reward function r′ and generate new experience data (s t i ,s t+1 i ,a i ,r′,g j o ,g d ) is added to the playback cache.

[0026] Furthermore, the step S4 is specifically as follows:

[0027] S41, evaluates the Q value of executing a certain action in the current state through the review neural network in the SAC algorithm,

[0028]

[0029] in, is the Q value of executing action a according to strategy π based on the current observation state s, For expectation, t ,a t is the state and action of step t, ρ π is the distribution of states and actions under policy execution interaction, γ tis the t-th power of the discount factor, r(·) is the reward function, H(·) is the information enthalpy, s0, a0 are the initial state and action, and α is the temperature;

[0030] Optimize the loss function J Q for:

[0031]

[0032] Among them, θ is the parameter of the review neural network, To replay the cache, is the Q value given by the review neural network, π φ (·) is the probability distribution given by the actor neural network;

[0033] S42, observe the current state through the actor neural network and select the one that can maximize The optimization loss function J of the actor neural network is π for:

[0034]

[0035] Where φ represents the parameters of the actor neural network, represents the value function given by the review neural network, D KL represents the Kullback-Leibler divergence;

[0036] S43, using the gradient descent algorithm to optimize the actor and review neural network, for each optimization step, the optimization algorithm randomly samples a set of experience (s t i ,s t+1 i ,a i ,r i ), and use gradient descent to optimize the actor and review neural network:

[0037]

[0038] Among them, φ T ,θ T represents the parameters of the actor and review neural network at the Tth iteration, λ φ , θ represents the learning rate of the actor and judge neural network, It means to find the gradient.

[0039] Because the present invention adopts the strategy of providing tasks from easy to difficult in curriculum learning, compared with the existing technology, it avoids the problem of gradient disappearance caused by inability to obtain positive feedback in a low-friction environment, and improves the training success rate and training stability of high-difficulty problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the environmental parameter course learning method in one embodiment of the present invention.

[0041] Figure 2 The flowchart is a method for training a robotic arm to complete a cloth folding task by using curriculum learning in one embodiment of the present invention.

[0042] Figure 3 A schematic diagram of an experiment conducted for a strategy obtained in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0044] like Figure 2 As shown, in a preferred embodiment of the present invention, the method of using curriculum learning to train a robot arm to complete a cloth folding task mainly includes the following steps:

[0045] S1, build a virtual environment using physical simulation technology to ensure that the robot arm can interact with the virtual environment to fold the cloth, such as Figure 1 As shown, the environment parameters are provided according to the course, so that the robot arm agent and the environment can be interactively trained;

[0046] S2, let the robot arm interact in the virtual environment for multiple task cycles. At the beginning of each cycle, the parameter sampler randomly adjusts the virtual environment parameters. The virtual environment parameters in this embodiment are the dynamic friction coefficient μ. After each step of interaction, the interaction data and reward value of the robot arm in the virtual environment are collected as experience data.

[0047] S3, after completing a number of task cycles, the HER algorithm is used to enhance the collected experience data;

[0048] S4, based on the original experience data and enhanced experience data, the SAC algorithm is used to optimize the robot arm strategy;

[0049] S5, repeat steps S2-S4 until the SAC algorithm converges and obtains the cloth folding strategy of the robot arm.

[0050] In a specific implementation of the present invention, the step S1 is specifically:

[0051] S11. Represent the fabric as a spring - mass model, discretize the fabric into mass points in three - dimensions, and connect the mass points with springs.

[0052] S12. Conduct physical simulation by numerically solving ordinary differential equations where \(x\) is the position of the mass point, \(t\) is the time, \(k\) is the spring stiffness coefficient, and \(F\) ext is the external force.

[0053] S13. For the robotic arm simulation, assume that before the task starts, the robotic arm has grasped a specific end point of the fabric. The movement of the robotic arm is equivalent to applying an external force to this point for displacement. Therefore, a certain external force can be added to this point.

[0054] In a specific implementation of the present invention, step S2 is specifically as follows:

[0055] S21. For the task of the robotic arm folding the fabric, in this embodiment, the observation space is defined as that is, \(n\) feature points are selected on the fabric, and their three - dimensional Euclidean coordinates are used as the observations of the robotic arm agent. The action space is that is, at each time step, the robotic arm makes a small displacement in the operation space. In addition, for each task cycle, the method can also be specified by the user the task objective to be completed. The target space is that is, the states that the user hopes the \(m\) feature points on the fabric will achieve, and the states of the \(m\) feature points on the current fabric. Usually, \(m < n\) is selected so that this task does not become a partially observable Markov decision process;

[0056] S22. With the definitions of the observation space, action space, etc. for this task in S21, for the robotic arm agent, this embodiment gives the following reward function to guide. At each time step, the robotic arm agent will receive the following rewards:

[0057]

[0058] where \(g\) d is the posture that the user hopes the fabric will achieve, \(g\) o is the current fabric posture, \(\delta\) is the error bound, \(\|\cdot\|\) is the modulus operation, and \(r(\cdot)\) is the reward value;

[0059] S23. In this embodiment, considering the diversity of experience collection and the large amount of useless state space that may be brought about by too long a cycle, when the number of interaction steps exceeds a certain limit or the task is completed, the entire scene will be reset, and the states of the fabric and the robotic arm will be reset to the initial state. In this embodiment, if \(\|g\) d - g o \|\leq\delta\) continuously exceeds 200 time steps, it is considered that the task is completed;

[0060] S24, to accelerate the training rate, the environment parameter sampler will sample new g according to the course strategy d And the new dynamic friction factor μ. Specifically, the environment sampler will sample the dynamic friction factor μ according to the Gaussian distribution, and adjust the mean of μ according to the mean reward obtained by the current strategy. When the mean reward increases, the mean of μ will be gradually reduced;

[0061] S25, further, for experience generation, the following steps need to be performed: for each cycle, the environment and the robot agent are initialized, and then the robot agent performs action a according to the strategy π based on the current observation state s and obtains the corresponding reward r. After completing an interaction, the sextuple (s t ,s t+1 ,a,r,g o ,g d ) into the playback buffer for use in the training step; this experience collection step will last for several cycles;

[0062] In a specific implementation of the present invention, the step S3 is specifically to replay the training experience stored in the cache. Due to the sparse nature of the reward function, the method first uses the HER technology to strengthen the experience data. The specific method is as follows: for the experience data in a cycle, for the experience tuple (s t i ,s t+1 i ,a i ,r i ,g i o ,g d ), the HER algorithm will randomly select the target g that has been reached in a certain step in the cycle j o ,j≥i, and according to g j o ,g d Calculate the new reward function r′ to generate new experience (s t i ,s t+1 i ,a i ,r′,g j o ,g d ) is added to the playback cache;

[0063] In a specific implementation of the present invention, in step S4, the SAC algorithm used is a deep reinforcement learning algorithm. The SAC algorithm involves two types of neural networks: actor and reviewer neural networks. The method obtains the best strategy by optimizing these two neural networks.

[0064] S41, the review is to use the neural network to evaluate the Q value of performing a certain action in the current state. The traditional Q value is:

[0065]

[0066] In order to maximize the reward value while ensuring exploration diversity, the SAC algorithm adds an entropy term to the Q value to adjust the diversity of the strategy:

[0067]

[0068] in, is the Q value of executing action a according to strategy π based on the current observation state s, For expectation, t ,a t is the state and action of step t, ρ π is the distribution of states and actions under policy execution interaction, γ t is the t-th power of the discount factor, r(·) is the reward function, H(·) is the information enthalpy, s0, a0 are the initial state and action, and α is the temperature;

[0069] This embodiment uses the temperature difference method to optimize the review so that the neural network can accurately predict That is, the loss function is set as:

[0070]

[0071] Among them, θ is the parameter of the review neural network, To replay the cache, is the Q value given by the review neural network, π φ (·) is the probability distribution given by the actor neural network;

[0072] S42, the purpose of the actor neural network is to observe the current state and select the one that can maximize action, for which the loss function is set as:

[0073]

[0074] Where φ represents the parameters of the actor neural network, represents the value function given by the review neural network, D KL represents the Kullback-Leibler divergence;

[0075] S43, for actors and judges, the SAC algorithm uses a gradient descent algorithm for optimization. For each optimization step, the optimization algorithm randomly samples a set of experience (s t i ,st+1 i ,a i ,r i ), and use gradient descent to optimize actor review:

[0076]

[0077] Among them, φ T ,θ T represents the parameters of the actor and review neural network at the Tth iteration, λ φ , θ represents the learning rate of the actor and judge neural network, It means to find the gradient.

[0078] After completing a certain number of optimization steps, the experience collection step in S2 will continue to be executed, and this cycle will continue until the SAC algorithm converges. The actor neural network in the converged SAC algorithm will give the robot arm execution strategy.

[0079] Figure 3 is a schematic diagram of strategy implementation obtained using the method of the present invention, Figure 3 From left to right in the figure are the initial state, mid-term, and final results of the test platform. The strategy training time is 1000000 steps, the learning rate is 0.0001, and it takes about 3 days to train on a desktop computer with a GPU of 1080ti. The goal is to fold a piece of cloth in half. The strategy trained by the present invention is run on a notebook with a CPU of AMD Ryzen 5-3500U CPU and 8GB memory, and a UR5 robot arm is controlled through a network connection to complete the test.

[0080] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A method for training a robotic arm to complete a cloth folding task using curriculum learning, characterized in that: include: S1, build a virtual environment using physical simulation technology to ensure that the robotic arm can interact with the virtual environment to fold fabric; The step S1 is specifically as follows: S11, representing the cloth as a spring mass point model, first discretizing the cloth into mass points in three dimensions, and the mass points are connected by springs; S12, by numerically solving ordinary differential equations Perform physical simulation, where x is the mass point position, t is the time, k is the spring constant, F ext For external forces; S13, for the robot arm simulation, it is assumed that the robot arm has grasped a specific end point of the cloth before the task starts, and the movement of the robot arm is equivalent to applying an external force to the point to displace it; S2, let the robot arm interact in the virtual environment for multiple task cycles. At the beginning of each cycle, the parameter sampler randomly adjusts the virtual environment parameters. After each step of interaction, the interaction data and reward value of the robot arm in the virtual environment are collected as experience data; The step S2 is specifically as follows: S21, for the robot arm folding cloth task, the observation space is defined as That is, n feature points are selected on the cloth, and their three-dimensional Euclidean coordinates are used as the observation quantity of the robot arm; the action space is That is, at each time step, the robot performs a small displacement in the operating space; the target space is That is, the state that the user wants the m feature points on the fabric to achieve, and the state of the m feature points on the current fabric; S22, based on the definition of observation space, action space, and target space in the task of folding cloth by the robot arm, for each time step, the robot arm will receive the following reward: Among them, g d is the posture that the user wants the fabric to achieve, g o is the current cloth posture, δ is the error limit, ‖·‖ is the modulo operation, and r(·) is the reward value; S23, when the number of interaction steps exceeds the limit or the task is completed, the states of the cloth and the robot arm are reset to the initial state; S24, the environmental parameter sampler samples new g according to the course strategy d and the new kinetic friction factor μ; S25, for each task cycle, the virtual environment and the robot are initialized. Then the robot performs action a according to the strategy π based on the current observation state s and obtains the corresponding reward r. After completing an interaction, the six-tuple (s t ,s t+1 ,a,r,g o ,g d ) into the replay buffer for use in the training step for several task cycles; S3, after completing a number of task cycles, the HER algorithm is used to enhance the collected experience data; The step S3 is specifically as follows: For the experience tuple (s t i ,s t+1 i ,a i ,r i ,g i o ,g d ), the HER algorithm is used to randomly select the achieved target g in a certain step of the task cycle j o ,j≥i, and according to g j o ,g d Calculate the new reward function r′ and generate new experience data (s t i ,s t+1 i ,a i ,r′,g j o ,g d ) is added to the playback cache; S4, based on the original experience data and enhanced experience data, the SAC algorithm is used to optimize the robot arm strategy; The step S4 is specifically as follows: S41, evaluates the Q value of executing a certain action in the current state through the review neural network in the SAC algorithm, in, is the Q value of executing action a according to strategy π based on the current observation state s, For expectation, t ,a t is the state and action of step t, ρ π is the distribution of states and actions under policy execution interaction, γ t is the t-th power of the discount factor, r(·) is the reward function, H(·) is the information enthalpy, s0, a0 are the initial state and action, and α is the temperature; Optimize the loss function J Q for: Among them, θ is the parameter of the review neural network, To replay the cache, is the Q value given by the review neural network, π φ (·) is the probability distribution given by the actor neural network; S42, observe the current state through the actor neural network and select the one that can maximize The optimization loss function J of the actor neural network is π for: Among them, φ represents the parameters of the actor neural network, represents the value function given by the review neural network, D KL represents the Kullback-Leibler divergence; S43, using the gradient descent algorithm to optimize the actor and review neural network, for each optimization step, the optimization algorithm randomly samples a set of experience (s t i ,s t+1 i ,a i ,r i ), and use gradient descent to optimize the actor and review neural network: Among them, φ T ,θ T represents the parameters of the actor and review neural network at the Tth iteration, λ φ , θ represents the learning rate of the actor and review neural network, It means to find the gradient; S5, repeat steps S2-S4 until the SAC algorithm converges and obtains the cloth folding strategy of the robot arm.

2. The method of using course learning to train a robot arm to complete a cloth folding task according to claim 1, characterized in that: In step S23, if ||g d -g o If ||≤δ exceeds 200 consecutive time steps, the task is considered completed.

3. The method of using course learning to train a robot arm to complete a cloth folding task according to claim 1, characterized in that: The environment sampler samples the dynamic friction factor μ according to the Gaussian distribution, and adjusts the mean of μ according to the mean reward obtained by the current strategy. When the mean reward increases, the mean of μ will be gradually reduced.