A robot autonomous assembly method based on residual reinforcement learning

Abstract

The present invention relates to a robot autonomous assembly method based on residual reinforcement learning, the steps comprising: according to the state s of the robot assembly process t Define Cartesian space control instructions; set k1 = 0 when the robot end approaches the assembly surface; when F ext >F max When the robot makes the first contact with the assembly surface, it determines whether the object to be assembled has entered the assembly hole. When entering the insertion phase, k2 is set to 0, and the robot performs the insertion operation. When the external force F is detected, ext >F d / 2, the assembly is successful. The present invention designs corresponding assembly strategies for the three stages of approach, search, and insertion in the assembly process, and uses a residual reinforcement learning method based on the fusion of vision and force perception and a force control strategy based on an analytical method to achieve end-to-end robot self-learning assembly.

Description

Technical Field

[0001] The invention relates to a robot autonomous assembly method based on residual reinforcement learning, and belongs to the technical field of intelligent control. Background Art

[0002] Autonomous robot assembly is an important part of intelligent manufacturing and has always been a hot topic of research. Most previous methods rely on prior knowledge such as geometric parameters and posture information of assembly parts, which is difficult to estimate in unstructured environments. Combining the complementary information of vision and touch, humans can effortlessly complete almost all operational tasks that interact with the environment. However, in unstructured and changing environments, refined robot operations such as assembly still face huge challenges. The main reason is that it is difficult for robots to use traditional visual methods to accurately locate the objects to be assembled and that the force factor needs to be considered when the robot contacts the environment.

[0003] Reinforcement learning is considered to be the most likely to give robots human-like capabilities. Robots can learn complex skills from multiple sensor observations through interaction with the environment. At present, most work on assembly tasks only considers single force information or visual information, and all of them are in a structured environment with prior knowledge such as the geometric information of the objects to be assembled. There is also some work that attempts to integrate vision and force perception as well as robot body information to learn to form a multimodal prior space, and then use reinforcement learning to learn assembly skills. This method requires a large number of manually designed robot paths, which is labor-intensive and cumbersome.

[0004] In addition, most gradient-based reinforcement learning algorithms have low sample efficiency, large data requirements, and long training times. Therefore, most reinforcement learning-based work is verified in simulation and has not been deployed to actual robot hardware. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, propose a robot autonomous assembly method based on residual reinforcement learning, design corresponding assembly strategies for the three stages of approach, search and insertion in the assembly process, and use a residual reinforcement learning method based on the fusion of vision and force perception and a force control strategy based on an analytical method to achieve end-to-end robot self-learning assembly.

[0006] The solution of the present invention is:

[0007] A robot autonomous assembly method based on residual reinforcement learning, the steps are as follows:

[0008] (1) According to the state s of the robot during assembly t Define Cartesian space control instructions;

[0009] (2) When the robot end approaches the assembly surface: set k1 = 0, and the robot's Cartesian motion increment is determined by the visual strategy π θ (s t )Decide;

[0010] (3) When F ext >F max When the robot makes its first contact with the assembly surface, the robot switches from approaching to searching; through the deep reinforcement learning strategy π θ and analytical force control strategy π H Work together to maximize the custom reward function r t ,Strategy π θ As in the approach phase, the robot is guided to move according to the image to maximize the reward function, and the strategy π H Responsible for the smooth interaction between the robot end and the environment, including the deep reinforcement learning strategy π θ The output is u RL , analyzing the force control strategy π H The output is u H ;

[0011] (4) Determine whether to switch to the insertion stage, that is, determine whether the object to be assembled enters the assembly hole: when the condition |z-z0|<h and the external force F ext When both ≤∈ are satisfied, the search phase enters the insertion phase;

[0012] Among them, z represents the coordinate of the robot end on the z-axis, z0 represents the coordinate of the bottom of the hole in the z-axis direction, h represents the height of the hole, and ∈ is a small quantity;

[0013] (5) Entering the insertion phase, set k2 = 0, the robot performs the insertion operation, and when the external force F is detected ext >F d / 2, the assembly is successful.

[0014] Furthermore, the deep reinforcement learning strategy π θ The online training adopts PPO as the training algorithm. Gaussian noise is added to the output command and training parameters during training. The noise can be expressed as N(μ,σ 2 ), μ is the mean, σ is the variance, and during training, the reward function r is set according to the characteristics of the assembly task. t , which can be expressed as

[0015]

[0016] Among them, R, S, I, C and F represent approach, search, insertion, completion and failure respectively. t 、vt represents the displacement and rotation increments, the subscripts R and S represent the distance between the assembly axis and the hole at the initial moment of each stage, respectively, and k d and k S is the scale factor.

[0017] Furthermore, the state set s of the robot and the environment t It includes the depth image of the hand-eye camera and the 6-axis force / torque sensor installed at the end of the robot. The shaft and hole parts to be assembled by the robot are included in the depth image of the hand-eye camera.

[0018] Furthermore, during the search phase, the robot end applies a constant force F to the normal direction of the assembly hole surface. d , the control rate of constant force is

[0019] F u =F d +k p ·F e +k i ∫F e dt,

[0020] where k p ,k i are the proportional and integral parameters respectively.

[0021] Furthermore, F e =F d -F ext is the error between the expected value and the actual value of the force. Dynamic admittance control is used in other directions.

[0022] Furthermore, the control rate is expressed as

[0023]

[0024] Furthermore, the control rate is discretized for program control

[0025]

[0026] Where M, B, K are the desired mass, damping and stiffness of the robot in motion, and Δt is the control period.

[0027] Furthermore, is the path tracking impedance error.

[0028] Furthermore, in step (1), the Cartesian space control instruction x u =k1π H (s t )+k2π θ (s t ), where π θ is the deep reinforcement learning strategy, πH To analyze the force control strategy, k1,k2∈[0,1] are the proportional coefficients.

[0029] Furthermore, in step (2), the visual strategy consists of two parts, namely, a feature extraction convolutional neural network and a reinforcement learning decision network.

[0030] The beneficial effects of the present invention compared with the prior art are:

[0031] (1) The present invention designs corresponding assembly strategies for the three stages of approach, search, and insertion in the assembly process, respectively, and uses a residual reinforcement learning method based on the fusion of vision and force perception and a force control strategy based on an analytical method to achieve end-to-end robot self-learning assembly;

[0032] (2) The method of the present invention does not require prior knowledge of the geometry of the assembly object, nor does it require manual collection of training data sets. In addition, the proposed assembly method has high sample efficiency, short training time, and strong generalization ability for unknown objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a residual reinforcement learning framework diagram of the present invention;

[0034] Figure 2 A flowchart of the end-to-end training process combining vision and force perception;

[0035] Figure 3 To control the algorithm process. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the embodiments.

[0037] 1. Residual reinforcement learning framework based on vision and analytical force control method

[0038] First, obtain the state set s of the robot and the environment t , including the depth image of the hand-eye camera and the 6-axis force / torque sensor installed at the end of the robot. The field of view of the hand-eye camera includes the shaft hole parts to be assembled by the robot. The residual reinforcement learning strategy consists of three parts: the reward function customized by the user according to the characteristics of the task, the deep reinforcement learning strategy π θ and analytical force control strategy π H . Deep reinforcement learning policy π θ and analytical force control strategy π H work together to maximize the defined reward r t , which can be expressed as

[0039] x u =k1π H (s t )+k2πθ (s t )

[0040] x u is the Cartesian space control instruction of the robot, k1,k2∈[0,1] is the proportional coefficient. The flow chart of residual reinforcement learning is as follows Figure 1 shown.

[0041] 2. Phased assembly strategy based on state recognition

[0042] like Figure 2 As shown, the staged assembly strategy according to the state recognition of the present invention divides the assembly process into three stages, which are respectively represented as approach, search and insertion. The steps are as follows:

[0043] (1) Approach stage

[0044] In the approach phase, since the robot end is not in contact with the environment, k1 is set to 0, and the Cartesian motion increment of the robot is determined by the visual strategy π θ (s t ), the visual strategy consists of two parts, namely the feature extraction convolutional neural network and the reinforcement learning decision network. Among them, the feature extraction convolutional neural network is composed of 5 convolutional layers, the input image size is cropped to 3×240×320, from front to back, the convolution kernel sizes are 8×8, 4×4, 3×3, 2×2 and 2×2, and the step sizes of the convolution kernel movement are 4, 2, 2, 1, 1, each layer of convolution is followed by a ReLU activation function, and there is a fully connected layer at the end of the convolution layer, and finally a 1×512 feature vector is output. The feature vector is represented in the present invention as the relative position relationship between assemblies. The reinforcement learning decision network in the present invention adopts a general learning architecture composed of an action network and an evaluation network, which will not be repeated.

[0045] (2) Search phase

[0046] The state of external force between approach and search is identified as the dividing point. ext >F max , it is considered that the robot has made the first contact with the assembly surface, and then the robot state switches from approach to search.

[0047] In the search phase, the assembly axis at the end of the robot contacts the surface of the assembly hole. At this time, the deep reinforcement learning strategy π θ and analytical force control strategy π H work together to maximize the defined reward r t ,Strategy π θ As in the approach phase, the robot is guided to move according to the image to maximize the reward, and the strategy π HResponsible for the smooth interaction between the robot end and the environment. In the present invention, during the search phase, the robot end applies a constant force F to the normal direction of the assembly hole surface. d , the control rate of constant force is

[0048] F u =F d +k p ·F e +k i ∫F e dt,

[0049] where k p ,k i are proportional and integral parameters respectively, F e =F d -F ext is the error between the expected value and the actual value of the force. Dynamic admittance control is used in other directions, and the control rate is expressed as

[0050]

[0051] We further discretize the control rate for program control

[0052]

[0053]

[0054] Where M, B, K are the desired mass, damping and stiffness of the robot in motion. Δt is the control period, is the path tracking impedance error. This strategy can not only accurately track the robot end motion increment generated by the visual strategy, but also make compliant behavior to external disturbances, effectively protecting the safety of the robot. In the approach and search phases, the robot mainly completes the contact and alignment of the assembly axis and assembly hole.

[0055] (3) Insertion phase

[0056] The state recognition conditions for the second step search and the third step insertion are, |z-z0|<h and the external force F ext <∈Both conditions must be met at the same time to switch the assembly state. z represents the coordinate of the robot end on the z axis, z0 represents the coordinate of the bottom of the hole in the z axis direction, and h represents the height of the hole. ∈ is a small quantity. In the third stage, k2=0, the robot performs the insertion operation. When the external force detects F ext >F d / 2, the assembly is successful.

[0057] 3. Force control algorithm based on analytical method

[0058] In the analytical force control strategy: the force control algorithm block diagram designed by the present invention based on the analytical method is as follows Figure 3 The present invention combines explicit force control and admittance control in a parallel force-position framework, effectively achieving fast and efficient robot search while ensuring safety during robot training and deployment.

[0059] S=diag(s1,s2…s6),s∈{0,1}

[0060] To select the matrix, diag means converting the vector (s1, s2…s6) into a diagonal matrix. In the present invention, the assembly axis is clamped by the gripper installed at the end of the robot, and the 6-axis force / torque sensor is installed between the robot and the gripper, so the interaction force between the assembly axis and the assembly hole is subjected to median filtering for noise processing, the dead zone is subjected to drift processing, and the gravity compensation eliminates the influence of the gripper gravity.

[0061] In this invention, the action space in the deep reinforcement learning strategy is defined as

[0062] A=[Δx,Δy,Δz,Δα x ,Δα y ,Δα z ], are the displacement increment and rotation increment in Cartesian space, respectively, and the observation space is the image of the hand-eye camera The feedback amount of force control is F ext =[F x ,F y ,F z ,M x ,M y ,M z ]. The robot Cartesian space is represented by a 6-dimensional smooth manifold It can be further expressed as a 6-dimensional vector: in is the displacement increment, u and θ are the rotation axis and rotation angle respectively.

[0063] 4. Residual Reinforcement Learning Training Algorithm

[0064] The residual reinforcement learning training algorithm is summarized in more detail as follows:

[0065] (1) Initialization strategy π θ and π H ;

[0066] (2) For each Episode;

[0067] (3) Get status s t ;

[0068] (4) Calculate the reinforcement learning strategy uRL ~π θ (u t |s t );

[0069] (5) Calculate the residual reinforcement learning strategy u t =u RL +u H ;

[0070] (6) The robot executes u t , with priority p i Storage t-1 ,A t-1 ,R t ,γ t ,S t );

[0071] (7) Adopting priority batch sampling state;

[0072] (8) Update priority memory;

[0073] (9) Optimizing the reinforcement learning strategy π θ .

[0074] In order to enable the residual reinforcement learning strategy to be executed safely and efficiently on the robot, we also proposed a safe training mechanism. The description is as follows:

[0075] (1) Get parameter T max ,T one ,F max ,v max ,acc max ,a max ,v max ;

[0076] (2) At the maximum number of training steps T max Inside;

[0077] (3) Obtain the initial state s0;

[0078] (4) At the maximum number of continuous exploration steps T one Inside;

[0079] (5) Get the initial state s t ;

[0080] (6) Calculate the reinforcement learning strategy u RL ~π θ (u t |s t );

[0081] (7) Get the current external force F ext , integrated acceleration acc and velocity v;

[0082] (8) Get the 2-norm of the integrated acceleration acc norm and the 2-norm of velocity v norm ;

[0083] (9) If acc norm >acc max , then acc*=|acc max / acc norm |;

[0084] (10) If υ norm >υ max , then υ*=υ max / υ norm |;

[0085] (11) Calculate the residual strategy u t =u RL +u H ;

[0086] (12) If |u t -x| / Δt>ν max , then Cartesian space pose interpolation L i (u t )~ν max ;

[0087] (13) The robot executes u t ;

[0088] (14) If F ext >F max Or a>a max Or T>T one -1;

[0089] (15) Reset the robot;

[0090] (16) The whole process is repeated until it ends.

[0091] Among them, T max ,T one ,F max ,v max ,acc max ,a max ,v max They are the total time step, the maximum time step of a continuous exploration, the maximum contact force, the maximum integrated velocity, the maximum integrated acceleration, the maximum policy action increment and the maximum Cartesian velocity.

[0092] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A robot autonomous assembly method based on residual reinforcement learning, characterized in that: Here are the steps: (1) According to the state s of the robot during assembly t Define Cartesian space control instructions; (2) When the robot end approaches the assembly surface: set the proportional coefficient k1 = 0, and the Cartesian motion increment of the robot is determined by the deep reinforcement learning strategy π θ Decide; (3) When the external force F at the end of the robot ext >F max When F max is the maximum value allowed by the external force, the robot makes the first contact with the assembly surface, and the robot state switches from approaching to searching; through the deep reinforcement learning strategy π θ and analytical force control strategy π H Work together to maximize the custom reward function r t , deep reinforcement learning strategy π θ As in the approach phase, the robot is guided to move according to the image to maximize the reward function, and the strategy π H Responsible for the smooth interaction between the robot end and the environment, including the deep reinforcement learning strategy π θ The output is u RL , analyzing the force control strategy π H The output is u H ; (4) Determine whether to switch to the insertion stage, that is, determine whether the object to be assembled enters the assembly hole: when the condition |z-z0|<h and the external force F ext <∈When both are satisfied, the search phase enters the insertion phase; Among them, z represents the coordinate of the robot end on the z-axis, z0 represents the coordinate of the bottom of the hole in the z-axis direction, h represents the height of the hole, and ∈ is a small quantity; (5) Entering the insertion phase, the proportional coefficient k2 is set to 0, and the robot performs the insertion operation. When the external force F is detected, ext >F d / 2, F d If the external force is the expected one, the assembly is successful.

2. The robot autonomous assembly method based on residual reinforcement learning according to claim 1, characterized in that: Deep reinforcement learning policy π θ The online training algorithm used is PPO. Gaussian noise is added to the output command and training parameters during training. The noise can be expressed as N(μ,σ 2 ), μ is the mean, σ is the variance, and during training, the reward function r is set according to the characteristics of the assembly task. t , which can be expressed as Among them, R, S, I, C and F represent approach, search, insertion, completion and failure respectively. t 、v t represents the displacement and rotation increments, the subscripts R and S represent the distance between the assembly axis and the hole at the initial moment of each stage, respectively, and k d and k S is the scale factor.

3. The robot autonomous assembly method based on residual reinforcement learning according to claim 1, characterized in that: The state set s of the robot and the environment t It includes the depth image of the hand-eye camera and the 6-axis force / torque sensor installed at the end of the robot. The shaft hole parts to be assembled by the robot are included in the depth image of the hand-eye camera.

4. The robot autonomous assembly method based on residual reinforcement learning according to claim 1, characterized in that: During the search phase, the robot end applies a desired external force F to the normal direction of the assembly hole surface. d , the desired control rate of the external force is F u =F d +k p ·F e +k i ·∫F e dt, where k p ,k i are proportional and integral parameters respectively; F e =F d -F ext is the error between the expected value and the actual value of the force. Dynamic admittance control is used in other directions.

5. The robot autonomous assembly method based on residual reinforcement learning according to claim 4 is characterized in that: The control rate is expressed as: Among them, x e is the path tracking impedance error, For x e The first derivative of For x e The second derivative of The control rate is discretized for program control: Among them, M, B, and K are the desired mass, damping, and stiffness of the robot in motion, respectively. is x at time t e , Δt is the control period.

6. The robot autonomous assembly method based on residual reinforcement learning according to claim 5, characterized in that: Path tracking impedance error Among them, x p is the end position of the robot.

7. The robot autonomous assembly method based on residual reinforcement learning according to claim 1, characterized in that: In step (1), the Cartesian space control instruction x u =k1π H +k2π θ , where π θ is the deep reinforcement learning strategy, π H To analyze the force control strategy, k1,k2∈[0,1] are the proportional coefficients.

8. The robot autonomous assembly method based on residual reinforcement learning according to claim 1, characterized in that: In step (2), the visual strategy consists of two parts: a feature extraction convolutional neural network and a reinforcement learning decision network.

Images