Robot assembly learning method based on teaching reward state machine and residual reinforcement learning

Abstract

The present invention provides a robot assembly method based on a teaching reward state machine and residual reinforcement learning, belonging to the field of robotics and intelligent manufacturing. The method includes the following steps: building a physical hardware platform for robot assembly and a virtual simulation environment for reinforcement learning training; collecting the 6D pose teaching trajectory of the assembly object during the assembly process on the physical platform through drag-and-drop teaching; abstractly representing key information and state transitions during teaching, and constructing a reward function based on the state machine to improve learning efficiency; establishing a residual reinforcement learning method based on admittance control, and optimizing the admittance control assembly strategy through residual reinforcement learning to improve the safety of robot assembly. The present invention combines the advantages of a teaching reward state machine and residual reinforcement learning. It can not only quickly guide the robot to complete tasks through teaching, but also achieve autonomous learning and improved adaptability of the robot in complex environments through reinforcement learning. At the same time, it improves the safety of the system through admittance control.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing, and in particular relates to a robot assembly learning method based on a teaching reward state machine and residual reinforcement learning. Background Art

[0002] With the development of intelligent manufacturing technology, robots are increasingly being used in real-world production applications, particularly in assembly. Today, robots are required not only to possess precise operational capabilities but also to possess human-like learning and adaptability to acquire key skills and master a variety of similar yet diverse tasks. Deep reinforcement learning can endow robots with human-like learning capabilities, combining information from environmental interactions with intelligent decision-making. This allows robots to perform tasks flexibly and autonomously in complex environments, driving innovation and advancements in intelligent manufacturing. Recent studies have demonstrated limitations in deep reinforcement learning algorithms, including low sample efficiency, large data requirements, and long training times. These limitations make real-world deployments cumbersome and labor-intensive. This is primarily due to the large, high-dimensional space robots explore in unstructured and constantly changing environments, making the training process unstable in practice.

[0003] Robot teaching technology has always been a key technology in robot applications and has been successfully and widely used in robot operation tasks and continuous control problems. The advantages of robot teaching technology are intuitive operation, rapid deployment capabilities, flexible adaptation to different tasks and environments, cost reduction, and promotion of human-machine collaboration, making robot systems easier to learn and use, and having broad application potential. Recent studies have shown that robot teaching technology challenges the balance between teaching flexibility and complexity in applications, while reinforcement learning faces the challenges of training data acquisition costs and algorithm complexity. How to guide reinforcement learning training through teaching still faces many challenges. The integration of teaching technology and reinforcement learning is a difficult problem that requires continuous research and innovation to overcome. The teaching reward machine is a new method proposed in this invention to balance teaching technology and reinforcement learning technology.

[0004] Some studies have shown that robot control methods based on reinforcement learning algorithms, due to their instability, may negatively impact the performance and safety of robotic applications, whereas methods based on compliant control significantly enhance the smoothness and flexibility of robotic task execution. Compliant control-based robot learning methods face challenges in terms of real-time performance, smoothness, stability, and sample efficiency, requiring continuous optimization and balance in algorithm design and practical application to enable efficient and compliant robot execution in complex environments. Residual reinforcement learning offers a new solution, and this invention innovatively applies it to the field of intelligent and compliant robotic assembly.

[0005] In the robot reinforcement learning assembly method based on visual teaching and virtual-reality transfer, the efficiency issue of robot reinforcement learning training is taken into consideration, and the training efficiency is improved through visual teaching. However, this method has a small scope of application and is only applicable to online reinforcement learning methods. It also has poor adaptability and flexibility to the environment. In the robot autonomous assembly method based on residual reinforcement learning, the assembly process is mainly segmented, and end-to-end autonomous learning of robot assembly is achieved by combining vision and force perception. The learning efficiency issue is not considered, so a large amount of training resources are required. In the robot shaft-hole assembly method based on deep reinforcement learning and admittance control, the actual safety and compliance issues in assembly are taken into consideration. However, in actual applications, it is impossible to add existing teaching experience, and there is an instability problem, which reduces the safety of the method. Summary of the Invention

[0006] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology, and to provide a robot assembly learning method based on a teaching reward state machine and residual reinforcement learning. The reward function is designed through a state machine, and the teaching technology is combined with reinforcement learning to solve the limitations of the reinforcement learning algorithm training efficiency in the background technology, the difficulties in the integration of teaching technology, and the system security and stability problems. Combined with the residual reinforcement learning algorithm, combined with the robot's compliant control method, and using information including the robot body and force sensor, a method for robot intelligent assembly task learning is realized.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a robot assembly method based on a teaching reward state machine and residual reinforcement learning, comprising the following steps:

[0009] Step S0: Building a physical hardware platform for robot assembly and a virtual simulation environment for reinforcement learning training;

[0010] Step S1: on the physical platform, collect the 6D pose teaching trajectory of the assembly object during the assembly process by dragging and teaching;

[0011] Step S2: abstractly represent the key information and state transitions in the teaching process, and construct a reward function based on the state machine to improve learning efficiency;

[0012] Step S3: Establish a residual reinforcement learning method based on admittance control, optimize the admittance control assembly strategy through residual reinforcement learning, and improve the safety of robot assembly.

[0013] As a preferred technical solution, the S0 includes the following steps:

[0014] S01. Build a robot as an intelligent agent in a simulation environment, import the physical simulation model of the robot and the assembly environment into the simulation environment, and fix the mechanical gripper to the end of the robot;

[0015] S02. Create an axis-hole assembly object, place the base of the assembly hole within the robot's operating space, and adjust the position of the simulation model based on the position of the hole in reality.

[0016] S03. Control the robot's joint motion through the simulator's physical interface, simulate collision relationships, design the robot's motion controller, and define the state space and action space of the simulation environment:

[0017] S t =[x e ,y e ,z e ,x t ,y t ,z t ,f x ,f y ,f z ,s m ]

[0018] A t =[δ x ,δ y ,δ z ]

[0019] Among them, (x e ,y e ,z e ) is the position of the end effector, (x t ,y t ,z t ) indicates the position of the assembly hole, (f x ,f y ,f z ) represents the external force on the end of the robot, s m is generated from the state machine, (δ x ,δ y ,δ z ) is the displacement of the end effector, S t is the state space of reinforcement learning, A t The action space for reinforcement learning;

[0020] S04. When the robot interacts with the environment, it controls the input of the action space to control the robot. The reinforcement learning environment will update the state of the environment based on the action input. The specific execution steps are as follows:

[0021] Step S041: Obtain the state of the environment, obtain a new state based on the admittance control principle, and calculate the control signal of the robot; then input the state into the residual reinforcement learning actor network to obtain the offset of the control signal, and calculate the actual action of the robot according to the set weights;

[0022] Step S042: After the robot performs the action, it is determined whether it is a safe action based on the magnitude of the force. If it exceeds the threshold, the environment will perform a reset operation to initialize the robot to its original state.

[0023] As a preferred technical solution, S1 includes the following steps:

[0024] Step S11: calibrate the camera's intrinsic and extrinsic parameters using the eye-on-hand robot camera installation method and the robot hand-eye calibration method;

[0025] Step S12: Place four different markers at fixed positions around the assembly hole, move the robot end to the top of the assembly object, photograph the position of the assembly object, obtain a record of the assembly object and the markers, and locate the markers using the camera.

[0026] Step S13: The image coordinate system is transferred to the world coordinate system through the intrinsic and extrinsic parameters of the camera, the actual physical position of the marker is calculated, and the position of the assembly object is calculated in combination with the position of the robot end;

[0027] Step S14: calibrating the robot's tools;

[0028] Step S15: drag the robot end so that the robot can successfully complete the robot shaft-hole assembly action, record the position of the robot end in real time, and collect a set of 6D pose data to obtain the assembly teaching trajectory.

[0029] As a preferred technical solution, the S11 includes the following steps:

[0030] S111. Select a high-precision camera and a suitable light source and install them at the end of the robot.

[0031] S112. Select a calibration target, place a marker in the work area, and perform a series of movements in the work area by moving the robot arm to ensure that the camera can observe the calibration target from different angles and positions, and obtain image data of the marker at different angles;

[0032] S113, using image processing algorithms to detect and extract the position of the markers and calculate the transformation matrix between the robot end and the camera;

[0033] S114: Verify the calibration result, move the robot arm to a new position, compare the actually observed target position with the position calculated in the calibration result, and verify the calibration result.

[0034] As a preferred technical solution, the S14 includes the following steps:

[0035] S141. Install the end effector, move the robot, select four robot orientations with relatively large differences, make the end of the robot as close to the reference point as possible, record the position and posture of the four points, and obtain the position and posture of the four end coordinates;

[0036] S142. Convert the positions and postures of the four points into four transformation matrices T1, T2, T3, and T4, and solve the robot tool coordinate system using the following formula:

[0037]

[0038]

[0039]

[0040] Among them, the transformation matrix between the end and the tool Transformation matrix between base and tool Keep the transformation relationship between the base and the end unchanged Substitute into the equation to solve, is the rotation matrix between the tool and the basis, is the rotation matrix between the end and the tool, is the rotation matrix between the end and the base, p Bo 、p Etp 、p Btp is the translation vector between the base, tool and end coordinate systems.

[0041] As a preferred technical solution, S2 includes the following steps:

[0042] Step S21: Clean the collected trajectory data, detect and remove abnormal values ​​and outliers, improve the smoothness and stability of the data through mean filtering, and then reduce the number of data points through downsampling;

[0043] Step S22: Segment the processed trajectory and extract key information, decompose the continuous actions demonstrated by the instructor into independent subtasks, and use a clustering algorithm to group the instructor's actions into clusters with similar motion characteristics. Each cluster corresponds to a subtask, and a state machine is defined;

[0044] Step S23: Design a reward function for each state. When a state transitions, define rewards or penalties based on the achievement of the goal, prompting the robot to try new actions after the state transition, and define the reward function of the state machine.

[0045] As a preferred technical solution, the S22 includes the following steps:

[0046] Step S221: Determine the starting and ending states of the state machine based on the starting and ending positions of the assembly, define the intermediate states of the state machine based on the segmented trajectory, and define the states based on the robot body information and environment information;

[0047] Step S222: defining migration conditions based on the robot's action space and external state response, including the states of the shaft-hole assembly: approach, hole search, and assembly;

[0048] Step S223: The robot performs the taught assembly action and tests the reward machine to ensure that it can achieve the expected system behavior.

[0049] As a preferred technical solution, the S23 includes the following steps:

[0050] S231, the reward is determined by the state machine defined by the teaching. A reward is defined for each state. When the state machine successfully migrates, the reward opportunity generates a reward based on the value of the migration, driving the robot to learn towards the final successful state;

[0051] S232. One type of the above rewards is a dense reward, that is, a positive reward set under a certain state machine that is inversely proportional to the distance; the other type is a sparse reward, which is related to the state machine migration. When the state machine migrates forward, a reward is given, and when the state machine migrates backward, a penalty is given.

[0052] As a preferred technical solution, S3 includes the following steps:

[0053] Step S31: Design a controller for robot motion using the admittance control principle. When the robot end is affected by external forces, the position of the robot end effector can be adjusted in real time to achieve robot installation control in Cartesian space.

[0054] Step S32: Residual reinforcement learning is used to improve the control system's anti-interference capability. When the assembly target position shifts, the robot adjusts the output of the admittance controller through residual reinforcement learning and superimposes this output on the admittance control signal to control the robot's completion of the assembly task. The task is considered complete when the robot reaches a certain assembly position and the forces in each direction are within a certain threshold.

[0055] As a preferred technical solution, the S31 is specifically as follows:

[0056] Use force sensors to capture the force information of the external environment at the end of the robot in real time, and feed the force information back to the robot controller through the admittance control algorithm. Adjust the posture and movement of the robot according to the changes in external force or position. By introducing the end stiffness matrix K e and the terminal damping matrix B e To achieve this, the specific formula is as follows:

[0057]

[0058] Among them F ext represents the force exerted by the external environment on the end of the robot, x d is the desired end position, x is the actual end position, is the desired terminal velocity, The actual terminal velocity is converted from the terminal control signal to the joint space through the robot inverse kinematics, and the robot is controlled by the robot joint controller.

[0059] As a preferred technical solution, the S32 is specifically as follows:

[0060] S321 uses the improved DDPG reinforcement learning algorithm DDPGfD, the actor-critic framework based on continuous space deterministic policy gradient, and the model-free reinforcement learning algorithm. Before training, the teaching data is saved to the experience replay buffer to pre-train the policy network; the residual reinforcement learning environment will pass the policy a=k1π A (f)+k2π θ (s|θ π )+N generates actions and then generates new migration data e=(s,a,r=RM(s,a),s`,P(.|s,a)) which are then stored in the experience replay buffer for training. Among them, k1 and k2 are the weights of the admittance control strategy and the reinforcement learning strategy, π A (f) is the admittance control strategy, f is the external environmental force, π θ (s|θ π ) is the reinforcement learning strategy, θ π is the policy network weight, N is the generated random noise, s is the state information, a is the action information, r is the state machine reward, s` is the next state information, and P(.|s,a) is the state transition probability;

[0061] S322, during the training process, sampling is performed from the experience pool. DDPGfD uses priority replay to sample from the teaching and exploration data, and calculates the probability of sampling by priority. The priority is calculated as follows Among them, δ i is the gradient error, λ is the weight, is the loss of the actor network, ∈ is a constant variable, ∈ D is the constant variable of the teaching data;

[0062] S323 calculates the loss L1(θ Q )=E (s,a,r,s′)～D [R1-Q(s,a|θ Q )] 2 , among which, reward R1=r+λQ′(s′,π′(s′|θ π′ )|θ Q′ ),Q′(.|θ Q′ ) and π′(.|θ π′ ) is the target network, θ is the weight; calculate the gradient updating the network of actors and critics;

[0063] The reinforcement learning network structure adopts a convolutional neural network with input and output as . For the residual reinforcement learning of admittance control, the actor network input of reinforcement learning is set to S t =[x e ,y e ,z e ,x t ,y t ,z t ,f x ,f y ,f z ,s m ], the output is A t =[δ x ,δ y ,δ z ], which is used to be superimposed on the output of the admittance controller. The actual position of the robot end control is [x+δ x ,y+δ y ,z+δ z ], [x,y,z] is the admittance control output.

[0064] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0065] This invention combines the advantages of a teaching-based reward state machine and residual reinforcement learning. It can not only quickly guide a robot to complete tasks through teaching, but also achieve autonomous learning and improved adaptability in complex environments through reinforcement learning, while also improving system safety through admittance control. The method of the present invention uses an abstract state machine to enable the robot to learn more environmental characteristics and patterns, thereby finding the optimal strategy. Furthermore, the guidance provided by teaching significantly reduces algorithm training time, fully combining the exploratory nature of reinforcement learning with the efficiency of expert teaching. The method of the present invention not only effectively addresses the variability and uncertainty in assembly tasks, but also adapts to the requirements of diverse industrial scenarios, thereby increasing the application scope and effectiveness of robots in industrial assembly. By combining residual reinforcement learning with an admittance control algorithm, the present invention enables further learning based on existing control strategies. The robot can achieve compliant control while also responding to external environmental information, addressing safety issues while also improving its adaptability and generalization capabilities in unknown environments. Therefore, the present invention offers significant innovation and practicality in improving robot assembly efficiency, adaptability, and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 This is a flow chart of the robot assembly learning method based on teaching reward state machine and residual reinforcement learning of the present invention;

[0068] Figure 2 It is a framework of a preferred embodiment of the robot assembly learning method based on teaching reward state machine and residual reinforcement learning of the present invention;

[0069] Figure 3 A flowchart of a reinforcement learning environment design method for a preferred embodiment of the robot assembly learning method based on a teaching reward state machine and residual reinforcement learning of the present invention;

[0070] Figure 4 A flowchart of a teaching reward machine design method of a preferred embodiment of a robot assembly learning method based on a teaching reward state machine and residual reinforcement learning of the present invention;

[0071] Figure 5 The present invention provides a control framework for a residual reinforcement learning method based on admittance control, which is a preferred embodiment of the robot assembly learning method based on a teaching reward state machine and residual reinforcement learning. DETAILED DESCRIPTION

[0072] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0073] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0074] In one embodiment, Figure 1 、 Figure 2 As shown, the present invention provides a robot assembly learning method based on a teaching reward state machine and residual reinforcement learning, comprising the following steps:

[0075] Step S0: Build a robot shaft-hole assembly hardware platform using a six-axis Elite robot. The range of motion of each joint is [-360°, 360°]. The position control of the robot end is achieved through the robot kinematic solver. A two-finger gripper is selected as the clamping device for the assembly object, which can perform both opening and closing operations. A simulation model of the robot and assembly environment is built in the MuJoCo simulator, and a reinforcement learning environment is designed based on the reinforcement learning algorithm.

[0076] Further, such as Figure 3 As shown, the simulation training environment in step S0 is specifically:

[0077] Step S01: Build a robot as an intelligent agent in a simulation environment, import the physical simulation model of the Elite robot and the assembly environment into the MuJoCo simulation environment, and fix a two-finger gripper to the end of the robot;

[0078] Step S02: Create an axis-hole assembly object, place the base of the assembly hole within the robot's operating space, and adjust the position of the simulation model based on the position of the hole in reality;

[0079] Step S03: Control the joint motion of the robot through the physical interface of the simulator, simulate the collision relationship, design the controller of the robot motion, and define the state space and action space of the simulation environment.

[0080] St =[x e ,y e ,z e ,x t ,y t ,z t ,f x ,f y ,f z ,s m ]

[0081] A t =[δ x ,δ y ,δ z ]

[0082] Among them, (x e ,y e ,z e ) is the position of the end effector, (x t ,y t ,z t ) indicates the position of the assembly hole, (f x ,f y ,f z ) represents the external force on the end of the robot. m is generated from the state machine. (δ x ,δ y ,δ z ) is the displacement of the end effector. S t is the state space of reinforcement learning, A t The action space for reinforcement learning.

[0083] Step S04: When the robot interacts with the environment, it controls the robot by controlling the input of the action space. The reinforcement learning environment will update the state of the environment based on the action input. The specific execution steps are as follows:

[0084] S041. Obtain the state of the environment, obtain a new state based on the admittance control principle, and calculate the control signal of the robot;

[0085] S042. Input the state into the residual reinforcement learning actor network to obtain the control signal offset. The robot's actual motion is calculated based on the set weights. The output range of the motion space is [-1mm, 1mm]. After the action is executed, the force is used to determine whether it is a safe action. If the force is greater than 10N, the environment will perform a reset operation, initializing the robot to its original state.

[0086] Step S1: Fix an RGB camera on the end of the robot, obtain the actual physical position of the assembly hole through visual methods, clamp the assembly object on the end of the robot, and drag the end of the robot to obtain the teaching trajectory of the robot assembly; the teaching reward machine design method is as follows Figure 4 shown.

[0087] Furthermore, step S1 specifically includes:

[0088] Step S11: calibrate the camera's intrinsic and extrinsic parameters using the eye-on-hand robot camera installation method and the robot hand-eye calibration method;

[0089] Furthermore, the step S11 includes the following steps:

[0090] Step S111: Select a high-precision RGB camera and a natural light source, and install the camera on the end of the robot using an eye-on-hand installation method;

[0091] Step S112: Select a 9*6 checkerboard as the calibration target, place a checkerboard in the working area, and perform a series of movements in the working area by moving the robot arm to ensure that the camera can observe the calibration target from different angles and positions, and obtain image data of the checkerboard at different angles;

[0092] Step S113: Use image processing algorithms to detect and extract feature points of the calibration target and solve the transformation matrix between the robot end and the camera;

[0093] Step S114: Verify the calibration result, move the robot arm to a new position, compare the actually observed target position with the position calculated in the calibration result, and verify the calibration result.

[0094] Step S12: Place four different ArUco codes at fixed positions around the assembly hole, move the robot end over the assembly hole, photograph the assembly hole, obtain an image of the assembly hole and the ArUco code, record the position of the robot end, and locate the ArUco code using the camera;

[0095] Step S13: The image coordinate system is transferred to the world coordinate system through the intrinsic and extrinsic parameters of the camera, the actual physical position of the ArUco code is calculated, and the position of the assembly hole is calculated in combination with the position of the robot end;

[0096] Step S14: calibrating the robot's tools;

[0097] Furthermore, step S14 is specifically as follows:

[0098] Step S141: Move the robot, select four robot orientations with relatively large differences, make the end of the robot as close to the reference point as possible, record the positions and postures of the four points, and obtain the positions and postures of the four end coordinates;

[0099] Step S142: Convert the positions and postures of the four points into four transformation matrices T1, T2, T3, and T4. Solve the robot tool coordinate system using the following formula:

[0100]

[0101]

[0102]

[0103] Among them, the transformation matrix between the end and the tool Transformation matrix between base and tool Keep the transformation relationship between the base and the end unchanged Substitute into the equation to solve, is the rotation matrix between the tool and the basis, is the rotation matrix between the end and the tool, is the rotation matrix between the end and the base, p Bo 、p Etp 、p Btp is the migration vector between the base, tool and end coordinate systems;

[0104] Step S15: drag the robot end so that the robot can successfully complete the robot shaft-hole assembly action, record the position of the robot end in real time, and collect a set of 6D pose data to obtain the assembly teaching trajectory.

[0105] Step S2: Preprocess the acquired teaching trajectory by denoising, filtering, and downsampling the data. Then, extract key information and state transition relationships, segment the assembly process, design a state machine, and design a reward machine function based on conditional reinforcement learning and sparse reward functions. This includes:

[0106] Step S21: Clean the collected trajectory data, detect and remove abnormal values ​​and outliers, improve the smoothness and stability of the data through mean filtering, and then reduce the number of data points through downsampling;

[0107] Step S22: Segment the processed trajectory and extract key information, decompose the continuous actions demonstrated by the instructor into independent subtasks, and use a clustering algorithm to group the instructor's actions into clusters with similar motion characteristics. Each cluster corresponds to a subtask, and a state machine is defined;

[0108] Furthermore, the state machine design method in step S22 is as follows:

[0109] Step S221: Determine the starting and ending states of the state machine based on the starting and ending positions of the assembly, define the intermediate states of the state machine based on the segmented trajectory, and define the states based on the robot body information and environment information;

[0110] Step S222: defining migration conditions based on the robot's action space and external state response, including the states of the shaft-hole assembly: approach, hole search, assembly, etc.;

[0111] Step S223: The robot performs the taught assembly action and tests the reward machine to ensure that it can achieve the expected system behavior.

[0112] Step S23: Design a reward function for each state. When a state transitions, define rewards or penalties based on the achievement of the goal, prompting the robot to try new actions after the state transition, and define the reward function of the state machine.

[0113] Furthermore, the step S23 is as follows:

[0114] Step S231: The reward is determined by the state machine defined by the teaching. A reward is defined for each state. When the state machine successfully migrates, the reward opportunity generates a reward based on the value of the migration, driving the robot to learn towards the final successful state.

[0115] Step S232: One type of the above rewards is a dense reward, that is, a positive reward set under a certain state machine that is inversely proportional to the distance; the other type is a sparse reward, which is related to the state machine migration. When the state machine migrates forward, a reward of 1 is given, and when the state machine migrates backward, a penalty of -1 is given.

[0116] Step S3: Install a force sensor between the robot's end and the actuator, design a basic assembly strategy based on the admittance control algorithm, and use a residual reinforcement learning method based on force sensor information and a force control strategy based on fused admittance control to achieve end-to-end robot assembly learning. This specifically includes:

[0117] Step S31: Design a controller for robot motion using the admittance control principle. When the robot end is affected by external forces, the position of the robot end effector can be adjusted in real time to achieve robot installation control in Cartesian space.

[0118] Furthermore, Figure 5 As shown, the step S31 is specifically as follows:

[0119] The ATI force sensor is used to capture the force information of the external environment at the end of the robot in real time, and the force information is fed back to the robot controller through the admittance control algorithm to adjust the robot's posture and movement according to the changes in external force or position. By introducing the end stiffness matrix K e and the terminal damping matrix B e The specific formula is as follows: Among them F ext represents the force exerted by the external environment on the end of the robot, x d is the desired end position, x is the actual end position, is the desired terminal velocity, The actual terminal velocity is converted from the terminal control signal to the joint space through the robot inverse kinematics, and the robot is controlled by the robot joint controller.

[0120] Step S32: Residual reinforcement learning is used to improve the control system's anti-interference capability. When the assembly target position shifts, the robot adjusts the output of the admittance controller through residual reinforcement learning and superimposes this output on the admittance control signal to control the robot's completion of the assembly task. The task is considered complete when the robot reaches a certain assembly position and the forces in each direction are within a certain threshold.

[0121] Furthermore, the step S32 is specifically as follows:

[0122] Step S321, the present invention uses an improved DDPG reinforcement learning algorithm DDPGfD, an actor-critic framework based on continuous space deterministic policy gradient, and a model-free reinforcement learning algorithm. Before training, the teaching data is saved to the experience replay buffer to pre-train the policy network. The residual reinforcement learning environment will pass the policy a=k1π A (f)+k2π θ (s|θ π )+N generates an action and then generates new migration data e=(s,a,r=RM(s,a),s`,P(.|s,a)) which is then stored in the experience replay buffer for training, where s is the state information, a is the action information, r is the state machine reward, s` is the next state information, and P(.|s,a) is the state transition probability;

[0123] Step S322: During the training process, samples are taken from the experience pool. DDPGfD uses priority replay to sample from the teaching and exploration data. The probability of sampling is calculated by priority. The priority is calculated as follows

[0124] Step S323 calculates the loss L1(θ Q )=E(s,a,r,s′)～D [R1-Q(s,a|θ Q )] 2 , calculate the gradient Updated actor and critic network.

[0125] Furthermore, the reinforcement learning network structure adopts a convolutional neural network with input and output as . For the residual reinforcement learning of admittance control, the actor network input of reinforcement learning is set to S t =[x e ,y e ,z e ,x t ,y t ,z t ,f x ,f y ,f z ,s m ], the output is A t =[δ x ,δ y ,δ z ], which is used to be superimposed on the output of the admittance controller. The actual position of the robot end control is [x+δ x ,y+δ y ,z+δ z ], [x,y,z] is the admittance control output.

[0126] The present invention discloses a robot assembly learning method based on a teaching reward state machine and residual reinforcement learning. It combines the advantages of the teaching reward state machine and residual reinforcement learning. It can not only quickly guide the robot to complete tasks through teaching, but also realize autonomous learning and adaptability of the robot in complex environments through reinforcement learning, while improving the safety of the system through admittance control. This method can not only effectively cope with the changes and uncertainties in assembly tasks, but also adapt to the requirements of different industrial scenarios, and improve the application scope and effectiveness of robots in the field of industrial assembly. Therefore, the present invention has significant innovation and practicality in improving the efficiency, adaptability and safety of robot assembly.

[0127] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

[0128] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A robot assembly method based on teaching reward state machine and residual reinforcement learning, characterized by: The steps include: Step S0: Building a physical hardware platform for robot assembly and a virtual simulation environment for reinforcement learning training; Step S1: on the physical platform, collect the 6D pose teaching trajectory of the assembly object during the assembly process by dragging and teaching; Step S2: abstractly represent the key information and state transitions in the teaching process, and construct a reward function based on the state machine to improve learning efficiency; The S2 comprises the following steps: Step S21: Clean the collected trajectory data, detect and remove abnormal values ​​and outliers, improve the smoothness and stability of the data through mean filtering, and then reduce the number of data points through downsampling; Step S22: Segment the processed trajectory and extract key information, decompose the continuous actions demonstrated by the instructor into independent subtasks, and use a clustering algorithm to group the instructor's actions into clusters with similar motion characteristics. Each cluster corresponds to a subtask, and a state machine is defined; Step S23: Design a reward function for each state. When a state transitions, define rewards or penalties based on the achievement of the goal, prompting the robot to try new actions after the state transition. Define the reward function of the state machine. The S22 includes the following steps: Step S221: Determine the starting and ending states of the state machine based on the starting and ending positions of the assembly, define the intermediate states of the state machine based on the segmented trajectory, and define the states based on the robot body information and environment information; Step S222: defining migration conditions based on the robot's action space and external state response, including the states of the shaft-hole assembly: approach, hole search, and assembly; Step S223: The robot performs the taught assembly action and tests the reward machine to ensure that it can achieve the expected system behavior; The S23 includes the following steps: S231, the reward is determined by the state machine defined by the teaching. A reward is defined for each state. When the state machine successfully migrates, the reward opportunity generates a reward based on the value of the migration, driving the robot to learn towards the final successful state; S232. One type of the above rewards is a dense reward, which is a positive reward set under a certain state machine and is inversely proportional to the distance; the other type is a sparse reward, which is related to the state machine migration. When the state machine migrates forward, a reward is given, and when the state machine migrates backward, a penalty is given; Step S3: Establish a residual reinforcement learning method based on admittance control, optimize the admittance control assembly strategy through residual reinforcement learning, and improve the safety of robot assembly.

2. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 1 is characterized in that: The S0 comprises the following steps: S01. Build a robot as an intelligent agent in a simulation environment, import the physical simulation model of the robot and the assembly environment into the simulation environment, and fix the mechanical gripper to the end of the robot; S02. Create an axis-hole assembly object, place the base of the assembly hole within the robot's operating space, and adjust the position of the simulation model according to the position of the hole in reality; S03. Control the robot's joint motion through the simulator's physical interface, simulate collision relationships, design the robot's motion controller, and define the state space and action space of the simulation environment: S t =[x e ,y e ,z e ,x t ,y t ,z t ,f x ,f y ,f z ,s m ] A t =[δ x ,d y ,d z ] Among them, (x e ,y e ,z e ) is the position of the end effector, (x t ,y t ,z t ) indicates the position of the assembly hole, (f x ,f y ,f z ) represents the external force on the end of the robot, s m is generated from the state machine, (δ x ,δ y ,δ z ) is the displacement of the end effector, S t is the state space of reinforcement learning, A t The action space for reinforcement learning; S04. When the robot interacts with the environment, it controls the input of the action space to control the robot. The reinforcement learning environment will update the state of the environment based on the action input. The specific execution steps are as follows: Step S041: Obtain the state of the environment, obtain a new state based on the admittance control principle, and calculate the control signal of the robot; then input the state into the residual reinforcement learning actor network to obtain the offset of the control signal, and calculate the actual action of the robot according to the set weights; Step S042: After the robot performs the action, it is determined whether it is a safe action based on the magnitude of the force. If it exceeds the threshold, the environment will perform a reset operation to initialize the robot to its original state.

3. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 1 is characterized in that: Said S1 comprises the following steps: Step S11: calibrate the camera's intrinsic and extrinsic parameters using the eye-on-hand robot camera installation method and the robot hand-eye calibration method; Step S12: Place four different markers at fixed positions around the assembly hole, move the robot end to the top of the assembly object, photograph the position of the assembly object, obtain a record of the assembly object and the markers, and locate the markers using the camera. Step S13: The image coordinate system is transferred to the world coordinate system through the intrinsic and extrinsic parameters of the camera, the actual physical position of the marker is calculated, and the position of the assembly object is calculated in combination with the position of the robot end; Step S14: calibrating the robot's tools; Step S15: drag the robot end so that the robot can successfully complete the robot shaft-hole assembly action, record the position of the robot end in real time, and collect a set of 6D pose data to obtain the assembly teaching trajectory.

4. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 3 is characterized in that: The S11 includes the following steps: S111. Select a high-precision camera and a suitable light source and install them at the end of the robot. S112. Select a calibration target, place a marker in the work area, and perform a series of movements in the work area by moving the robot arm to ensure that the camera can observe the calibration target from different angles and positions, and obtain image data of the marker at different angles; S113, using image processing algorithms to detect and extract the position of the markers and calculate the transformation matrix between the robot end and the camera; S114: Verify the calibration result, move the robot arm to a new position, compare the actually observed target position with the position calculated in the calibration result, and verify the calibration result.

5. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 3 is characterized in that: The S14 includes the following steps: S141. Install the end effector, move the robot, select four robot orientations with relatively large differences, make the end of the robot as close to the reference point as possible, record the position and posture of the four points, and obtain the position and posture of the four end coordinates; S142. Convert the positions and postures of the four points into four transformation matrices T1, T2, T3, and T4, and solve the robot tool coordinate system using the following formula: Among them, the transformation matrix between the end and the tool Transformation matrix between base and tool Keep the transformation relationship between the base and the end unchanged Substitute into the equation to solve, is the rotation matrix between the tool and the base, is the rotation matrix between the end and the tool, is the rotation matrix between the end and the base, p Bo 、p Etp 、p Btp is the translation vector between the base, tool and end coordinate systems.

6. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 1 is characterized in that: The S3 includes the following steps: Step S31: Design a controller for robot motion using the admittance control principle. When the robot end is affected by external forces, the position of the robot end effector can be adjusted in real time to achieve robot installation control in Cartesian space. Step S32: Improve the anti-interference ability of the control system through residual reinforcement learning. When the assembly target position is offset, the robot adjusts the output of the admittance controller through residual reinforcement learning, and superimposes the output on the admittance control signal to control the robot to complete the assembly task; it is set that the assembly task is completed when the robot reaches a certain assembly position and the force in each direction is within a certain threshold range.

7. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 6 is characterized in that: The S31 is specifically as follows: Use force sensors to capture the force information of the external environment at the end of the robot in real time, and feed the force information back to the robot controller through the admittance control algorithm. Adjust the posture and movement of the robot according to the changes in external force or position. By introducing the end stiffness matrix K e and the terminal damping matrix B e To achieve this, the specific formula is as follows: Among them F ext represents the force exerted by the external environment on the end of the robot, x d is the desired end position, x is the actual end position, is the desired terminal velocity, is the actual terminal velocity; the terminal control signal is converted into joint space through the robot inverse kinematics, and the robot control is realized through the robot's joint controller.

8. The robot assembly method based on teaching reward state machine and residual reinforcement learning according to claim 6 is characterized in that: The S32 is specifically as follows: S321 uses the improved DDPG reinforcement learning algorithm DDPGfD, the actor-critic framework based on continuous space deterministic policy gradient, and the model-free reinforcement learning algorithm. Before training, the teaching data is saved to the experience replay buffer to pre-train the policy network; the residual reinforcement learning environment will pass the policy a=k1π A (f)+k2π θ (s|θ π )+N generates an action and then generates new migration data e=(s,a,r=RM(s,a),s`,P(.|s,a)) which is then stored in the experience replay buffer for training; where k1 and k2 are the weights of the admittance control strategy and the reinforcement learning strategy, π A (f) is the admittance control strategy, f is the external environmental force, π θ (s|θ π ) is the reinforcement learning strategy, θ π is the policy network weight, N is the generated random noise, s is the state information, a is the action information, r is the state machine reward, s` is the next state information, and P(.|s,a) is the state transition probability; S322, during the training process, sampling is performed from the experience pool. DDPGfD uses priority replay to sample from the teaching and exploration data, and calculates the probability of sampling by priority. The priority is calculated as follows Among them, δ i is the gradient error, λ is the weight, is the loss of the actor network, ∈ is a constant variable, ∈ D is the constant variable of the teaching data; S323 calculates the loss L1(θ Q )=E (s,a,r,s′)～D [R1-Q(s,a|θ Q )] 2 , among which, reward R1=r+λQ′(s′,π′(s′|θ π′ )|θ Q′ ),Q′(.|θ Q′ ) and π′(.|θ π′ ) is the target network, θ is the weight; calculate the gradient updating the network of actors and critics; The reinforcement learning network structure adopts a convolutional neural network with input and output as . For the residual reinforcement learning of admittance control, the actor network input of reinforcement learning is set to S t =[x e ,y e ,z e ,x t ,y t ,z t ,f x ,f y ,f z ,s m ], the output is A t =[δ x ,δ y ,δ z ], which is used to be superimposed on the output of the admittance controller. The actual position of the robot end control is [x+δ x ,y+δ y ,z+δ z ], [x,y,z] is the admittance control output.

Images