Servomotor cylinder inner wall weld joint robot grinding multi-objective optimization method and system based on multi-task reinforcement learning

Abstract

The invention belongs to the related technical field of grinding robots, and discloses a servomotor cylinder inner wall weld joint robot grinding multi-objective optimization method and system based on multi-task reinforcement learning. The method comprises the following steps: constructing simulation models of a moving unit, a polishing robot and a servomotor cylinder body; the grinding process is divided into the process that a grinding head moves towards a weld joint starting point and the process that the grinding head moves after reaching the starting point; respectively constructing respective multi-task reinforcement learning optimization models for the two processes, wherein the multi-task reinforcement learning optimization models take a state space as input and an action space as output; and changing the length and the position of the welding seam to perform polishing simulation, and training the multi-task reinforcement learning optimization model of the two processes until the preset training times are reached and the reward function of the optimization target is converged. Through the method, the problem that multi-target optimization such as collision avoidance, grinding accessibility and rigidity maximization cannot be considered at the same time in the robot grinding process is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to grinding robots, and more specifically, relates to a multi-objective optimization method and system for robot grinding of inner wall welds of a servomotor cylinder body based on multi-task reinforcement learning. Background Art

[0002] The servomotor is a component in a hydroelectric unit that converts hydraulic energy into the mechanical energy required by the transmission mechanism. During operation, due to reasons such as sediment deposition, the servomotor cylinder and the piston that reciprocates axially inside it will scrape against each other, resulting in axial scratches on the bottom. The leakage caused by the scratches will affect its normal operation, and the scratches need to be inspected, planed away in a regular manner, welded, milled, polished, and repaired.

[0003] After the axial scratches are regularly planed, repair welding is performed. The weld extends along the axial direction and presents a regular long strip. The weld needs to be polished to improve the finish and shape accuracy of the inner wall of the cylinder body after repair, and to remove welding residues such as welding slag. The weld is polished by a UGV trolley equipped with a robotic arm. When the end grinding disc approaches the grinding starting point, the trolley is used to move with the robotic arm; when the end grinding disc moves to the grinding starting point, the trolley stops moving, and the robotic arm completes the weld grinding work alone.

[0004] Since the weld in the cylinder is long, if the position of the trolley during grinding and the redundant angle around the end are unreasonable, it will lead to the inability to grind the entire weld and the robot's rigidity is weak; in addition, during the motion control process, it is necessary to consider avoiding collisions between the robot's own connecting rods and collisions with the cylinder. Traditional decision-making control methods cannot consider collision avoidance, grinding accessibility, and rigidity optimization at the same time. Modern artificial intelligence technology can be used for decision-making control with multiple optimization objectives. Therefore, multi-task reinforcement learning is used to control the motion process of the trolley and the robot arm with the end approaching the starting point and optimizing the adjustment. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a multi-objective optimization method and system for robot grinding of the inner wall weld of a relay cylinder body based on multi-task reinforcement learning, which solves the problem that multi-objective optimization such as collision avoidance, grinding accessibility and stiffness maximization cannot be considered simultaneously during the robot grinding process.

[0006] To achieve the above object, according to one aspect of the present invention, a multi-objective optimization method for robot grinding of inner wall weld of servomotor cylinder based on multi-task reinforcement learning is provided, and the method comprises the following steps:

[0007] A simulation model of a grinding robot grinding a servo cylinder is constructed, and the grinding process is divided into the process of the grinding head moving toward the starting point of the weld and the movement process after the grinding head reaches the starting point. A multi-task reinforcement learning optimization model is constructed for each of the two processes. The multi-task reinforcement learning optimization model takes the state space as input and the action space as output.

[0008] The length and position of the weld are changed, and the grinding simulation is performed using the simulation model. The data of the two processes obtained from each simulation are used to train the respective multi-task reinforcement learning optimization models until each of the multi-task reinforcement learning optimization models reaches a preset number of training times and the reward function of the optimization objective converges.

[0009] Further preferably, the state space corresponding to the multi-task reinforcement learning optimization models of the two processes is the same, and the state space is composed of the joint angles of the grinding robot, the spatial coordinates of the grinding robot base, the length of the weld, the position vector between the end of the grinding robot and the starting point of the weld, and the rotation angle of the end of the grinding robot relative to its initial posture.

[0010] Further preferably, the action space of the multi-task reinforcement learning optimization model of the process of the grinding head moving toward the starting point of the weld is composed of the change in the position of the grinding robot end, the change in the rotation angle of the grinding robot end, and the change in the position of the mobile unit moving along the axis of the cylinder.

[0011] Further preferably, the optimization goal of the multi-task reinforcement learning optimization model of the process of the grinding head moving toward the starting point of the weld is to avoid collisions between the connecting rods of the grinding robot itself and with the relay cylinder, and to ensure that the grinding head approaches the starting point of the weld.

[0012] Further preferably, the action space of the multi-task reinforcement learning optimization model of the movement process of the grinding head after reaching the starting point is composed of the change in the rotation angle of the end of the grinding robot and the change in the position of the mobile unit moving along the axis direction of the cylinder body.

[0013] Further preferably, the optimization goal of the multi-task reinforcement learning optimization model of the movement process of the grinding head after reaching the starting point is that the grinding head at the end of the grinding robot can reach every part of the weld and the average end stiffness of the grinding is as large as possible.

[0014] Further preferably, the relationship between the reward functions of the multi-task reinforcement learning optimization model of the two processes is as follows:

[0015] reward=α1·r1+α2·r2+α3·r3

[0016] Among them, r1 is the reward for approaching the target point, r2 is the reward for collision, and r3 is the reward for reachability and terminal evaluation stiffness.

[0017] Further preferably, r1, r2 and r3 are in accordance with the following relationship:

[0018]

[0019] Where l1 is the distance between the grinding head and the grinding starting point before the action is performed, l2 is the distance between the grinding head and the grinding starting point after the action is performed, and d t is the distance threshold for determining when the grinding head reaches the target point, R is the radius of the servomotor cylinder, is the distance between joint i and the cylinder axis, ||d 磨盘 || is the distance between the center of the grinding disc and the axis of the cylinder, ||d r || is the distance between the joint that is prone to self-interference and the robot body, d ε , l s is the safety distance threshold, K N It is the average stiffness of the end of the robot arm when the position of the mobile unit and the rotation angle of the grinding disc around the axis are fixed.

[0020] Further preferably, the reward coefficients α1, α2, α3 of the two processes are different.

[0021] According to another aspect of the present invention, a multi-objective optimization system for robot grinding of inner wall welds of a servomotor cylinder body based on multi-task reinforcement learning is provided, the system comprising an actuator for executing the above-mentioned multi-objective optimization method for robot grinding of inner wall welds of a servomotor cylinder body based on multi-task reinforcement learning.

[0022] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0023] 1. The present invention divides the process of robot welding seam grinding into two different processes, distinguishes the shared knowledge of the two processes and their own unique knowledge, constructs multi-task reinforcement learning optimization models for different processes, builds a multi-task reinforcement learning framework, and improves the training speed, generalization and control effect;

[0024] 2. The present invention constructs respective inputs and outputs for different processes, separates the outputs and reward functions of the two, and optimizes different processes for different main optimization objectives, which can obtain better optimization effects;

[0025] 3. The present invention adopts a reinforcement learning method to control the movement of the end to the starting point of grinding, while considering avoiding collisions, accessibility, and optimizing the average stiffness of grinding. The state, action space, and reward function of the task are designed. There is no need to establish an accurate model of the system, nor does it require expert knowledge. It learns and discovers the optimal behavior through continuous interaction with the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a multi-objective optimization method for robot grinding of inner wall weld seams of a servomotor cylinder body based on multi-task reinforcement learning constructed according to a preferred embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of a MuJoCo simulation experiment platform constructed according to a preferred embodiment of the present invention;

[0028] Figure 3 It is a case where a robot arm collides with itself in a problem considered in reinforcement learning constructed according to a preferred embodiment of the present invention, wherein (a) the robot arm collides with itself, and (b) the robot arm collides with a cylinder;

[0029] Figure 4 It is a situation in which the entire weld cannot be polished in the problem considered by the reinforcement learning constructed according to the preferred embodiment of the present invention, wherein (a) the trolley position is too far and the far point cannot be polished, and (b) the trolley position is too close and the near point cannot be polished;

[0030] Figure 5 It is a reinforcement learning action network structure constructed according to the preferred embodiment of the present invention;

[0031] Figure 6 It is a reinforcement learning evaluation network structure constructed according to the preferred embodiment of the present invention;

[0032] Figure 7 is a reinforcement learning training flow chart constructed according to a preferred embodiment of the present invention;

[0033] Figure 8 It is a reward function variation curve constructed according to the preferred embodiment of the present invention for executing 8 million action steps training;

[0034] Fig. 9 It is a reinforcement learning optimization effect test diagram constructed according to the preferred embodiment of the present invention, wherein (a) the distance between each joint and the tube wall, (b) the distance between the end and the starting point, and (c) the change of the stiffness index during the adjustment process. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0036] like Figure 1As shown in the figure, a multi-objective optimization method for robot grinding of the inner wall weld of a servomotor cylinder based on multi-task reinforcement learning is presented. The method specifically includes the following steps:

[0037] S1 is based on the MuJoCo simulation platform. Figure 2 The environment for robot grinding of the weld seam on the inner wall of the servomotor cylinder shown in the figure. The mobile unit (in this embodiment, the mobile unit is a Husky trolley equipped with a UR16e robot arm), the servomotor cylinder, and the three-dimensional model of the weld seam, the inner diameter d of the cylinder is 90 cm, and a 3D camera is added. The surface appearance of the inner wall of the servomotor cylinder is set to be similar to the real situation. The weld seam extends along the axial direction of the cylinder (x-axis). The distance between the trolley and the servomotor cylinder in the y-direction is constant. The trolley moves along the x-axis direction. The trolley position, weld seam position, and length are randomized within a certain range each time the environment is initialized.

[0038] To grind the weld, it is necessary to know the position of the weld relative to the robot base. Based on the 3D camera in the simulation environment, a deep learning vision algorithm is used to identify and segment the weld, and the position of the weld in the robot base coordinate system is calculated.

[0039] S21 collects weld images of different shapes and located on different workpieces in MuJoCo from different angles, and performs data enhancement by rotating, filtering, adding noise, and changing brightness on the collected images, obtaining a total of 3789 images. The labeling software anylabeling is used to mark the weld positions and train the segmentation model Mask R-CNN.

[0040] After identifying the position of the weld in the image, the position of the weld in the robot base coordinate system is calculated according to the following formula:

[0041]

[0042] in, base X, base Y, base Z is the position of the target point in the robot base coordinate system; u, v are the pixel coordinates of the target point; cam Z is the depth of the target point; K is the camera internal parameter matrix which is related to the camera focal length and viewpoint center and can be determined according to the camera parameters; is the homogeneous transformation matrix from the robot base coordinate system to the camera coordinate system, which can be calculated by obtaining the object pose in the simulation environment.

[0043] S2 analyzes the tasks and divides them into the form of multi-task reinforcement learning. When controlling the end to move toward the grinding starting point, the robot must be prevented from colliding; when the end reaches the grinding starting point, the end position remains unchanged, and the average grinding stiffness is optimized. The robot must be prevented from colliding and the entire weld can be polished. Therefore, for the two processes of the end moving toward the starting point and the optimization adjustment after the end reaches the starting point, avoiding robot collisions is used as shared knowledge, and different strategies are used for control to create experience pools for the two strategies. When the experience pool of the strategy is full, the action-evaluation network of the corresponding strategy is updated according to the PPO algorithm, and the experience pool is cleared.

[0044] According to the characteristics of the task, the Markov process elements of the two processes of the end moving to the starting point and the optimization adjustment after the end reaches the starting point are set. The axis vector direction of the robot end is kept in the direction during grinding, and reinforcement learning is used to control the movement of the trolley, the end position movement of the robot arm, and the rotation movement around the axis vector.

[0045] (1) State space

[0046] Analyze the influencing factors of the optimal control of the car and robotic arm motion, and set the state space (input), action space (output) and reward function of the Markov process.

[0047] The state space is set up as:

[0048] Among them, q is the size of the robot's six joint angles; p base_link is the spatial position of the robot base mounted on the vehicle; is the length vector of the weld to be polished, the cylinder axis is the x-axis, is the position vector between the end and the target starting point, θ t is the rotation angle of the end relative to the initial posture, and the initial posture is It is the end normal direction when grinding at a certain inclination angle.

[0049] (2) Action Space

[0050] The main purpose of the end moving to the starting point is to make the end quickly reach the grinding starting point without collision. The action space is set as: a = {Δdp, Δθ, Δx}

[0051] Where Δdp = {Δx, Δy, Δz} is the change in the position of the robot end, Δθ is the change in the rotation angle around the end axis, and Δx is the change in the position in the direction of motion of the robot.

[0052] The optimization adjustment process after the end reaches the starting point is mainly aimed at adjusting the position of the trolley and the redundant angle of the end so that the end can reach the entire weld without collision and optimize the average grinding stiffness. The action space is set as: a = {Δθ, Δx}

[0053] Among them, Δθ is the change in the rotation angle around the terminal axis, and Δx is the change in the position of the car in the direction of motion.

[0054] (3) Reward Function

[0055] During the process of the end moving to the starting point, the reward function includes the reward for the end approaching the target point and the reward for the robot colliding. During the optimization and adjustment process after the end reaches the starting point, the reward function includes the reward for grinding average stiffness and accessibility, and the reward for the robot colliding. The specific reward function design criteria are:

[0056] The reward for the terminal to approach the target point: the closer the terminal is to the target point, the greater the reward value. If the terminal is far away from the target point, a penalty will be given. The reward for the terminal to approach the target point is set to:

[0057]

[0058] Where l1 is the distance between the end and the starting point of grinding before the action is performed, l2 is the distance between the end and the starting point of grinding after the action is performed, and d t It is the distance threshold for determining whether the end point reaches the target point.

[0059] Robot collision reward: Figure 3 As shown in the figure, the robot collision is divided into the robot colliding with the relay cylinder and the robot interfering with itself. At the same time, the robot is out of the range of motion. The robot collision reward is set as:

[0060]

[0061] Where R is the radius of the servo cylinder, is the distance between joint i and the cylinder axis, ||d 磨盘 || is the distance between the center of the grinding disc and the axis of the cylinder body, ||d r || is the distance between the joint that is prone to self-interference and the robot body, d ε , l s is the safety distance threshold. The situations where the joints are out of the range of motion and the car is out of the range of motion are also considered.

[0062] Reachability and terminal average stiffness bonus: e.g. Figure 4As shown in the figure, when the inverse kinematics solution of a point on the weld does not exist or is discontinuous, the entire weld cannot be ground. Within 60 adjustment steps, the average end stiffness of the entire weld is optimized, and the accessibility and end average stiffness bonus are set as:

[0063]

[0064] Among them, K N The average stiffness of the end of the robot arm at each point when the trolley position is fixed and the grinding wheel rotation angle is fixed to grind the entire weld.

[0065] The reward function is set as: reward = α1·r1+α2·r2+α3·r3, where α1, α2, and α3 are the coefficients of the three rewards.

[0066] S42 carried out the robot joint stiffness identification experiment, using a laser tracker to measure the target ball to record the deformation under different forces. The joint stiffness values ​​obtained by least squares fitting are shown in Table 1, where J p is the Jacobian matrix, K q is the joint stiffness matrix.

[0067] Table 1

[0068]

[0069] The end stiffness value is calculated according to the following relationship:

[0070]

[0071] Two task process action-evaluation networks are set up, and the action networks and evaluation networks of the two processes realize knowledge sharing by sharing some network layers.

[0072] Action network settings such as Figure 5 As shown, the shared part is a fully connected layer h1 with a size of 256 and a fully connected layer h2 with a size of 128. The fully connected layer size of the approach process is 5, and the fully connected layer size of the optimization adjustment process is 2. The fully connected layers all use the tanh function as the activation function.

[0073] The evaluation network settings are as follows Figure 6 As shown, the shared parts are a fully connected layer h1 of size 256 and a fully connected layer h2 of size 128. Both h1 and h2 use the tanh function as the activation function, and the size of the fully connected layer in the approximation process and the optimization adjustment process is 1.

[0074] S3 Press Figure 7 The process shown in the figure is to train the reinforcement learning network. The reward value changes during the training process are as follows Figure 8 shown.

[0075] A 60cm long weld was tested, and the distance between each joint and the obstacle, the distance between the end and the grinding starting point, and the change curve of the average grinding stiffness were shown in the following figure: Fig. 9 Each joint always keeps a safe distance from obstacles, and the end steadily approaches the grinding starting point. During the stiffness optimization and adjustment process, the average grinding stiffness is continuously improved and optimized.

[0076] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-objective optimization method for robot grinding of inner wall welds of a servomotor cylinder based on multi-task reinforcement learning, characterized in that: The method comprises the following steps: A simulation model of a grinding robot grinding a servo cylinder is constructed, and the grinding process is divided into the process of the grinding head moving toward the starting point of the weld and the movement process after the grinding head reaches the starting point. A multi-task reinforcement learning optimization model is constructed for each of the two processes. The multi-task reinforcement learning optimization model takes the state space as input and the action space as output. The length and position of the weld are changed, and the grinding simulation is performed using the simulation model. The data of the two processes obtained from each simulation are used to train the respective multi-task reinforcement learning optimization models until each of the multi-task reinforcement learning optimization models reaches a preset number of training times and the reward function of the optimization objective converges.

2. A multi-objective optimization method for robot grinding of inner wall welds of a servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 1, characterized in that: The state space corresponding to the multi-task reinforcement learning optimization model of the two processes is the same, which is composed of the joint angles of the grinding robot, the spatial coordinates of the grinding robot base, the length of the weld, the position vector between the end of the grinding robot and the starting point of the weld, and the rotation angle of the end of the grinding robot relative to its initial posture.

3. The multi-objective optimization method for robot grinding of inner wall weld of servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 1, characterized in that: The action space of the multi-task reinforcement learning optimization model of the process of the grinding head moving toward the starting point of the weld is composed of the change in the position of the grinding robot end, the change in the rotation angle of the grinding robot end, and the change in the position of the mobile unit moving along the axis of the cylinder.

4. A multi-objective optimization method for robot grinding of inner wall weld of servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 3, characterized in that: The optimization goal of the multi-task reinforcement learning optimization model of the process in which the grinding head moves toward the starting point of the weld is to avoid collisions between the connecting rods of the grinding robot itself and with the servomotor cylinder, and to ensure that the grinding head approaches the starting point of the weld.

5. The multi-objective optimization method for robot grinding of inner wall weld of servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 3, characterized in that: The action space of the multi-task reinforcement learning optimization model of the movement process of the grinding head after reaching the starting point is composed of the change in the rotation angle of the end of the grinding robot and the change in the position of the mobile unit moving along the axis direction of the cylinder body.

6. A multi-objective optimization method for robot grinding of inner wall welds of a servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 5, characterized in that: The optimization goal of the multi-task reinforcement learning optimization model of the movement process after the grinding head reaches the starting point is to enable the grinding head at the end of the grinding robot to reach every part of the weld and to make the average end stiffness of the grinding as large as possible.

7. A multi-objective optimization method for robot grinding of inner wall welds of a servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 1 or 2, characterized in that: The relationship between the reward functions of the multi-task reinforcement learning optimization model of the two processes is as follows: reward=α1·r1+α2·r2+α3·r3 Among them, r1 is the reward for approaching the target point, r2 is the reward for collision, and r3 is the reward for reachability and terminal evaluation stiffness.

8. The multi-objective optimization method for robot grinding of inner wall weld of servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 7, characterized in that: The r1, r2 and r3 are carried out according to the following relationship: Where l1 is the distance between the grinding head and the grinding starting point before the action is performed, l2 is the distance between the grinding head and the grinding starting point after the action is performed, and d t is the distance threshold for determining when the grinding head reaches the target point, R is the radius of the servomotor cylinder, is the distance between joint i and the cylinder axis, ||d 磨盘 || is the distance between the center of the grinding disc and the axis of the cylinder, ||d r || is the distance between the joint that is prone to self-interference and the robot body, d ε , l s is the safety distance threshold, K N It is the average stiffness of the end of the robot arm when the position of the mobile unit and the rotation angle of the grinding disc around the axis are fixed.

9. A multi-objective optimization method for robot grinding of inner wall weld seams of a servomotor cylinder body based on multi-task reinforcement learning as claimed in claim 7 or 8, characterized in that: The reward coefficients α1, α2, α3 of the two processes are different.

10. A multi-objective optimization system for robot grinding of inner wall welds of a servomotor cylinder based on multi-task reinforcement learning, characterized in that: The system includes an actuator, which is used to execute the multi-objective optimization method for robot grinding of the inner wall weld of a relay cylinder body based on multi-task reinforcement learning as described in any one of claims 1 to 9.

Images