A multi-objective optimization method and system for a relay cylinder inner wall welding seam robot based on multi-task reinforcement learning

Abstract

The present application belongs to the technical field of polishing robots, and discloses a polishing multi-objective optimization method and system for a servomotor cylinder inner wall weld seam robot based on multi-task reinforcement learning. The method comprises the following steps: constructing a simulation model of a mobile unit, a polishing robot and a servomotor cylinder; dividing the polishing process into a process of the polishing head moving towards the starting point of the weld seam and a process of the polishing head moving after reaching the starting point; constructing a respective multi-task reinforcement learning optimization model for each of the two processes, with the state space as the input and the action space as the output; changing the length and position of the weld seam for polishing simulation, training the multi-task reinforcement learning optimization models of the two processes until a preset number of training times is reached and the reward function of the optimization target converges. Through the present application, the problem of being unable to simultaneously consider multi-objective optimization such as collision avoidance, polishing accessibility and maximum stiffness maximization during the robot polishing process is solved.

Description

Technical Field

[0001] This invention belongs to the technical field of grinding robots, and more specifically, relates to a multi-objective optimization method and system for grinding weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning. Background Technology

[0002] The relay is a component in a hydroelectric generator that converts hydraulic energy into the mechanical energy required for the transmission mechanism. During operation, due to reasons such as sediment deposition, the cylinder of the relay will scrape against the piston inside it, which moves axially and reciprocates, resulting in axial scratches at the bottom. Leakage caused by the scratches will affect its normal operation. Repair work such as inspection, regular planing, welding, milling, and grinding is required to repair the scratches.

[0003] After the axial scratches are regularly removed, repair welding is performed, with the weld extending axially and appearing as a regular elongated strip. The weld needs to be ground to improve the smoothness and shape accuracy of the repaired cylinder inner wall, and to remove welding slag and other welding residues. A UGV carriage equipped with a robotic arm is used to grind the weld. When the end grinding disc approaches the grinding start point, the carriage moves with the robotic arm; once the end grinding disc reaches the grinding start point, the carriage stops, and the robotic arm completes the weld grinding work independently.

[0004] Because the weld seam inside the cylinder is relatively long, if the position of the cart during grinding and the redundant angle around the end effector are not properly adjusted, it will result in the inability to grind the entire weld seam and weak robot rigidity. Furthermore, during motion control, collisions between the robotic arm's links and with the cylinder must be avoided. Traditional decision-making and control methods cannot simultaneously consider collision avoidance, grinding accessibility, and rigidity optimization. Modern artificial intelligence technology can be used for decision-making and control with multiple optimization objectives. Therefore, multi-task reinforcement learning is employed to control the movement of the cart and robotic arm as the end effector approaches the starting point and undergoes optimized adjustments. Summary of the Invention

[0005] To address the above-mentioned deficiencies or improvement needs of existing technologies, this invention provides a multi-objective optimization method and system for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning, which solves the problem that the robot cannot simultaneously consider multiple objectives such as collision avoidance, grinding accessibility, and stiffness maximization during the grinding process.

[0006] To achieve the above objectives, according to one aspect of the present invention, a multi-objective optimization method for grinding weld seams on the inner wall of a relay cylinder using a robot based on multi-task reinforcement learning is provided. This method includes the following steps:

[0007] A simulation model of the grinding relay cylinder of the grinding robot is constructed, and the grinding process is divided into the process of the grinding head moving towards the starting point of the weld and the process of the grinding head moving after reaching the starting point. For the two processes, a multi-task reinforcement learning optimization model is constructed, which takes the state space as input and the action space as output.

[0008] The length and position of the weld are changed, and grinding simulation is performed using the simulation model. The data from the two processes obtained in each simulation are used to train the respective multi-task reinforcement learning optimization model until each multi-task reinforcement learning optimization model reaches the preset number of training times and the reward function of the optimization target converges.

[0009] More preferably, the state spaces corresponding to the multi-task reinforcement learning optimization models of the two processes are the same. The state space consists of the joint angle of the grinding robot, the spatial coordinates of the grinding robot base, the length of the weld, the position vector between the end of the grinding robot and the start of the weld, and the rotation angle of the end of the grinding robot relative to its initial posture.

[0010] More preferably, the action space of the multi-task reinforcement learning optimization model for the process of the grinding head moving towards the weld start point consists of the change in the end position of the grinding robot, the change in the rotation angle of the end position of the grinding robot, and the change in the position of the moving unit moving along the cylinder axis.

[0011] More preferably, the optimization objective of the multi-task reinforcement learning optimization model for the process of the grinding head moving towards the weld start point is to avoid collisions between the links of the grinding robot itself and with the cylinder of the relay, and to ensure that the grinding head moves closer to the weld start point.

[0012] More preferably, the action space of the multi-task reinforcement learning optimization model for the motion process of the grinding head after reaching the starting point consists of the change in the rotation angle of the end of the grinding robot and the change in the position of the moving unit along the cylinder axis.

[0013] More preferably, the optimization objective of the multi-task reinforcement learning optimization model for the motion process of the grinding head after reaching the starting point is to ensure that the grinding head at the end of the grinding robot can reach every part of the weld and that the average end stiffness of the grinding is as large as possible.

[0014] More preferably, the relationship between the reward functions of the multi-task reinforcement learning optimization model for the two processes is as follows:

[0015] reward=α1·r1+α2·r2+α3·r3

[0016] Where r1 is the reward for approaching the target point, r2 is the reward for a collision, and r3 is the reward for reachability and end-point evaluation stiffness.

[0017] More preferably, r1, r2, and r3 are defined according to the following relationship:

[0018]

[0019] Where l1 is the distance between the grinding head and the grinding starting point before the action is performed, l2 is the distance between the grinding head and the grinding starting point after the action is performed, and d t This is the threshold distance for determining when the grinding head reaches the target point, where R is the radius of the relay cylinder. It is the distance between joint i and the cylinder axis, ||d 磨盘 || is the distance between the center of the grinding disc and the axis of the cylinder, ||d r || is the distance between the joints prone to self-interference and the robot body, d ε l s K is the safe distance threshold. N It is the average value of the end effector stiffness of the robotic arm when the position of the moving unit and the rotation angle of the grinding disc around the axis are fixed.

[0020] More preferably, the reward coefficients α1, α2, and α3 of the two processes are different.

[0021] According to another aspect of the present invention, a multi-objective optimization system for grinding weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning is provided. The system includes an actuator for executing the multi-objective optimization method for grinding weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning described above.

[0022] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0023] 1. This invention divides the process of robot grinding welds into two different processes, distinguishes the shared knowledge of the two processes as well as their unique knowledge, and constructs multi-task reinforcement learning optimization models for each of the different processes, thus building a multi-task reinforcement learning framework and improving training speed, generalization and control effect.

[0024] 2. This invention constructs separate inputs and outputs for different processes, distinguishes between their outputs and reward functions, and optimizes different processes for different main optimization objectives, thereby achieving better optimization results;

[0025] 3. This invention uses reinforcement learning to control the movement of the end effector toward the grinding starting point, while considering collision avoidance, accessibility, and optimization of the average grinding stiffness. It designs the task's state, action space, and reward function. It does not require an accurate model of the system or expert knowledge, but learns and discovers the optimal behavior through continuous interaction with the environment. Attached Figure Description

[0026] Figure 1 This is a flowchart of a multi-objective optimization method for grinding weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning, constructed according to a preferred embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the MuJoCo simulation experimental platform constructed according to a preferred embodiment of the present invention;

[0028] Figure 3 The reinforcement learning construct according to the preferred embodiment of the present invention considers the situation of a robot arm colliding with a problem, wherein (a) the robot arm collides with itself, and (b) the robot arm collides with a cylinder.

[0029] Figure 4 The reinforcement learning constructed according to the preferred embodiment of the present invention considers the situation in which the entire weld seam cannot be ground, wherein (a) the trolley position is too far away and the far point cannot be ground, and (b) the trolley position is too close and the near point cannot be ground;

[0030] Figure 5 This is a reinforcement learning action network structure constructed according to a preferred embodiment of the present invention;

[0031] Figure 6 This is a reinforcement learning evaluation network structure constructed according to a preferred embodiment of the present invention;

[0032] Figure 7 This is a flowchart of reinforcement learning training constructed according to a preferred embodiment of the present invention;

[0033] Figure 8 The reward function variation curve for training with 8 million action steps is constructed according to a preferred embodiment of the present invention.

[0034] Figure 9 The diagram shows the reinforcement learning optimization effect test constructed according to a preferred embodiment of the present invention, wherein (a) the distance between each joint and the pipe wall, (b) the distance between the end point and the starting point, and (c) the change of stiffness index during the adjustment process. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0036] like Figure 1As shown, a multi-objective optimization method for grinding weld seams on the inner wall of a relay cylinder using a robot based on multi-task reinforcement learning is presented. The specific steps include:

[0037] S1 is based on the MuJoCo simulation platform and can be built as follows: Figure 2 The environment shown depicts the robotic grinding of the weld seam on the inner wall of the relay cylinder. A 3D model of the moving unit (in this embodiment, the moving unit is a Husky cart equipped with an UR16e robotic arm), the relay cylinder, and the weld seam is used. The inner diameter d of the cylinder is 90cm, and a 3D camera is added. The surface appearance of the inner wall of the relay cylinder is set to be similar to the real situation. The weld seam extends along the cylinder axis (x-axis). The distance between the cart and the relay cylinder in the y-direction is constant. The cart moves along the x-axis. The cart position, weld seam position, and length are randomized within a certain range each time the environment is initialized.

[0038] To grind the weld, it is necessary to know the position of the weld relative to the robot base. Based on the 3D camera in the simulation environment, a deep learning vision algorithm is used to identify and segment the weld, and calculate the position of the weld in the robot base coordinate system.

[0039] S21 acquired images of welds of different shapes and locations on different workpieces from various angles in MuJoCo. The acquired images were then subjected to data augmentation operations including rotation, filtering, noise addition, and brightness adjustment, resulting in a total of 3789 images. The weld locations were labeled using the labeling software AnyLabeling, and the Mask R-CNN segmentation model was trained on these images.

[0040] After identifying the location of the weld in the image, its position in the robot's base coordinate system is calculated using the following formula:

[0041]

[0042] in, base X, base Y, base Z represents the position of the target point in the robot's base coordinate system; u and v represent the pixel coordinates of the target point. cam Z represents the depth of the target point; K is the camera intrinsic parameter matrix, which is related to the camera focal length and the center of view, and can be determined based on the camera parameters. This is the homogeneous transformation matrix from the robot's base coordinate system to the camera coordinate system, which can be calculated from the object poses obtained in the simulation environment.

[0043] S2 analyzes the task and divides it into a multi-task reinforcement learning approach. During the control of the end effector's movement towards the grinding starting point, collisions must be avoided. Once the end effector reaches the grinding starting point, its position remains unchanged, and the average grinding stiffness is optimized, again aiming to avoid collisions and allow for grinding of the entire weld seam. Therefore, avoiding collisions is used as shared knowledge for both the end effector's movement towards the starting point and the optimization adjustment after reaching the starting point. Different strategies are used for control, creating two experience pools for each strategy. When a strategy's experience pool is full, the corresponding action-evaluation network is updated according to the PPO algorithm, and the experience pool is cleared.

[0044] Based on the characteristics of the task, Markov process elements are set for the two processes: the end effector's movement towards the starting point and the optimization adjustment after the end effector reaches the starting point. The robot's end effector axis vector direction is maintained in the direction during grinding, and reinforcement learning is used to control the movement of the cart, the position movement of the robotic arm's end effector, and the rotational movement around the axis vector.

[0045] (1) State space

[0046] The influencing factors of the motion optimization control of the trolley and the robotic arm are analyzed, and the state space (input), motion space (output), and reward function of the Markov process are set.

[0047] The state space is set as follows:

[0048] Where q represents the angle of the robot's six joints; p base_link The spatial location of the robot base mounted on the cart; The x-axis represents the length vector of the weld seam to be ground, with the cylinder block axis as the x-axis. The position vectors of the end point and the starting point of the target are given. θ t The angle of rotation about the axis of the end effector relative to the initial attitude is . This refers to the end normal when grinding at a certain angle.

[0049] (2) Action space

[0050] The primary objective of the end-effector's movement towards the starting point is to quickly reach the grinding starting point without collision. The motion space is defined as: a = {Δdp, Δθ, Δx}

[0051] Where Δdp={Δx,Δy,Δz} is the change in position of the robot's end effector, Δθ is the change in rotation angle around the end effector axis, and Δx is the change in position in the direction of the vehicle's motion.

[0052] The optimization and adjustment process after the end reaches the starting point mainly aims to ensure that the end can reach the entire weld seam without collision by adjusting the trolley position and the end redundancy angle, and to optimize the average stiffness of grinding. The motion space is set as: a={Δθ,Δx}

[0053] Where Δθ is the change in rotation angle around the end axis, and Δx is the change in position of the trolley in the direction of motion.

[0054] (3) Reward function

[0055] The reward function for the end effector's movement towards the starting point includes rewards for approaching the target point and for collisions. The reward function for the optimization and adjustment process after the end effector reaches the starting point includes rewards for improving average stiffness and accessibility, and for collisions. The specific reward function design criteria are as follows:

[0056] Reward for the end point moving closer to the target point: The closer the end point is to the target point, the greater the reward value. If the end point moves away from the target point, a penalty is imposed. The reward for the end point moving closer to the target point is set as follows:

[0057]

[0058] Where l1 is the distance between the end point of the action before execution and the starting point of grinding, l2 is the distance between the end point of the action after execution and the starting point of grinding, and d t To determine the distance threshold from the endpoint to the target point.

[0059] Robot collision reward: such as Figure 3 As shown, robot collisions are categorized into collisions between the robot and the relay cylinder, and interference with the robot itself. The situation where the robot exceeds its range of motion is also considered. Therefore, the collision reward is set as follows:

[0060]

[0061] Where R is the radius of the relay cylinder. Let |d| be the distance between joint i and the cylinder axis. 磨盘 || is the distance between the center of the grinding disc and the axis of the cylinder, ||d r || represents the distance between the joints prone to self-interference and the robot body, d ε l s This is a safe distance threshold. It also considers situations where joints or the vehicle exceeds their range of motion.

[0062] Accessibility and end-effector mean stiffness bonus: such as Figure 4As shown, when the inverse kinematic solution at a point on the weld is nonexistent or discontinuous, grinding of the entire weld is not permitted. Within 60 adjustment steps, the average end stiffness of grinding the entire weld is optimized. The reachability and end average stiffness reward are then set as follows:

[0063]

[0064] Among them, K N The average value of the end stiffness of the robotic arm at each point when grinding the entire weld seam with the trolley position fixed and the grinding disc rotation angle around the axis fixed.

[0065] The reward function is set as: reward=α1·r1+α2·r2+α3·r3, where α1, α2, and α3 are the coefficients of the three rewards.

[0066] S42 conducted a robot joint stiffness identification experiment, using a laser tracker to measure the deformation of a target ball under different forces, and then... The stiffness values ​​of each joint were obtained by fitting using the least squares method, as shown in Table 1, where J p Let K be a Jacobian matrix. q Here is the joint stiffness matrix.

[0067] Table 1

[0068]

[0069] The end stiffness value is calculated according to the following formula:

[0070]

[0071] The action-evaluation network for two tasks is set up, and the action network and evaluation network of the two processes achieve knowledge sharing by sharing some network layers.

[0072] Action network settings such as Figure 5 As shown, the shared part consists of a fully connected layer h1 of size 256 and a fully connected layer h2 of size 128. The fully connected layer size in the approximation process is 5, and the fully connected layer size in the optimization process is 2. All fully connected layers use the tanh function as the activation function.

[0073] Evaluate network settings such as Figure 6 As shown, the shared part consists of a fully connected layer h1 of size 256 and a fully connected layer h2 of size 128. Both h1 and h2 use the tanh function as the activation function. The size of the fully connected layer in the approach process and the optimization process is 1.

[0074] S3 press Figure 7 The process shown is used to train the reinforcement learning network. The reward value changes during the training process as follows: Figure 8 As shown.

[0075] Tests were conducted on a 60cm long weld, controlling the changes in distances between each joint and obstacle, the distance between the end point and the grinding start point, and the average grinding stiffness as shown in the curves. Figure 9 As shown, each joint maintains a safe distance from the obstacle at all times, and the end effector steadily approaches the grinding starting point. During the stiffness optimization and adjustment process, the average grinding stiffness is continuously improved and optimized.

[0076] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-objective optimization method for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning, characterized in that, The method includes the following steps: A simulation model of the grinding relay cylinder of the grinding robot is constructed, and the grinding process is divided into the process of the grinding head moving towards the starting point of the weld and the process of the grinding head moving after reaching the starting point. For the two processes, a multi-task reinforcement learning optimization model is constructed, which takes the state space as input and the action space as output. The length and position of the weld are changed, and the grinding simulation is performed using the simulation model. The data from the two processes obtained in each simulation are used to train the respective multi-task reinforcement learning optimization model until each multi-task reinforcement learning optimization model reaches the preset number of training times and the reward function of the optimization target converges. The action space of the multi-task reinforcement learning optimization model for the process of the grinding head moving towards the weld start point consists of the change in the end position of the grinding robot, the change in the rotation angle of the end position of the grinding robot, and the change in the position of the moving unit moving along the cylinder axis. The optimization objective of the multi-task reinforcement learning optimization model for the process of the grinding head moving towards the weld start point is to avoid collisions between the links of the grinding robot itself and with the cylinder of the relay, and to ensure that the grinding head moves closer to the weld start point. The action space of the multi-task reinforcement learning optimization model for the motion process of the grinding head after reaching the starting point consists of the change in the rotation angle of the end of the grinding robot and the change in the position of the moving unit along the cylinder axis. The optimization objective of the multi-task reinforcement learning optimization model for the motion process of the grinding head after reaching the starting point is that the grinding head at the end of the grinding robot can reach every part of the weld.

2. The multi-objective optimization method for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning as described in claim 1, characterized in that, The state spaces corresponding to the multi-task reinforcement learning optimization models of the two processes are the same. The state space consists of the joint angles of the grinding robot, the spatial coordinates of the grinding robot base, the length of the weld, the position vector between the end of the grinding robot and the start of the weld, and the rotation angle of the end of the grinding robot relative to its initial posture.

3. A multi-objective optimization method for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning, as described in claim 1 or 2, characterized in that... The relationship between the reward functions of the multi-task reinforcement learning optimization model in the two processes is as follows: in, Rewards for getting closer to the target point. As a reward for a collision, Rewards are given for accessibility and end-effector stiffness.

4. The multi-objective optimization method for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning as described in claim 3, characterized in that, The , and Perform according to the following formula: in, It is the distance between the grinding head and the starting point of the grinding process before the action is performed. It is the distance between the grinding head and the starting point of the grinding process after the action is performed. It is the threshold for determining the distance from the grinding head to the target point. It is the radius of the relay cylinder. It is the distance between joint i and the cylinder axis. It is the distance between the center of the grinding disc and the axis of the cylinder. It refers to the distance between joints prone to self-interference and the robot body. , For safe distance threshold, It is the average value of the end effector stiffness of the robotic arm when the position of the moving unit and the rotation angle of the grinding disc around the axis are fixed.

5. The multi-objective optimization method for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning as described in claim 3, characterized in that, The reward coefficients of the two processes different.

6. A multi-objective optimization system for robotic grinding of weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning, characterized in that, The system includes an actuator for performing a multi-objective optimization method for grinding weld seams on the inner wall of a relay cylinder based on multi-task reinforcement learning, as described in any one of claims 1-5.

Images