Online position correction method for contact network maintenance mechanical arm based on SAC algorithm

Abstract

The invention relates to a SAC algorithm-based online position correction method for an overhead line system maintenance mechanical arm, and belongs to the technical field of overhead line system maintenance. The invention discloses a method for correcting the online position of a contact network maintenance mechanical arm based on an SAC algorithm. The method comprises the steps that a training environment is built based on a Gazebo simulation platform; recognizing a target bolt based on a depth camera carrying a YOLOV5s algorithm, and converting the target bolt into a three-dimensional coordinate taking the camera as a coordinate origin; training the mechanical arm by adopting an SAC algorithm in the training environment; on the basis of SAC output actions, the mechanical arm is controlled to move through a Moveit frame to conduct environment interaction with the environment, and experience sampling and network parameters are updated. The terminal point of the tail end of the mechanical arm is updated based on the target point detected by the depth camera, and low difference of each planned path is kept, so that the robustness of a mechanical arm path planning system and the accuracy of the tail end of the mechanical arm are improved, and a better camera view is provided for subsequent maintenance operation.

Description

Technical Field

[0001] The invention relates to an online position correction method for a contact network maintenance mechanical arm based on a SAC algorithm, and belongs to the technical field of contact network maintenance. Background Art

[0002] In the railway system, the maintenance of the contact network currently relies mainly on manual work, which has safety risks, low work efficiency, high costs and other problems. With the development of robot technology, the robot arm will play a more important role in the maintenance of the contact network arm along the railway. In traditional industrial application environments, the path planning of the robot arm mostly adopts offline path planning with manual teaching, rather than online planning. The path of the robot arm is planned by manual teaching, so that the robot arm can perform a certain task repeatedly. Compared with the fixed environment application of manual teaching, there are maintenance vehicle parking errors and work platform control errors in the environmental background of the robot arm for maintenance of the contact network arm. The superposition of the two errors causes the offline path to be unable to accurately reach the pre-planned target point, and the depth camera carried at the end of the maintenance robot arm cannot normally shoot the bolt target or the shooting angle is not good, making it impossible for subsequent maintenance operations to be carried out normally.

[0003] In summary, due to the poor robustness of the offline planning system, it is not fully suitable for the environmental background of the robotic arm repairing the overhead contact network wrist arm. Summary of the invention

[0004] In order to overcome the defects existing in the prior art, the present invention aims to provide an online position correction method for a contact network maintenance robot arm based on the SAC algorithm. The method updates the end point of the robot arm based on the depth camera detection target point and maintains low variability in each planned path. It aims to improve the robustness of the robot arm path planning system and the accuracy of the robot arm end, and provide a better camera field of view for subsequent maintenance operations.

[0005] The technical solution provided by the present invention to solve the above technical problems is: a method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm, comprising the following steps:

[0006] Step 1: Build a training environment based on the Gazebo simulation platform;

[0007] Step 2: Identify the target bolts in the training environment based on the depth camera equipped with the YOLOV5s algorithm and convert them into three-dimensional coordinates with the camera as the coordinate origin;

[0008] Step 3: Use the SAC algorithm to train the robot arm in the training environment to obtain the SAC output action;

[0009] Step 4: Based on the SAC output action, the Moveit framework is used to control the movement of the robot arm and interact with the environment, and the network parameters of the training environment are updated;

[0010] Step 5: Repeat steps 1 to 4, and continue to repeat the process of environment interaction, experience sampling, and network parameter update until the strategy converges or reaches the specified training round.

[0011] A further technical solution is that the training environment in step 1 includes a three-dimensional model of the contact network, a robotic arm, a depth camera, and a simulation platform.

[0012] A further technical solution is that the state space S of the training environment in step 1 is:

[0013]

[0014] Where: θ is the angle value of each joint of the current robotic arm, v is the speed of each joint of the current robotic arm, P is the Euclidean coordinate of the target bolt obtained by the depth camera, and T is the Euclidean coordinate of the final target;

[0015] The action space A is:

[0016] A=(a1,a2,a3,a4,a5,a6)

[0017] The rewards are as follows:

[0018]

[0019] Where: (x t ,y t ,z t ) is used to set the end point coordinates, and k1 and k2 are reward weight parameters.

[0020] A further technical solution is that the specific process of step 2 is:

[0021] Step 21: Mount a depth camera on the end of the robotic arm to photograph the target bolt on the contact network wrist arm, and output the pixel frame of the bolt position after processing using the YOLOV5s algorithm;

[0022] Step 22: Filter the depth value of the entire pixel frame to remove outliers and invalid points in the depth image, and take the average value of the filtered pixels as the depth value;

[0023] Step 23: Use the camera internal parameters to calculate the three-dimensional coordinates (x a ,y a ,z a ).

[0024] A further technical solution is that the outlier point determination in step 22 is as follows:

[0025] Set the outlier threshold to when The point is considered to be an outlier;

[0026]

[0027] Where: d i ,d j is the depth value.

[0028] A further technical solution is that the calculation formula in step S23 is:

[0029]

[0030] Where: (xpixel, ypixel) is the pixel coordinate, fx is the horizontal focal length, f y is the vertical focal length, c x is the optical center point of the image width, c y is the optical center point in the image height direction.

[0031] A further technical solution is that in step 3, the strategy network of the YOLOV5s target detection and SAC algorithm is initialized, and the strategy adopts a Gaussian strategy to output a noisy continuous action, two Q value networks, and a corresponding target Q network.

[0032] The present invention has the following beneficial effects:

[0033] (1) The present invention uses a robotic arm to overhaul a contact network as the background, obtains the three-dimensional coordinates of a target bolt through a depth camera, and performs real-time error update on the target coordinates, controls the robotic arm to approach the target, thereby ensuring that the end of the robotic arm is aligned with the target bolt, realizes online planning of the overhaul robotic arm, improves the robustness of the overhaul robotic arm, and thus improves the overhaul efficiency;

[0034] (2) The present invention adopts a gradual path update. Compared with the traditional path planning which adopts the path planning from the starting point to the end point, this method approaches the coordinates of the target object according to the coordinates of the target object. If the target object is slightly offset during the planning process of the robot arm, the robot arm can re-acquire the three-dimensional coordinates of the target bolt through the depth camera and continue planning until the end of the robot arm reaches the set threshold range;

[0035] (3) The low robustness problem of offline path planning is solved. That is, offline path planning may sometimes deviate due to the parking error of the maintenance vehicle or maintenance platform. This method can compensate for the error within a certain range, provide a better field of view for subsequent visual servoing, and improve the efficiency of maintenance operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is the training environment diagram;

[0037] Figure 2 Correct the floor plan for the camera field of view;

[0038] Figure 3 Convergence diagram of the SAC and PPO training robot arm. DETAILED DESCRIPTION

[0039] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The present invention provides an online position correction method for a contact network maintenance mechanical arm based on a SAC algorithm, comprising the following steps:

[0041] Step 1: Build a training environment based on the Gazebo simulation platform (such as Figure 1 As shown), it includes a three-dimensional model of the contact network, a mechanical arm, a depth camera, and a simulation platform;

[0042] State space selection. The invention uses a six-axis robotic arm as a training object, selects the robotic arm angle and angular velocity, and the target point coordinates and final target coordinates obtained from the depth camera as the state space. The state space S is shown below:

[0043]

[0044] Where: θ is the angle value of each joint of the current robotic arm, v is the speed of each joint of the current robotic arm, P is the Euclidean coordinate of the target bolt obtained by the depth camera, and T is the Euclidean coordinate of the final target;

[0045] Action space selection, considering that angular acceleration can directly or indirectly affect the angle value and speed of the robot arm joint, the angular acceleration of each joint of the robot arm is used as the action space, and the action space A is as follows:

[0046] A=(a1,a2,a3,a4,a5,a6)

[0047] Reward function design,The reward function is designed using the three-dimensional coordinates of the bolt fed back by the depth camera after identifying the target bolt. When there is a docking error, the target bolt is not in the center of the camera's field of view. In reality, there is a certain offset. After correction by the robotic arm, the center of the depth camera's field of view is aligned with the target bolt or close to the center. The schematic diagram is shown below. Figure 2 As shown, the rewards are as follows:

[0048]

[0049] Where: (x t ,y t ,z t ) is to set the end point coordinates, k1 and k2 are reward weight parameters;

[0050] Step 2: Identify the target bolts in the training environment based on the depth camera equipped with the YOLOV5s algorithm and convert them into three-dimensional coordinates with the camera as the coordinate origin;

[0051] Step 21: Mount a depth camera on the end of the robotic arm to photograph the target bolt on the contact network wrist arm, and output the pixel frame of the bolt position after processing using the YOLOV5s algorithm;

[0052] Step 22: Filter the depth value of the entire pixel frame to remove outliers and invalid points in the depth image, and take the average value of the filtered pixels as the depth value;

[0053] If there are K depth values ​​d1...d k , their benefit level S scores The specific calculation process is as follows:

[0054]

[0055] Where: d i ,d j is the depth value;

[0056] Set the outlier threshold to when The point is considered to be an outlier;

[0057] Assume that after filtering, there are K s depth value, then the depth value of the pixel frame is d avg :

[0058]

[0059] Step 23: Use the camera internal parameters to calculate the three-dimensional coordinates (x a ,y a ,z a );

[0060]

[0061] Where: (xpixel, ypixel) is the pixel coordinate, fx is the horizontal focal length, f y is the vertical focal length, c x is the optical center point of the image width, cy is the optical center point in the image height direction;

[0062] Step 3: Use the SAC algorithm to train the robot arm in the training environment to obtain the SAC output action;

[0063] Initialize the policy network of the YOLOV5s target detection and SAC algorithm. The policy adopts the Gaussian strategy, outputs a noisy continuous action, two Q-value networks, and the corresponding target Q network, and the experience playback area is used to store the data of the interaction between the agent (robotic arm) and the environment during the training process, and initialize various parameters, such as temperature coefficient, discount factor, soft update coefficient, etc.

[0064] AC is an algorithm based on maximum entropy. Its core concept is to balance exploration and exploitation by maximizing expected returns and entropy. It uses a soft strategy to improve the robustness and efficiency of the strategy, and combines the value function to evaluate the performance of the strategy, thereby achieving a more stable learning process. Therefore, the optimal strategy is:

[0065]

[0066] Among them, π is used to update the strategy that has found the maximum total reward, and ρ π Indicates that under strategy π (s t ,a t ), H(·) represents the entropy value, and α represents the hyperparameter that controls the randomness of the optimal strategy and weighs the entropy;

[0067] For the goal of maximum entropy, entropy is also considered as part of the reward amount. In the past, the Q value was calculated by accumulating the rewards, and the goal was to maximize the Q value.

[0068] The Q value of SAC can be calculated using the improved Bellman variance based on entropy, and the value function is expressed as follows:

[0069]

[0070] The state value function is obtained according to the action value function, and its expression is as follows:

[0071]

[0072] It represents the expected reward in a certain state.

[0073] Step 4: Based on the SAC output action, the Moveit framework is used to control the movement of the robot arm and interact with the environment, and the network parameters of the training environment are updated;

[0074] For the update of the Q network, the Bellman error is minimized:

[0075]

[0076] For updating the policy network parameters, the Kullback-Leibler (KL) divergence is minimized:

[0077]

[0078] The action here uses the re-parameterization technique, namely:

[0079]

[0080] Among them, the f function outputs the mean and variance, ε is the noise, and the policy network update function is:

[0081]

[0082] The S hyperparameter α uses an adaptive method and changes during the training process. Its update function is:

[0083]

[0084] Step 5: Repeat steps 1 to 4, and continue to repeat the process of environment interaction, experience sampling, and network update until the strategy converges or reaches the specified training round.

[0085] The robot arm interacts with the environment for training. In a training round, the environment initializes S1 and selects action a according to the current strategy. t =π θ (s t ), execute action a t , get reward r t , the environment state becomes S t+1 , will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool R. When the robot reaches the target point or the number of interactions between the robot and the environment in a round reaches the maximum, the training of this round ends and the environment is reinitialized for a new round of training. When there is enough experience stored in the experience replay pool, a certain amount of data is extracted from the experience replay pool R by random sampling for training, thereby updating various parameters.

[0086] After the robot interacts with the environment a certain number of times, its understanding of the environment continues to deepen. By updating the strategy, Q-value function, and entropy regularization coefficient α, the algorithm gradually optimizes the strategy so that the agent can maximize the expected reward. As training progresses, the strategy and Q-value function gradually stabilize, and the algorithm's loss value converges, indicating that the strategy has found a near-optimal behavior in the current environment.

[0087] The present invention trains the robot arm to approach the target center. Compared with the traditional path planning algorithm, the path planning path based on reinforcement learning is more stable and reduces the path planning overhead. Finally, based on the path planning strategy after the deep reinforcement learning algorithm converges, the target bolt center detected by the depth camera is approached as the end point, thereby realizing the online correction of the robot arm path and providing a more accurate field of view for subsequent visual servoing.

[0088] The above description is not intended to limit the present invention in any form. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are within the scope of the technical solution of the present invention.

Claims

1. A method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm, characterized in that: The following steps are involved: Step 1: Build a training environment based on the Gazebo simulation platform; Step 2: Identify the target bolts in the training environment based on the depth camera equipped with the YOLOV5s algorithm and convert them into three-dimensional coordinates with the camera as the coordinate origin; Step 3: Use the SAC algorithm to train the robot arm in the training environment to obtain the SAC output action; Step 4: Based on the SAC output action, the Moveit framework is used to control the movement of the robot arm and interact with the environment, and the network parameters of the training environment are updated; Step 5: Repeat steps 1 to 4, and continue to repeat the process of environment interaction, experience sampling, and network parameter update until the strategy converges or reaches the specified training round.

2. The method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm according to claim 1, characterized in that: The training environment in step 1 includes a three-dimensional model of the contact network, a robotic arm, a depth camera, and a simulation platform.

3. The method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm according to claim 1, characterized in that: The state space S of the training environment in step 1 is: Where: θ is the angle value of each joint of the current robotic arm, v is the speed of each joint of the current robotic arm, P is the Euclidean coordinate of the target bolt obtained by the depth camera, and T is the Euclidean coordinate of the final target; The action space A is: A=(a1,a2,a3,a4,a5,a6) The rewards are as follows: Where: (x t ,y t ,z t ) is used to set the end point coordinates, and k1 and k2 are reward weight parameters.

4. The method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 21: Mount a depth camera on the end of the robotic arm to photograph the target bolt on the contact network wrist arm, and output the pixel frame of the bolt position after processing using the YOLOV5s algorithm; Step 22: Filter the depth value of the entire pixel frame to remove outliers and invalid points in the depth image, and take the average value of the filtered pixels as the depth value; Step 23: Use the camera internal parameters to calculate the three-dimensional coordinates (x a ,y a ,z a ).

5. The method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm according to claim 4, characterized in that: The outlier point determination in step 22 is as follows: Set the outlier threshold to S` scores , when S` scores ≥S scores , then the point is considered to be an outlier; Where: d i d j is the depth value.

6. The method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm according to claim 4, characterized in that: The calculation formula in step S23 is: Where: (xpixel, ypixel) is the pixel coordinate, fx is the horizontal focal length, f y is the vertical focal length, c x is the optical center point of the image width, c y is the optical center point in the image height direction.

7. The method for online position correction of a contact network maintenance mechanical arm based on a SAC algorithm according to claim 1, characterized in that: In step 3, the policy network of the YOLOV5s target detection and SAC algorithm is initialized. The policy adopts a Gaussian strategy and outputs a noisy continuous action, two Q value networks, and a corresponding target Q network.

Images