Online position correction method for catenary maintenance mechanical arm based on SAC algorithm

Abstract

This invention relates to an online position correction method for a robotic arm used in overhead contact line maintenance based on the SAC algorithm, belonging to the field of overhead contact line maintenance technology. The invention discloses an online position correction method for a robotic arm used in overhead contact line maintenance based on the SAC algorithm, including: building a training environment based on the Gazebo simulation platform; using a depth camera equipped with the YOLOv5s algorithm to identify target bolts and convert them into three-dimensional coordinates with the camera as the origin; training the robotic arm using the SAC algorithm in the training environment; and controlling the robotic arm's movement and interaction with the environment through the Moveit framework based on the SAC output actions, updating experience sampling and network parameters. This invention updates the end effector of the robotic arm based on target point detection by the depth camera and maintains low variability in each planned path, aiming to improve the robustness of the robotic arm path planning system and the accuracy of the robotic arm's end effector, providing a better camera field of view for subsequent maintenance operations.

Description

Technical Field

[0001] This invention relates to an online position correction method for a robotic arm for overhead contact line maintenance based on the SAC algorithm, belonging to the field of overhead contact line maintenance technology. Background Technology

[0002] In the railway system, the maintenance of overhead contact lines currently relies mainly on manual labor, which presents problems such as safety risks, low efficiency, and high costs. With the development of robotics technology, robotic arms will play a more important role in the maintenance of overhead contact line cantilever arms along railway lines. In traditional industrial applications, the path planning of robotic arms mostly adopts offline path planning with manual teaching, rather than online planning. The path of the robotic arm is planned by manual teaching, allowing the robotic arm to repeatedly perform a certain task. Compared with the fixed environment application of manual teaching, the environmental context of robotic arms maintaining overhead contact line cantilever arms includes errors in the parking of the maintenance vehicle and control errors of the work platform. The superposition of these two errors causes the offline path to fail to accurately reach the pre-planned target point, and the depth camera mounted on the end effector of the maintenance robotic arm cannot properly photograph the bolt target or has an inadequate shooting angle, making it impossible to carry out subsequent maintenance work normally.

[0003] In summary, offline planning systems have poor robustness and are not entirely suitable for the environment of robotic arms repairing overhead contact line cantilever arms. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention aims to provide an online position correction method for a robotic arm for overhead contact line maintenance based on the SAC algorithm. This method updates the end point of the robotic arm based on the target point detected by the depth camera and maintains low variability in each planned path, thereby improving the robustness of the robotic arm path planning system and the accuracy of the robotic arm end point, and providing a better camera field of view for subsequent maintenance operations.

[0005] The technical solution provided by this invention to solve the above-mentioned technical problems is: an online position correction method for a robotic arm for overhead contact line maintenance based on the SAC algorithm, comprising the following steps:

[0006] Step 1: Set up the training environment based on the Gazebo simulation platform;

[0007] Step 2: Based on the depth camera and equipped with the YOLOv5s algorithm, identify the target bolts in the training environment and convert them into three-dimensional coordinates with the camera as the origin;

[0008] Step 3: Train the robotic arm using the SAC algorithm in the training environment to obtain the SAC output action;

[0009] Step 4: Based on the SAC output action, control the movement of the robotic arm and interact with the environment through the Moveit framework to update the network parameters of the training environment;

[0010] Step 5: Repeat steps 1-4, continuously repeating the process of environmental interaction, experience sampling, and network parameter updates until the policy converges or the specified number of training rounds is reached.

[0011] A further technical solution is that the training environment in step 1 includes a 3D model of the overhead contact line, a robotic arm, a depth camera, and a simulation platform.

[0012] A further technical solution is that the state space S of the training environment in step 1 is:

[0013]

[0014] In the formula: θ is the angle value of each joint of the current robotic arm, v is the speed of each joint of the current robotic arm, P is the Euclidean coordinate of the target bolt obtained by the depth camera, and T is the Euclidean coordinate of the final target.

[0015] Action space A is:

[0016] A = (a1, a2, a3, a4, a5, a6)

[0017] The reward method is as follows:

[0018]

[0019] In the formula: (x t ,y t ,z t ) is used to set the endpoint coordinates, and k1 and k2 are the reward weight parameters.

[0020] A further technical solution is that the specific process of step 2 is as follows:

[0021] Step 21: Mount a depth camera on the end of the robotic arm to photograph the target bolt on the contact wire arm. After processing with the YOLOv5s algorithm, output the pixel box of the bolt position.

[0022] Step 22: Filter the depth value of the entire pixel box to remove outliers and invalid points in the depth image, and take the average value of the filtered pixels as the depth value.

[0023] Step 23: Calculate the three-dimensional coordinates (x, y) of the bolt in the camera coordinate system using the camera intrinsic parameters. a ,y a ,z a ).

[0024] A further technical solution is that the outlier determination in step 22 is as follows:

[0025] Set the outlier threshold to when Then the point is considered an outlier;

[0026]

[0027] In the formula: d i d j This represents the depth value.

[0028] A further technical solution is that the calculation formula in step S23 is:

[0029]

[0030] In the formula: (xpixel, ypixel) are pixel coordinates, fx is the focal length in the horizontal direction, and f y c is the focal length in the vertical direction. x c is the optical center point of the image width. y It is the optical center point in the height direction of the image.

[0031] A further technical solution is that, in step 3, the policy network for YOLOv5s target detection and SAC algorithm is initialized. This policy adopts a Gaussian policy and outputs a noisy continuous action, two Q-value networks, and a corresponding target Q-network.

[0032] The present invention has the following beneficial effects:

[0033] (1) This invention takes the maintenance of contact network by robotic arm as the background. It uses a depth camera to obtain the three-dimensional coordinates of the target bolt and the target coordinates to update the error in real time. It controls the robotic arm to move closer to the target, thereby ensuring that the end of the robotic arm is aligned with the target bolt. This realizes the online planning of the maintenance robotic arm, improves the robustness of the maintenance robotic arm, and thus improves the maintenance efficiency.

[0034] (2) The present invention adopts a step-by-step path update method. Compared with the traditional path planning, which adopts the path planning from the starting point to the end point, this method is based on the coordinates of the target object and moves closer to the coordinates. If the target object deviates slightly during the planning process of the robotic arm, the robotic arm can re-acquire the three-dimensional coordinates of the target bolt through the depth camera and continue planning until the end of the robotic arm reaches the set threshold range.

[0035] (3) It solves the problem of low robustness in offline path planning. That is, offline path planning may sometimes deviate due to the docking error of the maintenance vehicle or maintenance platform. However, this method can make up for the error within a certain range, provide a better field of view for subsequent visual servoing, and improve the efficiency of maintenance operations. Attached Figure Description

[0036] Figure 1 Training environment diagram;

[0037] Figure 2 Correct the plan view for the camera's field of view;

[0038] Figure 3 Convergence plot of the robotic arm trained for SAC and PPO. Detailed Implementation

[0039] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The present invention provides an online position correction method for a robotic arm for overhead contact line maintenance based on the SAC algorithm, comprising the following steps:

[0041] Step 1: Set up the training environment based on the Gazebo simulation platform (e.g., Figure 1 As shown in the figure, it includes a 3D model of the overhead contact line, a robotic arm, a depth camera, and a simulation platform;

[0042] State space selection: This invention uses a six-axis robotic arm as the training object, selecting the robotic arm angle and angular velocity, as well as the target point coordinates and the final target coordinates obtained from the depth camera, as the state space, as shown below:

[0043]

[0044] In the formula: θ is the angle value of each joint of the current robotic arm, v is the speed of each joint of the current robotic arm, P is the Euclidean coordinate of the target bolt obtained by the depth camera, and T is the Euclidean coordinate of the final target.

[0045] For motion space selection, considering that angular acceleration can directly or indirectly affect the angle values ​​and velocities of the robotic arm joints, the angular acceleration of each joint of the robotic arm is used as the motion space. Motion space A is shown below:

[0046] A = (a1, a2, a3, a4, a5, a6)

[0047] The reward function is designed using the 3D coordinates of the bolt detected by the depth camera. When there is a docking error, the target bolt is not in the center of the camera's field of view; in reality, there is a certain offset. After correction by the robotic arm, the center of the depth camera's field of view is aligned with or close to the target bolt. A schematic diagram is shown below. Figure 2 As shown, the reward method is as follows:

[0048]

[0049] In the formula: (x t ,y t ,z t () sets the endpoint coordinates, and k1 and k2 are the reward weight parameters;

[0050] Step 2: Based on the depth camera and equipped with the YOLOv5s algorithm, identify the target bolts in the training environment and convert them into three-dimensional coordinates with the camera as the origin;

[0051] Step 21: Mount a depth camera on the end of the robotic arm to photograph the target bolt on the contact wire arm. After processing with the YOLOv5s algorithm, output the pixel box of the bolt position.

[0052] Step 22: Filter the depth value of the entire pixel box to remove outliers and invalid points in the depth image, and take the average value of the filtered pixels as the depth value.

[0053] If K exists in depth values ​​d1...d k Their degree of interest group S scores The specific calculation process is as follows:

[0054]

[0055] In the formula: d i d j This is the depth value;

[0056] Set the outlier threshold to when Then the point is considered an outlier;

[0057] Assuming that K remains after filtering s If there are several depth values, then the depth of the pixel box is d. avg :

[0058]

[0059] Step 23: Calculate the three-dimensional coordinates (x, y) of the bolt in the camera coordinate system using the camera intrinsic parameters. a ,y a ,z a );

[0060]

[0061] In the formula: (xpixel, ypixel) are pixel coordinates, fx is the focal length in the horizontal direction, and f y c is the focal length in the vertical direction. x c is the optical center point of the image width.y The optical center point in the height direction of the image;

[0062] Step 3: Train the robotic arm using the SAC algorithm in the training environment to obtain the SAC output action;

[0063] Initialize the policy network for YOLOv5s object detection and SAC algorithm. The policy adopts a Gaussian policy and outputs a noisy continuous action, two Q-value networks, and the corresponding target Q-network. The experience replay area is used to store the data of the agent (robotic arm) interacting with the environment during training. Initialize various parameters, such as temperature coefficient, discount factor, soft update coefficient, etc.

[0064] AC is a maximum entropy-based algorithm whose core idea is to balance exploration and exploitation by maximizing expected reward and entropy. It employs a soft policy to improve the policy's robustness and efficiency, while combining a value function to evaluate the policy's performance, thus achieving a more stable learning process. Therefore, the optimal policy is:

[0065]

[0066] Where π is used to update the strategy that has found the maximum total reward, ρ π This indicates that under strategy π (s) t ,a t The distribution of H(·) is given by H(·), where H(·) represents the entropy value and α represents the degree of randomness in controlling the optimal strategy and the hyperparameter that balances entropy.

[0067] For the goal of maximizing entropy, entropy is also included as part of the reward amount. In the past, the Q value was calculated by accumulating the rewards, and the goal is to maximize the Q value.

[0068] The Q value of SAC can be calculated using the entropy-based improved Bellman variance, and the value function is expressed as follows:

[0069]

[0070] The state value function is derived from the action value function, and its expression is as follows:

[0071]

[0072] It represents the expected reward in a certain state.

[0073] Step 4: Based on the SAC output action, control the movement of the robotic arm and interact with the environment through the Moveit framework to update the network parameters of the training environment;

[0074] For the Q-network update, the Bellman error is minimized:

[0075]

[0076] For updating the policy network parameters, the Kullback-Leibler (KL) divergence is minimized:

[0077]

[0078] The action here uses a reparameterization technique, namely:

[0079]

[0080] Where the function f outputs the mean and variance, and ε is noise, the policy network update function is:

[0081]

[0082] The hyperparameter α of S is adaptive and changes during training. Its update function is:

[0083]

[0084] Step 5: Repeat steps 1-4, continuously repeating the process of environment interaction, experience sampling, and network update until the policy converges or the specified number of training rounds is reached.

[0085] The robotic arm interacts with the environment for training. In one training round, the environment is initialized (S1), and action a is selected based on the current policy. t =π θ (s t ), perform action a t Get reward r t The environmental state changes to S t+1 , will (s t ,a t ,r t ,s t+1 The data is stored in the experience replay pool R. When the robotic arm reaches the target point or the number of interactions between the robotic arm and the environment reaches its maximum in a round, the training round ends, and the environment is reinitialized for a new round of training. When there is enough experience stored in the experience replay pool, a certain amount of data is randomly sampled from the experience replay pool R for training, thereby updating various parameters.

[0086] After the robotic arm interacts with its environment a certain number of times, its understanding of the environment deepens. By updating the policy, Q-value function, and entropy regularization coefficient α, the algorithm gradually optimizes the policy, enabling the agent to maximize the expected reward. As training progresses, the policy and Q-value function gradually stabilize, and the algorithm's loss value converges, indicating that the policy has found a near-optimal behavior in the current environment.

[0087] This invention trains a robotic arm to approach the target center. Compared to traditional path planning algorithms, the path planning based on reinforcement learning is more stable and reduces path planning overhead. Finally, based on the path planning strategy converged by the deep reinforcement learning algorithm, the robotic arm approaches the target bolt center detected by the depth camera as the endpoint, thereby achieving online correction of the robotic arm's path and providing a more accurate field of view for subsequent visual servoing.

[0088] The above description is not intended to limit the present invention in any way. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall fall within the scope of the present invention.

Claims

1. A method for online position correction of a robotic arm for overhead contact line maintenance based on the SAC algorithm, characterized in that, Includes the following steps: Step 1: Set up the training environment based on the Gazebo simulation platform; The state space of the training environment S for: In the formula: θ These are the current angle values ​​of each joint of the robotic arm. v The current speed of each joint of the robotic arm, P To obtain the Euclidean coordinates of the target bolt for the depth camera, T The Euclidean coordinates for the final destination; Action space A for: The reward method is as follows: In the formula: To set the endpoint coordinates, , For reward weighting parameters; Step 2: Based on the depth camera and equipped with the YOLOv5s algorithm, identify the target bolts in the training environment and convert them into three-dimensional coordinates with the camera as the origin; Step 21: Mount a depth camera on the end of the robotic arm to photograph the target bolt on the contact wire arm. After processing with the YOLOv5s algorithm, output the pixel box of the bolt position. Step 22: Filter the depth value of the entire pixel box to remove outliers and invalid points in the depth image, and take the average value of the filtered pixels as the depth value. Step 23: Calculate the three-dimensional coordinates of the bolt in the camera coordinate system using the camera intrinsic parameters. ; Step 3: Train the robotic arm using the SAC algorithm in the training environment to obtain the SAC output action; Step 4: Based on the SAC output action, control the movement of the robotic arm and interact with the environment through the Moveit framework to update the network parameters of the training environment; Step 5: Repeat steps 1-4, continuously repeating the process of environmental interaction, experience sampling, and network parameter updates until the policy converges or the specified number of training rounds is reached.

2. The online position correction method for a robotic arm for overhead contact line maintenance based on the SAC algorithm according to claim 1, characterized in that, The training environment in step 1 includes a 3D model of the overhead contact line, a robotic arm, a depth camera, and a simulation platform.

3. The online position correction method for a contact network maintenance robotic arm based on the SAC algorithm according to claim 1, characterized in that, The outlier identification in step 22 is as follows: Set the outlier threshold to ,when If the value is 0, then the point is considered an outlier. In the formula: d i , d j This represents the depth value.

4. The online position correction method for a contact network maintenance robotic arm based on the SAC algorithm according to claim 1, characterized in that, The calculation formula in step 23 is: In the formula: For pixel coordinates, The focal length in the horizontal direction. The focal length is in the vertical direction. The optical center point of the image width. It is the optical center point in the height direction of the image.

5. The online position correction method for a contact network maintenance robotic arm based on the SAC algorithm according to claim 1, characterized in that, In step 3, the policy network for YOLOv5s object detection and the SAC algorithm is initialized. This policy adopts a Gaussian policy and outputs a noisy continuous action and two... Q Value network, and the corresponding target Q network.

Images