A visual servoing control method for power robots

Abstract

The application relates to a visual servo control method for a power robot, which comprises the following steps: acquiring a color image around a mechanical arm of the power robot in real time, wherein the color image contains an Aruco code of a target position; acquiring camera internal parameters, and combining image information of the target and a pose of an end effector of the power robot to calculate a target position pixel coordinate in a power robot coordinate system; detecting the Aruco code in the color image in real time, and calculating a new pose of the end effector of the power robot according to the target position pixel coordinate in the power robot coordinate system and a current position pixel coordinate, so that the end effector of the power robot moves from the current position; repeating all the above steps until an end condition is reached, and completing a control process. Compared with the prior art, the application has the advantages of improving the working efficiency of the power robot and the like.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a visual servo control method for electric robots. Background Technology

[0002] As the power grid continues to expand under high voltage and high magnetic field conditions, operational errors are increasing, and the capacity of personnel is insufficient, highlighting safety risks that threaten personnel safety. Against this backdrop, power inspection robots are being applied to a wider range of power scenarios, as traditional human-robot co-operation in insulated chambers is no longer adequate for the control performance and accuracy requirements of live-line working environments.

[0003] Existing remotely controlled robots face numerous challenges. For example, the uncertainty in visual servoing leads to positioning failures due to target occlusion during localization, grasping, and movement. Furthermore, limitations in sensor accuracy and control system stability hinder precise operations. In inspection scenarios, operators struggle to adapt to varying noise levels. Visual feedback may be delayed or inaccurate, and poor dynamic planning and obstacle avoidance capabilities make it difficult to handle changing environments and unexpected situations, impacting the robot's ability to perceive and respond in real-time. Currently, remote-controlled switching operations only provide visual images to the control unit, causing operators to become overly reliant on these images and prone to visual fatigue. Moreover, the inherent biases in manual visual image observation can lead to operator errors, reducing efficiency and potentially causing accidents. For instance, if the tool continues to rub against the high-voltage cabinet after contact with the visual image while the operator is remotely controlling the robot for switching operations, the operator cannot continue controlling the robot based on the visual image, reducing efficiency and posing a safety hazard. More importantly, existing technologies cannot achieve autonomous robot operation, and robots rely excessively on human intervention during operations, which greatly limits the efficiency and application scope of live-line working robots. Currently, live-line working robots have significant shortcomings in terms of comprehensive processing capabilities.

[0004] Currently, substation operation and maintenance lacks efficient and reliable intelligent systems and safety guarantees. Summary of the Invention

[0005] The purpose of this invention is to provide an operation method for visual servo control of electric robots to improve their working efficiency in scenarios with varying depth information.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A vision servo control method for electric robots includes the following steps:

[0008] Real-time acquisition of color images around the robotic arm of an electric robot, wherein the color images contain Aruco codes for the target location;

[0009] Obtain the camera intrinsic parameters, and combine the image information of the target with the pose of the electric robot end effector to calculate the target position pixel coordinates in the electric robot coordinate system;

[0010] The Aruco code in the color image is detected in real time, and the new pose of the electric robot end effector is calculated based on the target position pixel coordinates and the current position pixel coordinates in the electric robot coordinate system, so that the electric robot end effector moves from the current position.

[0011] Repeat all the above steps until the termination condition is met, thus completing the control process.

[0012] Furthermore, the step of calculating the target position pixel coordinates in the electric robot coordinate system includes:

[0013] Based on the camera intrinsic parameters, calculate the target position coordinates in the camera coordinate system;

[0014] Based on the pose of the electric robot's end effector, the target position coordinates in the camera coordinate system are converted to the target position pixel coordinates in the electric robot coordinate system using an inverse transformation matrix. The expression for the inverse transformation matrix is ​​as follows:

[0015]

[0016] In the formula, Let R be the inverse transformation matrix, R be the rotation matrix, and t be the translation vector.

[0017] Furthermore, the step of calculating the target position coordinates in the camera coordinate system includes:

[0018] Based on the depth value and the target location pixel coordinates (u,v), the mapping relationship is as follows:

[0019]

[0020] Based on the camera intrinsic parameters, the target position pixel coordinates (u,v) are transformed into the camera coordinate system to obtain the target position coordinates in the camera coordinate system:

[0021]

[0022] Z = z

[0023] In the formula, (X,Y,Z) are the three-dimensional coordinates in the camera coordinate system, and f x f y u0 and v0 are camera intrinsic parameters, where fx f y Let be the camera focal length, u0 and v0 be the x and y coordinates of the principal point of the image, respectively, and z be the distance from the camera to the target position.

[0024] Furthermore, the step of calculating the new pose of the electric robot end effector includes:

[0025] Calculate the error between the target position pixel coordinates and the current position pixel coordinates in the electric robot coordinate system;

[0026] Based on the aforementioned error, an adaptive gain controller is used to calculate the control gain;

[0027] Calculate the incremental motion that the end effector of the electric robot needs to perform based on the camera parameters and control gain;

[0028] The motion increment is converted into the pose increment of the electric robot end effector, and the new pose of the electric robot end effector is obtained based on the pose increment.

[0029] Furthermore, the error is calculated using Euclidean distance, and the expression for the error is as follows:

[0030] norm_error=||(u c -u goal ,v c -v goal )||2

[0031] In the formula, norm_error is the error, (u c ,v c ) represents the current pixel coordinates, (u goal ,v goal () represents the pixel coordinates of the target location.

[0032] Furthermore, the expression for calculating the control gain is:

[0033]

[0034] In the formula, λ x To control the gain, λ0 is the initial gain, also known as the control strength. ∞ It is the lower bound of the gain, λ' ∞ It is the adjustment factor for controlling the gain, and norm_error is the error.

[0035] Furthermore, the expression for the action increment is:

[0036]

[0037] In the formula, a is the increment, Δu vFor the action increment, C ij Let f be a coefficient matrix, representing the effect of changes in the camera coordinate system on pixel coordinates. x f y u0 and v0 are camera intrinsic parameters, where f x f y Let u0 and v0 be the camera focal length, u0 and v0 be the x and y coordinates of the principal point of the image, respectively, and z be the distance from the camera to the target position. c ,v c () represents the pixel coordinates of the current position.

[0038] Furthermore, the expression for the new pose of the electric robot end effector is:

[0039] T robot_new =T robot_current ·ΔT robot

[0040] In the formula, T robot_new For the new position, T robot_current Given the current pose, ΔT robot This is the pose increment.

[0041] Further, the termination condition is that the error between the target position pixel coordinates and the current position pixel coordinates in the electric robot coordinate system is less than or equal to a set threshold. Reaching the termination condition indicates that the electric robot's end effector has moved to the target position. The expression for the termination condition is:

[0042]

[0043] In the formula, end_location is the termination condition, u c [i][j] represents the coordinates of the current corner point, u goal [i][j] represents the coordinates of the target corner point, and threshold is the set threshold value.

[0044] Furthermore, when the robot's end effector moves from its current position, linear interpolation is used for a smooth transition:

[0045] T robot_final =T robot +ΔT robot

[0046] In the formula, T robot_final For the final pose of the robot's end effector, T robot Let Δt be the rotation matrix. robot This is the pose increment.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) To address the problem of target tracking with constantly changing depth information, this invention uses depth images to obtain depth values ​​and color images to continuously detect the Aruco code of the target position, so that the end effector of the electric robot can continuously update its pose toward the target position, thereby improving the flexibility of the end effector of the electric robot and thus improving work efficiency.

[0049] (2) During the pose update process, this invention can dynamically adjust the parameters of the control system based on the real-time motion state of the target object and environmental changes. The introduction of an adaptive gain controller offers real-time performance compared to traditional gain settings. It can automatically optimize the control gain by detecting system errors and environmental changes in real time, effectively avoiding control failures caused by environmental changes or error accumulation, thereby achieving more precise trajectory tracking and dynamic adjustment. This control strategy not only ensures that the robotic arm can work efficiently under different tasks and operating conditions, but also maintains stability under conditions of rapid movement or significant disturbance. Especially in dynamic industrial environments, it effectively improves the robustness and response speed of the system, ensuring that the robotic arm can accurately control the target pose.

[0050] (3) The method of uncalibrated visual servo control, adaptive parameter estimation and depth-independent Jacobian matrix estimation adopted in this invention can not only compensate for the depth information of uncalibrated camera parameters and unknown changing motion, and improve the robustness of motion in power scenarios, but also reduce redundant motion by having no constraints on depth information, which is beneficial to reducing the time for the camera to reach the target and improving the accuracy and efficiency of control. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0053] This embodiment provides a vision servo control method for electric robots, such as... Figure 1 As shown, the method includes the following steps:

[0054] Step S1: Initialize the JAKAS7 robotic arm and the RealSense Depth camera D457 to capture visual information around the robotic arm in real time.

[0055] The specific steps are as follows:

[0056] Step S1: Connect the JAKAS7 robotic arm controller via the jkrc library. Use a Python script to communicate with the robotic arm to acquire the robot's joint angle pose information in real time. The program can update and control the robot's TCP (Work Center Point) pose in real time, converting it to meters until the current position of the robot's end effector is printed. Initialize the RealSense camera and set its parameters, ensuring the depth stream and RGB video stream frame rates are consistent. Use the API provided by the camera SDK to acquire target depth information and color images. The depth image contains the Aruco code of the target location. Gaussian filtering is then used to blur the image, reducing noise and obtaining stable images from the sensor to ensure data quality.

[0057] Step S2: Start robot control and camera stream, and set the recognition dictionary and detection parameters for Aruco code tags.

[0058] Step S2 involves setting the recognition dictionary and detection parameters for the Aruco code markers, and marking the Aruco code corner coordinates in real time in each frame of the color image for subsequent calculations. To address the stability issue of corner detection caused by strong glare in the power cabinet operating environment, the detection model must ensure both high accuracy and real-time performance.

[0059] ARUCO codes, as pattern-based two-dimensional markers, possess high robustness and good detection accuracy, providing stable pose estimation in dynamic or complex environments. In dynamic environments, target objects may be affected by occlusion, lighting changes, or other interference sources. By leveraging the high robustness of ARUCO code localization, the system can ensure stable target detection and tracking even in complex environments. In practical applications, by utilizing the unique pattern and encoding method of ARUCO codes, the system can accurately and quickly extract the position and orientation of target objects from camera images. Compared to traditional visual servoing methods, ARUCO codes not only significantly improve positioning accuracy but also overcome interference factors such as lighting changes and background complexity, showing particular advantages in industrial automation. Combining ARUCO codes with visual servoing systems enables efficient target localization and precise control within a large workspace.

[0060] Step S3: Obtain camera intrinsic parameters.

[0061] In this section, the camera intrinsic parameters of this embodiment include focal length and principal point position. The pixel coordinates of the target marker are obtained through the camera and converted into three-dimensional coordinates in the camera coordinate system. The key formulas are as follows:

[0062] 1) Calculate the target position in the camera coordinate system

[0063] Based on the obtained target marker pixel coordinates (u,v), the mapping relationship is as follows:

[0064]

[0065] Where: z is the distance from the camera to the target position (unit: meters), f x f y Let u0 be the camera focal length (in pixels), and v0 be the horizontal and vertical coordinates of the principal point of the image, respectively.

[0066] Through camera intrinsic parameters f x f y Given u0 and v0, we can convert the pixel coordinates (u, v) into three-dimensional coordinates (X, Y, Z) in the camera coordinate system:

[0067]

[0068] Z = z

[0069] 2) Transform to robot coordinate system

[0070] Let the pose of the robot's end effector be (x, y, z, rx, ry, rz), where x, y, z are the positions of the robot's end effector, and rx, ry, rz are its orientations. The inverse transformation matrix... Transform the camera coordinate system to the robot coordinate system:

[0071]

[0072] Where R is the rotation matrix and t is the translation vector.

[0073] Step S4: Perform the positioning task, continuously detect Aruco code markers, calculate pixel errors, and drive the robotic arm to move towards the target position.

[0074] Once it reaches the designated location area, the robot will provide real-time feedback and adjustments based on the calculated increments until it reaches the predetermined accuracy.

[0075] (1) Calculation error: First, we calculate the current pixel (u c ,v c ) and target pixel (u goal ,v goal The error between the pixel coordinates and the Euclidean distance can be calculated.

[0076] norm_error=Π(u c -u goal ,v c -v goal )Π2

[0077] This error measures the distance between the current pixel and the target pixel. A larger error means more adjustments are needed.

[0078] (2) Calculate the control gain: Based on the error magnitude, use an adaptive gain controller to calculate the control gain λ. x :

[0079]

[0080] Where: λ0 is the initial gain (control strength), λ ∞ It is the lower bound of the gain, λ' ∞ It is the adjustment factor that controls the gain.

[0081] This formula determines the adjustment force required by the robot. The larger the error, the greater the gain, and the faster the robot adjusts.

[0082] This invention relates to an adaptive gain controller that dynamically adjusts the parameters of the control system based on the real-time motion state of the target object and environmental changes. Compared to traditional gain settings, the adaptive gain controller offers real-time performance, automatically optimizing the control gain by detecting system errors and environmental changes in real time. This effectively avoids control failures caused by environmental changes or error accumulation, thereby achieving more precise trajectory tracking and dynamic adjustment. This control strategy not only ensures the robotic arm operates efficiently under different tasks and operating conditions but also maintains stability under rapid movement or significant disturbances. Especially in dynamic industrial environments, it effectively improves the system's robustness and response speed, ensuring the robotic arm can accurately control the target's pose.

[0083] (3) Calculate pixel coordinate increment: using camera intrinsic parameters and the calculated gain λ x Calculate the incremental motion (in pixel coordinates) required by the robot's end effector. Define a coefficient matrix C. ij This indicates the effect of changes in the camera coordinate system on pixel coordinates:

[0084]

[0085] The coefficient matrix is ​​a Jacobian matrix, which is the pixel coordinate increment based on the color image information.

[0086] Then, the pseudo-inverse of the matrix is ​​obtained:

[0087]

[0088] In the formula, 'a' represents the increment, and this action increment Δu v This represents the required motion of the robot's end effector in the camera coordinate system, (u c ,v c() represents the pixel coordinates of the current position.

[0089] (4) Convert the increment to the robot end effector pose: We use the calculated motion increment Δu v As input, it is converted into the pose increment ΔT of the robot's end effector. robot The ultimate goal is to convert the increment in the camera coordinate system into motion commands for the robot's end effector. Assuming the motion increment 'a' is the pose increment calculated by the robot, the new pose of the robot's end effector can be obtained using the following formula:

[0090] T robot_new =T robot_current ·ΔT robot

[0091] In the formula, ΔT robot This is the pose increment.

[0092] The robot will provide real-time feedback and adjustments based on the calculated increments until the predetermined accuracy is achieved.

[0093] Step S5: When the positioning error is less than the set threshold, the positioning is marked as complete, the program stops and prints the final robot pose.

[0094] Step S5 involves checking the current pixel (u) after each action is performed. c v c ) and target pixel (u goal v goal The difference in error is used to determine whether the positioning process has ended. The termination condition is that the error is less than a set threshold.

[0095]

[0096] The formula uses the current pixel coordinates and target pixel coordinates obtained through color image processing methods, and controls the movement of the robotic arm by the difference between the two.

[0097] The localization process ends when the difference between all pixel coordinates is less than a set threshold.

[0098] Calculating the robot's end-effector pose: After each adjustment, the robot's end-effector pose can be obtained through inverse calculation. Assuming the robot's current end-effector pose is (x, y, z, rx, ry, rz), then:

[0099]

[0100] Among them, R robot It is a rotation matrix, t robot It is a translation vector.

[0101] Once the target position is calculated, the robot's end effector translates from its current position to the target position, typically using linear interpolation for a smooth transition.

[0102] T robot_final =T robot +ΔT robot

[0103] Then, the robot performs the pose change using linear interpolation.

[0104] This invention tightly integrates the target object's pose and positioning information with the robotic arm control algorithm, forming a closed-loop control system. Using precise pose data provided by ARUCO codes, combined with real-time adjustments from an adaptive gain controller, the system continuously adjusts the robotic arm's movements to accurately track the target, achieving a positioning accuracy within one pixel. In practical applications, this method not only accurately locates the target object but also ensures stable target tracking during movement, avoiding deviations caused by error accumulation or external interference. This system performs exceptionally well in high-speed and high-precision tasks, making it particularly suitable for automated assembly, precision operations, and other industrial applications requiring high-precision positioning.

[0105] In summary, this invention of robotic arm visual servo positioning combining ARUCO codes and adaptive gain controllers provides a high-precision, robust, and adaptively adjustable visual servo system suitable for various complex and dynamic industrial applications, offering new solutions and directions for technological innovation in related fields.

[0106] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0107] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0108] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0109] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A visual servo control method for electric robots, characterized in that, Includes the following steps: Real-time acquisition of color images around the robotic arm of an electric robot, wherein the color images contain Aruco codes for the target location; Obtain the camera intrinsic parameters, and combine the image information of the target with the pose of the electric robot end effector to calculate the target position pixel coordinates in the electric robot coordinate system; The Aruco code in the color image is detected in real time, and the new pose of the electric robot end effector is calculated based on the target position pixel coordinates and the current position pixel coordinates in the electric robot coordinate system, so that the electric robot end effector moves from the current position. The step of calculating the new pose of the electric robot end effector includes: Calculate the error between the target position pixel coordinates and the current position pixel coordinates in the electric robot coordinate system, wherein the error is calculated using Euclidean distance, and the expression for the error is: In the formula, For the error, ( , ) represents the current pixel coordinates, ( , () represents the pixel coordinates of the target location; Based on the aforementioned error, an adaptive gain controller is used to calculate the control gain, and the expression for the calculation of the control gain is as follows: In the formula, To control the gain, This is the initial gain, also known as the control strength. It is the lower limit of the gain. It is the adjustment factor for controlling the gain. For error; Based on the camera parameters and control gain, the required motion increment for the end effector of the electric robot is calculated. The expression for the motion increment is: In the formula, a For increments, For action increment, This is a coefficient matrix, representing the effect of changes in the camera coordinate system on pixel coordinates. , , , For camera intrinsic parameters, among which , For camera focal length, , These are the x and y coordinates of the principal point of the image, respectively. The distance from the camera to the target location, ( , () represents the pixel coordinates of the current position; The motion increment is converted into a pose increment of the electric robot end effector, and a new pose of the electric robot end effector is obtained based on the pose increment. The expression for the new pose of the electric robot end effector is as follows: In the formula, For the new position, For the current pose, For pose increment; Repeat all the above steps until the termination condition is met, thus completing the control process.

2. The visual servo control method for electric robots according to claim 1, characterized in that, The steps for calculating the target position pixel coordinates in the electric robot coordinate system include: Based on the camera intrinsic parameters, calculate the target position coordinates in the camera coordinate system; Based on the pose of the electric robot's end effector, the target position coordinates in the camera coordinate system are converted to the target position pixel coordinates in the electric robot coordinate system using an inverse transformation matrix. The expression for the inverse transformation matrix is ​​as follows: In the formula, It is the inverse transformation matrix. It is a rotation matrix. It is a translation vector.

3. The visual servo control method for electric robots according to claim 2, characterized in that, The steps for calculating the target position coordinates in the camera coordinate system include: Based on depth value and target location pixel coordinates From the mapping relationship, we get: Based on the camera intrinsic parameters, the pixel coordinates of the target location are... Transform to the camera coordinate system to obtain the target's position coordinates in the camera coordinate system: In the formula, These are the three-dimensional coordinates in the camera coordinate system. , , , For camera intrinsic parameters, among which , For camera focal length, , These are the x and y coordinates of the principal point of the image, respectively. This represents the distance from the camera to the target location.

4. The visual servo control method for electric robots according to claim 1, characterized in that, The termination condition is that the error between the target position pixel coordinates and the current position pixel coordinates in the electric robot coordinate system is less than or equal to a set threshold. When the termination condition is met, it indicates that the electric robot end effector has moved to the target position. The expression for the termination condition is: In the formula, As the termination condition, These are the coordinates of the current corner point. The coordinates of the corner point of the target location. To set a threshold.

5. The visual servo control method for electric robots according to claim 1, characterized in that, When the robot's end effector moves from its current position, linear interpolation is used for a smooth transition: In the formula, For the final pose of the robot's end effector, For rotation matrix, This is the pose increment.

Images