Dual-arm live-line working robot system and its autonomous and remote operation control methods

Abstract

The application discloses a double-arm hot-line work robot system and an autonomous and remote control method thereof, and relates to the technical fields of intelligent operation and maintenance of power systems and robot control. When visual detection is normal, the robot can rely on target pose recognition and operation planning to autonomously complete tasks such as alignment, clamping and unscrewing of a lightning arrester without intervention of an operator. When visual recognition is limited, the robot system can be smoothly switched to a man-machine remote operation mode, real operation scenes are synchronously mapped to a virtual environment, and the operator is presented with the real operation scenes in an immersive manner through VR glasses, so that intuitive interactive operation and fine control are realized, and thus, the real sense of presence and continuity of work are ensured. The autonomous operation and the man-machine remote operation modes are integrated, the autonomous work efficiency under normal conditions is ensured, and the operator can remotely intervene under abnormal conditions, so that work interruption is avoided.

Description

Technical Field

[0001] This invention belongs to the field of intelligent operation and maintenance and robot control technology of power systems, and particularly relates to a dual-arm live-line working robot system and its autonomous and remote operation control methods. Background Technology

[0002] In the routine operation and maintenance of power distribution lines, the inspection and disassembly of surge arresters are high-frequency and rigid tasks. This work usually involves the accurate identification of the target component, two-point symmetrical clamping, controlled torque unloading, and safe release, which places high demands on the stability of the pose estimation and the continuity of the operation process.

[0003] Current engineering practices still primarily rely on insulated bucket trucks combined with close-range manual operation, or supplemented by single-arm robot prototypes for partial replacement. However, under conditions of complex tower structures, narrow working spaces, obvious repetitive textures on equipment surfaces, and adverse weather conditions such as wind, rain, and low light, visual recognition and depth perception are easily affected by occlusion and imaging distortion, making it difficult to maintain stable pose output over long periods. Once the recognition confidence level decreases, existing systems often end with shutdown alarms or close-range manual intervention, lacking a mechanism for a smooth transition from autonomous operation to remote takeover. It is difficult to simultaneously ensure operational continuity and live-line safety boundaries. Remote operation often employs absolute pose mapping, which is highly dependent on initial alignment. Even a small deviation at the starting point can amplify accumulated errors, directly transmitting them to the end effector and causing a decrease in control accuracy. Simultaneously, this method lacks a mechanism to suppress accumulated drift, making it prone to positional and attitude shifts during prolonged operation, compromising the stability and safety of surge arrester dismantling operations. These issues collectively constrain the achievement of stable, continuous, and accurate live-line arrester dismantling operations in the combined scenarios of "high altitude, complex structures, and adverse weather conditions." Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a dual-arm live-line working robot system and its autonomous and remote operation control method, in order to solve the problems of poor visual recognition stability, lack of smooth switching between autonomous and remote operation, and difficulty in balancing operation continuity and live-line safety in existing technologies.

[0005] The technical solution of this invention is as follows:

[0006] On one hand, the present invention provides a dual-arm live-line working robot system, including a ground control operation end and a high-altitude operation execution end;

[0007] The ground control terminal is used to issue control commands to the high-altitude operation terminal through the operator in the human-machine remote operation mode;

[0008] The high-altitude operation execution terminal is installed on the live-line working platform. In autonomous operation mode, it automatically performs operation tasks on the surge arrester; in human-machine remote operation mode, it performs operation tasks on the surge arrester according to the control instructions issued by the operator.

[0009] Furthermore, the high-altitude operation execution end includes two live-line collaborative robotic arms, a first industrial control computer, two RGB cameras, an end effector, a depth camera, and a platform support;

[0010] The platform support is placed on the live-line working platform to fix a depth camera. The depth camera is used to acquire video data streams of the working area and transmit them to the first industrial control computer. The video data stream includes color frames and depth frames at different times. Two collaborative live-line working robotic arms are each equipped with an RGB camera and an end effector. The end effector includes a gripper and an electric disassembly tool. The RGB camera acquires close-up video of the surge arrester and nut area and transmits it to the first industrial control computer. The gripper and electric disassembly tool are used to perform operations on the surge arrester and its nut. In autonomous operation mode, the first industrial control computer receives and processes the video data stream of the working area, transmits the color frames from the processed video data stream back to the ground control terminal, and simultaneously detects and segments the surge arrester and nut based on the color frames to obtain an instance mask of the surge arrester and nut in the image coordinate system. Based on the depth frames, it determines the coordinates and axial distance of the target pixels in the instance mask, and then... The optimal grasping pose of the two live-line working collaborative robotic arms is obtained by solving the problem. Based on the optimal grasping pose, trajectory planning is performed for the live-line working collaborative robotic arms, generating executable robotic arm motion commands and driving the two live-line working collaborative robotic arms to perform the operation task on the surge arrester. In human-machine remote operation mode, the first industrial control computer receives color frames from the video data stream of the work area acquired by the depth camera, and simultaneously receives auxiliary color frames from the close-range operation video of the surge arrester and nut area acquired by two RGB cameras, processes them, and then forwards them to the ground control terminal via a wired network. It receives control commands from the end effector of the live-line working collaborative robotic arms sent by the high-altitude operation execution terminal, calculates the motion trajectory of each joint of the live-line working collaborative robotic arm using inverse kinematics algorithms, and drives the two live-line working collaborative robotic arms to perform the operation task on the surge arrester. It also receives button commands from the high-altitude operation execution terminal and adjusts the two live-line working collaborative robotic arms according to the button commands.

[0011] Furthermore, the ground control terminal includes a VR interactive device, an independent display screen, and a second industrial control computer;

[0012] In autonomous operation mode, the second industrial control computer visualizes the color frames in the processed video data stream through an independent display screen. In human-machine remote operation mode, it visualizes the color frames in the video data stream of the work area obtained by the depth camera and the auxiliary color frames in the close-range work video of the surge arrester and nut area obtained by the RGB camera through an independent display screen, and generates a virtual scene which is sent to VR interactive devices, including a head-mounted VR device and a VR controller. The operator wears the head-mounted VR device and uses the VR controller to output the handheld pose information and button information. The second industrial control computer parses the handheld pose information and button information, converts them into control commands and button commands at the end of the live-line collaborative robotic arm, and sends them to the high-altitude work execution terminal through the network.

[0013] On the other hand, the present invention also provides an autonomous control method for a dual-arm live-line working robot system, comprising the following steps:

[0014] A1: The color frame and depth frame are acquired in real time by the depth camera and sent to the first industrial control computer. The first industrial control computer aligns the color frame and depth frame acquired by the depth camera and uses the aligned color frame and depth frame as RGB-D image data.

[0015] A2: After compressing and encoding the color frames in the RGB-D image data, the data is transmitted back to the independent display screen at the ground control terminal for visualization.

[0016] A3: In the first industrial control computer, the color frames in the RGB-D image data are input into the YOLOv11-Seg model to identify and segment the surge arrester and its nut, output the detection result and confidence level of each target, and filter the detection results with confidence levels lower than the set value; the detection result is the bounding box position, category and its instance mask in the image coordinate system;

[0017] The image coordinate system The origin is the top left corner of the color frame. , The axis is positive when it points to the right in the horizontal direction. The axis pointing vertically downwards is considered positive.

[0018] A4: Based on the instance mask and depth frame, determine the coordinates of the target pixel in the instance mask in the image coordinate system and its axial distance in the camera coordinate system; the camera coordinate system With the optical center of the depth camera as the origin , The axis pointing forward along the optical axis is considered positive. The axis is positive when it is horizontal to the right. The axis pointing vertically downwards is considered positive.

[0019] A5: Based on the axial distance of the target pixel in the camera coordinate system The pinhole model is used to backproject the coordinates of the target pixel from the image coordinate system to the camera coordinate system, thus obtaining the three-dimensional position vector of the target pixel in the camera coordinate system. ;

[0020] A6: The 3D position vector of the target pixel in the camera coordinate system Generate a point cloud, use GPD to evaluate feasible grasping on the point cloud, and combine centroid constraints to filter and obtain the optimal grasping pose. ,in This represents the rotation matrix of the optimal grasping pose in the camera coordinate system. This represents the translation vector of the optimal capture pose in the camera coordinate system. A set representing the poses of rigid bodies in three-dimensional space;

[0021] A7: Map the optimal capture pose in the camera coordinate system to the world coordinate system; the world coordinate system With the center of the live-line working platform as the origin , The axis points directly in front of the surge arrester. The axis points to the left side of the live-line working platform. The vertical axis of the live-line working platform points upwards, serving as a global reference frame for dual-arm operation and path planning;

[0022] (3);

[0023] (4);

[0024] in, To obtain the optimal grasp pose's position vector in the world coordinate system. The optimal grasping pose matrix in the world coordinate system is given. For camera external parameters, This represents the rotation matrix of the camera coordinate system relative to the world coordinate system. This represents the position vector of the optical center of the depth camera in the world coordinate system.

[0025] A8: Based on the optimal grasping posture in the world coordinate system, the trajectory of the live-line working collaborative robot arm is planned so that the live-line working collaborative robot arm can perform the operation task on the surge arrester.

[0026] Further, A4 specifically refers to: in the instance mask, the coordinates of the target pixel are as follows: The corresponding depth is ; Obtain the axial distance in the camera coordinate system , representing the actual distance of the target pixel relative to the optical center of the depth camera along the optical axis, where, This represents the horizontal coordinates of the target pixel in the image coordinate system. This represents the coordinates in the vertical direction of the image coordinate system.

[0027] Furthermore, the method for converting A5 is as follows:

[0028] (1);

[0029] (2);

[0030] in, Indicates the lower edge of the target pixel in the camera coordinate system Components in the axial direction, Indicates the lower edge of the target pixel in the camera coordinate system Components in the axial direction, Represents the camera intrinsic parameter matrix. , For camera internal parameters, This represents the equivalent focal length along the horizontal direction of the image in the camera's intrinsic parameters. This represents the equivalent focal length along the vertical direction of the image in the camera's intrinsic parameters. This indicates the horizontal offset of the optical axis in the image. This indicates the vertical offset of the optical axis in the image.

[0031] Furthermore, the A8 specifically includes the following steps:

[0032] A8.1: Define the coordinate system of the surge arrester body And solve from the lightning arrester body coordinate system Homogeneous transformation matrix to world coordinate system ;

[0033] After obtaining the optimal grasping pose in the world coordinate system, a unit vector is obtained by fitting the point cloud cylinder. The unit vector points outward from the nut, and the arrester's body coordinate system... With the geometric center of the surge arrester as the origin , The axis points along the longitudinal axis of the surge arrester towards the top of the surge arrester, and is parallel to the unit vector. Consistent; The axis is perpendicular to the arrester axis and points outward from the nut; The axis is determined by the right-hand rule;

[0034] Calculation from the lightning arrester body coordinate system The homogeneous transformation matrix to the world coordinate system is:

[0035] (9);

[0036] in, This refers to the orientation of the surge arrester's body coordinate system relative to the world coordinate system. In the coordinate system of the surge arrester body The unit vector along the axial direction, and ; In the coordinate system of the surge arrester body The unit vector along the axis, and let , express identity matrix In the world coordinate system The unit vector along the axial direction; In the coordinate system of the surge arrester body The unit vector along the axial direction, and ; Represents the coordinate system of the surge arrester body. The homogeneous transformation matrix to the world coordinate system;

[0037] A8.2: In the coordinate system of the surge arrester body Based on the optimal grasping posture, the clamping postures of two live-line cooperative robotic arms are preset with the lightning arrester body coordinate system as the reference.

[0038] (10);

[0039] (11);

[0040] in, , These represent the coordinate systems of the left and right live-line working collaborative robotic arms relative to the surge arrester body. The homogeneous transformation matrix, that is, in the lightning arrester body coordinate system The clamping posture of the two live-operation collaborative robotic arms; and This indicates the coordinate system of the left and right live-line working collaborative robotic arms relative to the lightning arrester body. The orientation and direction; and Let be the position vectors, representing the left and right end effectors of the live-line working collaborative robotic arm in the lightning arrester's body coordinate system, respectively. Spatial location;

[0041] A8.3: Based on the coordinate system of the surge arrester body Homogeneous transformation matrix to world coordinate system The clamping poses of the two live-line cooperative manipulators are transformed to the world coordinate system to obtain the target pose matrix of the end effectors of the two live-line cooperative manipulators in the world coordinate system.

[0042] (12);

[0043] (13);

[0044] in, , These are the target pose matrices of the left and right end effectors of the live-line cooperative robotic arms in the world coordinate system;

[0045] A8.4: Based on the target pose matrix of the end effectors of the two live-line cooperative manipulators in the world coordinate system, set the relative constraints between the two live-line cooperative manipulators;

[0046] (14);

[0047] in, represents the homogeneous transformation matrix of the right live-line cooperative manipulator end effector relative to the left live-line cooperative manipulator end effector; const is a constant;

[0048] A8.5: Based on the target pose matrices of the end effectors of two live-line collaborative robotic arms in the world coordinate system, in the unit vector... The direction is adjusted to allow for a predetermined approach distance and the axis is aligned to obtain the pre-approach pose;

[0049] (15);

[0050] (16);

[0051] in, This represents the target pose matrix of the left or right end effector of a live-line collaborative robotic arm in the world coordinate system. This represents the position vector of the end effector of the left or right live-line collaborative robotic arm. This represents the attitude matrix of the end effector of the left or right live-line collaborative robotic arm. This represents the target position vector of the end effector of the live-line working collaborative robotic arm in the world coordinate system during the pre-approach phase. This represents the attitude matrix of the end effector of a collaborative robotic arm used for live-line work during the pre-approach phase. To get closer; This is the attitude alignment function;

[0052] A8.6: Controlling the end effectors of two live-working collaborative robotic arms from their initial positions. sequentially approaching the pre-position With target pose Move until the distance to the surface of the surge arrester is reached. Reaching the threshold And throughout the movement, the two live-operation collaborative robotic arms must maintain relative constraints. This represents the initial attitude rotation matrix for the end effectors of two live-operation collaborative robotic arms. Let be the initial position vector of the end effectors of two live-working collaborative robotic arms;

[0053] Position and orientation interpolation during motion is:

[0054] (17);

[0055] (18);

[0056] in, Indicates the current moment. For normalized time variables, and , To plan the total time, This represents a normalized time interpolation function; Indicates at time The position vector of the end effector of the collaborative robotic arm for live-line working in the world coordinate system; The location of the target trajectory; This indicates the time of the end effector of the live-line working collaborative robotic arm. The posture quaternion; and These are the quaternions corresponding to the initial attitude and the target attitude, respectively; Represents a spherical linear interpolation function;

[0057] (19);

[0058] Apply speed limit:

[0059] (20);

[0060] in, For linear velocity, The modulus of linear velocity; Angular velocity, The magnitude of the angular velocity; This is the upper limit of the linear velocity at the end. This is the upper limit of the terminal angular velocity;

[0061] A8.7: When the surface distance between the end effectors of two live-line working collaborative robotic arms and the surge arrester... Entry threshold Inside, the automatic speed reduction is implemented, and the linear velocity of the end effectors of the two live-operation collaborative robotic arms decreases with distance until the target position is reached. ;

[0062] (twenty one);

[0063] in, Indicates at time Linear velocity of the end effector of a collaborative robotic arm for live-line work. At normal speed, This is the deceleration coefficient; This is a decreasing factor that monotonically changes with distance;

[0064] A8.8: After reaching the target position, the end effectors of the two live-line cooperative robotic arms attach and clamp the surge arrester and its nut, and make fine-tuning through superimposed force or torque closed loop during the attachment and clamping of the surge arrester and its nut.

[0065] The correction amount during the fine-tuning process is:

[0066] (twenty two);

[0067] (twenty three);

[0068] in, This represents the position correction vector. Represents the attitude correction matrix. Represents the desired force vector. The measured force vector, This is the torque error vector. Here is the position and attitude stiffness matrix. For vector antisymmetric matrix operators; Mapping to Lie group exponents;

[0069] A8.9: The nuts of the surge arrester are loosened or removed using an electrically operated disassembly tool on a live-line working collaborative robotic arm, and the torque is monitored in real time. ,when The machine will stop or reverse when the angular displacement reaches the set amount. To determine the upper limit threshold of the protection torque during the unscrewing process;

[0070] When entering the nut loosening / unloosening stage, ensure that the electric disassembly tool axis is collinear with the bolt axis, and that the angular position trajectory of the electric disassembly tool satisfies:

[0071] (twenty four);

[0072] in, This refers to the angular position of the electric disassembly tool around the bolt axis during the unscrewing process. This is the initial angular position when entering the unscrewing stage. The angular velocity of the electric disassembly tool about the bolt axis. For time variables in integration, The angular acceleration of the electric disassembly tool about the bolt axis. This is the upper limit of the permissible angular velocity during the unscrewing phase. This represents the upper limit of angular acceleration allowed during the unscrewing phase.

[0073] Another live-line working collaborative robotic arm maintains the relative constraint between the two live-line working collaborative robotic arms and uses the working gripper to hold the surge arrester;

[0074] A8.10: Based on the angular position trajectory generated during the nut tightening and loosening phase in A8.9. With end position trajectory Discretize to obtain a series of Cartesian keyframes The corresponding joint space point set is obtained by solving the inverse kinematics. Then, the resulting joint space point set Collision and singularity detection and time parameterization are performed to generate robotic arm motion commands that satisfy joint constraints and to execute the operation task using two live-working collaborative robotic arms.

[0075] (25);

[0076] (26);

[0077] in, Indicates the first Each joint at any time The position of the joint angle; The joint number; and No. Each joint at any time Angular velocity and angular acceleration; Indicates the first The maximum permissible angular velocity of each joint; Indicates the first The maximum permissible angular acceleration of each joint;

[0078] A8.11: After the operation task is completed, the two live-line cooperative robotic arms move along... The direction was withdrawn to the pre-approach position. Then return to the initial pose.

[0079] Thirdly, the present invention also provides a remote operation control method for a dual-arm live-line working robot system, comprising the following steps:

[0080] B1: When an abnormality occurs in the autonomous operation mode, the control mode switching logic is triggered. The first industrial control computer will stop sending robotic arm movement commands and send a takeover prompt to the remote operator interface of the ground control operation terminal. After takeover authority is granted, the system will switch to the human-machine remote operation mode.

[0081] B2: Real-time color frames are captured by a depth camera, and auxiliary color frames are acquired by two RGB cameras and sent to the first industrial control computer. The color frames captured in real time by the depth camera and the auxiliary color frames acquired by the two RGB cameras are then compressed and encoded and sent back to the independent display screen of the ground control terminal for visualization.

[0082] B3: The second industrial control computer creates a virtual scene based on the color frames captured in real time by the depth camera and the auxiliary color frames obtained by the two RGB cameras, and sends it to the head-mounted VR device.

[0083] B4: The operator wears a VR headset and outputs the handheld pose information through the VR handheld controller. The handheld pose information is sent to the second industrial control computer via TCP Socket and parsed to obtain the control instructions of the real live-line collaborative robotic arm.

[0084] B5: The control commands of the live-line working collaborative robotic arm are sent to the first industrial control computer at the high-altitude operation execution end. Using the inverse kinematics algorithm, the target angles of each joint of the live-line working collaborative robotic arm are calculated to form the motion trajectory of the end effector of the live-line working collaborative robotic arm, and the live-line working collaborative robotic arm is driven to perform the corresponding operation tasks.

[0085] Furthermore, B4 specifically includes the following steps:

[0086] B4.1: Initialize the VR controller and the two real live-lined collaborative robotic arms in the high-altitude operation execution end of the virtual scene;

[0087] B4.2: The operator outputs the handheld pose information in the Unity world coordinate system through the VR handheld controller and sends it to the second industrial control computer to convert the handheld pose information in the Unity world coordinate system to the robot arm base coordinate system;

[0088] The Unity world coordinate system uses the XR Origin object created in the virtual scene as its origin. , The axis points to the right side of the scene. The axis points upwards. The axis points forward;

[0089] The base coordinate system of the robotic arm is centered at the base center of the live-line working collaborative robotic arm. , The axis pointing in front of the scene is considered positive. The axis pointing to the left of the scene is considered positive. The axis pointing upwards is positive;

[0090] B4.3: Convert the handle pose information in the robot arm base coordinate system from quaternion form to Euler angle increments;

[0091] Specifically: Let the quaternion be represented as... ,in For the real part, The imaginary part has its corresponding direction cosine matrix. The expression is as follows:

[0092] (38);

[0093] Through the cosine matrix of this direction Calculate the coordinates around the robot arm base coordinate system in sequence. Rotation angle of the axis , around Rotation angle of the axis , around Rotation angle of the axis :

[0094] (39);

[0095] (40);

[0096] (41);

[0097] B4.4: Perform master-slave operation space mapping from the Unity world coordinate system to the robot arm base coordinate system, and then obtain the control commands of the real live-line working collaborative robot arm based on the handle pose information output by the VR handle controller, so that the real live-line working collaborative robot arm can perform operation actions according to the handle pose information output by the VR handle controller.

[0098] Specifically, this involves: real-time acquisition of the current frame position of the VR controller. Position relative to the previous frame Calculate displacement increment :

[0099] (44);

[0100] in, To be in the Unity world coordinate system Linear displacement increment in the axial direction. To be in the Unity world coordinate system Linear displacement increment in the axial direction. To be in the Unity world coordinate system Linear displacement increment in the axial direction;

[0101] Meanwhile, the angle increment is calculated using quaternions, and the current frame rotation of the VR controller is set as a quaternion. The previous frame rotated to The incremental quaternion is:

[0102] (45);

[0103] Increment the quaternion Perform mathematical conversion to Euler angle increments Ultimately, the control commands for the actual live-line working collaborative robotic arm are constructed in incremental form as a 6-dimensional vector. :

[0104] (46);

[0105] in, These represent the VR controller's position in the Unity world coordinate system. Displacement increment in the axial direction, These respectively represent the VR controller's rotation based on the Unity world coordinate system. The rotational increments of the axes correspond to the three-dimensional displacement and three-axis rotational increments of the end effector, respectively. This is the sensitivity coefficient;

[0106] Effective displacement of the end effector:

[0107] (29);

[0108] Effective rotation of the end effector:

[0109] (30);

[0110] in, This represents the three-dimensional displacement modulus of the end effector. These represent the positions of the end effector in the robot arm's base coordinate system. The displacement increment in the direction; The modulus representing the change in the three-axis rotation angle of the end effector. These represent the end effector's coordinates around the robot arm's base coordinate system. Rotation angles of the three coordinate axes.

[0111] Furthermore, in the human-machine remote operation mode, the button information of the VR controller is acquired in real time, including the opening and closing commands of the work gripper, pause, and operation sensitivity adjustment;

[0112] The opening and closing command of the working gripper is used to adjust the opening and closing of the working gripper;

[0113] The pause is used to suspend the movement of the live-line working collaborative robotic arm;

[0114] The sensitivity adjustment is used to dynamically adjust the control sensitivity coefficient. When the sensitivity coefficient A value of 0 indicates that the system only responds to attitude control inputs.

[0115] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0116] The dual-arm live-line working robot and its autonomous and remote operation control method proposed in this invention, when visual detection is normal (i.e., in normal operation mode), can autonomously complete tasks such as alignment, clamping, and unscrewing of surge arresters based on target pose recognition and operation planning, forming a complete autonomous task planning and execution process, achieving efficient and stable automated operation without operator intervention. When visual recognition is limited by complex weather, obstruction, or lighting conditions, the robot system can smoothly switch to human-machine remote operation mode, where the operator takes over in real time through VR terminal mapping and low-latency incremental pose control. In this mode, the real operation scene is synchronously mapped onto the virtual environment and presented to the operator immersively through VR glasses, achieving intuitive interactive operation and precise control, thus ensuring a realistic sense of presence and continuity of the operation. This method synchronizes real operation with the virtual scene and integrates autonomous operation and human-machine remote operation modes, ensuring both autonomous operation efficiency under normal conditions and remote intervention by the operator in abnormal situations to avoid operation interruption. Through the above systematic design, this invention not only overcomes the shortcomings of a single vision solution being susceptible to adverse or interfering scenarios, but also takes into account the robustness of recognition, operational continuity, and human-machine collaborative safety in high-altitude live-line scenarios without changing the essential constraints of the surge arrester disassembly process. It significantly improves the environmental adaptability and overall reliability of the dual-arm live-line operation robot, especially in improving the success rate of surge arrester disassembly operations under special weather conditions such as rain, snow, strong light, and obstruction, ensuring the continuity and safety of live-line operations in power distribution networks. Attached Figure Description

[0117] Figure 1 This is a block diagram of the overall structure of the dual-arm live-line working robot system in an embodiment of the present invention;

[0118] Figure 2 This is a schematic diagram of data transmission of the dual-arm live-line working robot system in human-machine remote operation mode in an embodiment of the present invention;

[0119] Figure 3 This is a diagram illustrating the interaction between a dual-arm live-line working robot and a surge arrester, constructed in the Unity virtual environment according to an embodiment of the present invention.

[0120] Figure 4 This is a schematic diagram of the arrester disassembly process in the autonomous operation mode of this invention.

[0121] Figure 5 This is a flowchart of the virtual reality fusion operating system in an embodiment of the present invention;

[0122] Figure 6 This is a schematic diagram of the data stream for video transmission in an embodiment of the present invention. Detailed Implementation

[0123] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0124] Example 1:

[0125] This invention proposes a dual-arm live-line working robot system for the dismantling of surge arresters in power distribution networks. This system synchronizes real-world operations with virtual scenarios and integrates both autonomous and remote human-machine operation modes. When visual recognition is limited, it can switch to remote human-machine operation mode, achieving uninterrupted dual-mode control for the work task. Figure 1 As shown, it includes a ground control operation terminal and a high-altitude operation execution terminal;

[0126] The ground control terminal is used to issue control commands to the high-altitude operation terminal through the operator in the human-machine remote operation mode;

[0127] The high-altitude operation execution terminal is installed on the live-line working platform. In autonomous operation mode, it automatically performs operation tasks on the surge arrester; in human-machine remote operation mode, it performs operation tasks on the surge arrester according to the control instructions issued by the operator.

[0128] The high-altitude operation execution end includes two live-line collaborative robotic arms, a first industrial control computer, two RGB cameras, an end effector, a depth camera, and a platform support;

[0129] The platform support is placed on the live-line working platform to fix a depth camera. The depth camera is used to acquire video data streams of the working area and transmit them to the first industrial control computer. The video data stream includes color frames and depth frames at different times, providing basic data for subsequent recognition, positioning, and human-computer interaction. Two live-line working collaborative robotic arms are each equipped with an RGB camera and an end effector. The end effector includes a gripper and an electric disassembly tool. The RGB camera acquires close-up working video of the surge arrester and nut area and transmits it to the first industrial control computer. The gripper and electric disassembly tool are used to perform operations on the surge arrester and its nut. In autonomous operation mode, the first... The industrial control computer receives and processes the video data stream from the work area, transmitting the color frames from the processed video data stream back to the ground control terminal. Simultaneously, based on the color frames, it detects and segments the surge arrester and nut, obtaining an instance mask of the surge arrester and nut in the image coordinate system. Based on the depth frame, it determines the coordinates and axial distance of the target pixels in the instance mask, thereby solving for the optimal grasping pose of the two live-line collaborative robotic arms. Based on the optimal grasping pose, it performs trajectory planning for the live-line collaborative robotic arms, generating executable robotic arm motion commands, and driving the two live-line collaborative robotic arms to perform surge arrester alignment, clamping, and nut tightening / unloading operations without ground intervention. Figure 2 As shown, in the human-machine remote operation mode, the first industrial control computer receives color frames from the video data stream of the work area acquired by the depth camera, and simultaneously receives auxiliary color frames from the close-up work video of the surge arrester and nut area acquired by two RGB cameras, processes them, and then forwards them to the ground control terminal through a wired network to achieve auxiliary calibration during human-machine remote operation, increase the sense of presence, and ensure that the surge arrester and surrounding structures remain clearly visible in the image; it receives control commands from the end effector of the live-line working collaborative robot arm sent by the high-altitude work execution terminal, uses inverse kinematics algorithm to calculate the motion trajectory of each joint of the live-line working collaborative robot arm, and drives the two live-line working collaborative robot arms to perform the operation task on the surge arrester; it receives button commands and adjusts the two live-line working collaborative robot arms according to the button commands;

[0130] The ground control terminal includes a VR interactive device, an independent display screen, and a second industrial control computer.

[0131] In autonomous operation mode, the second industrial control computer visualizes the color frames in the processed video data stream through an independent display screen. In human-machine remote operation mode, it visualizes the color frames in the video data stream of the work area obtained by the depth camera and the auxiliary color frames in the close-range work video of the surge arrester and nut area obtained by the RGB camera through an independent display screen, and generates a virtual scene to send to VR interaction devices, including a head-mounted VR device and a VR controller. The operator wears the head-mounted VR device and uses the VR controller to output the handheld pose information and button information. The second industrial control computer parses the handheld pose information and button information, converts them into control commands and button commands at the end of the live-line collaborative robotic arm, and sends them to the high-altitude work execution terminal through the network.

[0132] The depth camera described in this embodiment is an Intel RealSense D435i depth camera. In terms of hardware interface, the depth camera and RGB camera are directly connected to the first industrial control computer via USB 3.0, which not only ensures real-time transmission of high-bandwidth data, but also provides continuous power to the depth camera and RGB camera. To cope with jitter and occlusion in the working environment, the system is equipped with a platform support to reduce the impact of the robotic arm's own movement on the image coverage.

[0133] This embodiment uses the Meta Quest3 virtual reality interaction device; such as Figure 3 As shown, the system constructs a 3D interactive scene between a dual-arm collaborative robot and a lightning arrester in the Unity virtual environment. The virtual robotic arm, the gripper, and the lightning arrester models are all reproduced to their actual scale. Operators can interact with the robot's posture and issue operation commands in this virtual scene through VR devices, enabling visual monitoring and interactive verification of the remote operation process, thereby improving the system's intuitiveness and sense of presence.

[0134] The autonomous and remote control method for a dual-arm live-line working robot system includes the following steps:

[0135] like Figure 4 As shown, the control method of the dual-arm live-line working robot system in autonomous operation mode is as follows:

[0136] A1: The color frame and depth frame are acquired in real time by the depth camera and sent to the first industrial control computer. The first industrial control computer aligns the color frame and depth frame acquired by the depth camera and uses the aligned color frame and depth frame as RGB-D image data.

[0137] The specific process of aligning the color frames and depth frames captured by the depth camera is as follows:

[0138] The acquired image frames, including color frames and depth frames, enter the buffer pipeline inside the first industrial control computer; the color frames are converted into BGR matrices for subsequent visual processing, and the depth frames retain the spatial distance information and are aligned with the color frames, thereby forming RGB-D image data for subsequent local visual algorithm processing;

[0139] A2: After compressing and encoding the color frames in the RGB-D image data, the data is transmitted back to the independent display screen at the ground control terminal for visualization.

[0140] In this embodiment, the color frame is compressed and encoded into JPEG format, and pushed to an independent display screen on the ground control terminal for visualization via an HTTP-MJPEG stream;

[0141] A3: In the first industrial control computer, the color frames in the RGB-D image data are input into the YOLOv11-Seg model to identify and segment the surge arrester and its nut, output the detection result and confidence level of each target, and filter the detection results with confidence levels lower than the set value; the detection result is the bounding box position, category and its instance mask in the image coordinate system;

[0142] The image coordinate system The origin is the top left corner of the color frame. , The axis is positive when it points to the right in the horizontal direction. The axis is positive when it points vertically downwards, and the unit is pixels.

[0143] A4: Based on the instance mask and depth frame, determine the coordinates of the target pixel in the instance mask in the image coordinate system and its axial distance in the camera coordinate system;

[0144] Specifically, in the example mask, the coordinates of the target pixel are: The corresponding depth is This yields the axial distance in the camera coordinate system. , representing the actual distance of the target pixel relative to the optical center of the depth camera (the origin of the camera coordinate system) along the optical axis, where, This represents the horizontal coordinates of the target pixel in the image coordinate system. Represents the coordinates in the vertical direction of the image coordinate system;

[0145] The camera coordinate system With the optical center of the depth camera as the origin , The axis pointing along the optical axis towards the scene (front) is considered positive. The axis is positive when it is horizontal to the right. The axis pointing vertically downwards is considered positive.

[0146] A5: Based on the axial distance of the target pixel in the camera coordinate system The pinhole model is used to backproject the coordinates of the target pixel from the image coordinate system to the camera coordinate system, thus obtaining the three-dimensional position vector of the target pixel in the camera coordinate system. ;

[0147] The conversion method is as follows:

[0148] (1);

[0149] (2);

[0150] in, Indicates the lower edge of the target pixel in the camera coordinate system Components in the axial direction, Indicates the lower edge of the target pixel in the camera coordinate system Components in the axial direction, Represents the camera intrinsic parameter matrix. , For camera internal parameters, This represents the equivalent focal length along the horizontal direction of the image in the camera's intrinsic parameters. This represents the equivalent focal length along the vertical direction of the image in the camera's intrinsic parameters. This indicates the horizontal offset of the optical axis in the image. This indicates the vertical offset of the optical axis in the image;

[0151] A6: The 3D position vector of the target pixel in the camera coordinate system Generate a point cloud, use GPD to evaluate feasible grasping on the point cloud, and combine centroid constraints to filter and obtain the optimal grasping pose. ,in This represents the rotation matrix of the optimal grasping pose in the camera coordinate system. This represents the translation vector of the optimal capture pose in the camera coordinate system. Represents a set of rigid body poses in three-dimensional space, with subscripts. This indicates that the optimal grab pose is defined in the camera frame.

[0152] A7: Map the optimal capture pose in the camera coordinate system to the world coordinate system;

[0153] The world coordinate system With the center of the live-line working platform as the origin , The axis points directly in front of the surge arrester. The axis points to the left side of the live-line working platform. The vertical axis of the live-line working platform points upwards, serving as a global reference frame for dual-arm operation and path planning;

[0154] (3);

[0155] (4);

[0156] in, To obtain the optimal grasp pose's position vector in the world coordinate system. The optimal grasping pose matrix in the world coordinate system is given. For camera external parameters, This represents the rotation matrix of the camera coordinate system relative to the world coordinate system. This represents the position vector of the optical center of the depth camera in the world coordinate system.

[0157] A8: As Figure 4 As shown, the trajectory planning of the live-line working collaborative robot arm is carried out according to the optimal grasping posture in the world coordinate system, so that the live-line working collaborative robot arm can perform the operation task of the surge arrester.

[0158] In this embodiment, the trajectory execution process includes four stages in sequence: first, a pre-approach operation is performed; second, alignment and clamping are completed; then, the nut is tightened or released; and finally, the safe withdrawal stage is entered.

[0159] A8.1: Define the coordinate system of the surge arrester body This serves as a local reference frame for the surge arrester to describe the relative pose and constraints of the two arms, and to solve for the coordinates of the surge arrester body. Homogeneous transformation matrix to world coordinate system ;

[0160] After completing target pose recognition and obtaining the optimal grasping pose in the world coordinate system, a unit vector is obtained by fitting the point cloud cylinder. (Pointing to the outside of the nut), the coordinate system of the surge arrester body. With the geometric center of the surge arrester as the origin , The axis is along the longitudinal axis of the surge arrester (with the unit vector). (Consistent), pointing to the upper end of the surge arrester; The axis is perpendicular to the arrester axis and points outward from the nut; The axis is determined by the right-hand rule;

[0161] (5);

[0162] (6);

[0163] in, The geometric center of the point cloud, Represents points in a point cloud. Indicates the point number. Indicates the number of points. The geometric center of the surge arrester;

[0164] The homogeneous pose matrix serves as the pose vector of the end effector of the live-line collaborative robotic arm. It is used for inverse kinematics calculation and path interpolation in the subsequent trajectory planning process, enabling automatic execution of alignment, clamping, and disassembly without human intervention.

[0165] (7);

[0166] (8);

[0167] in, Let be the homogeneous pose matrix. For transpose, These represent the lower edges of the end effector of the live-line collaborative robotic arm in the world coordinate system. , , Coordinates in the three directions of the axis; These represent the end effectors of a collaborative robotic arm for live-line work orbiting around the world coordinate system. , , The rotation angles of the three axes, together with the above six parameters, define the complete pose vector of the end effector of the live-line working collaborative robot arm in three-dimensional space;

[0168] Based on the above theory, the calculation is performed from the lightning arrester body coordinate system. Homogeneous transformation matrix to world coordinate system:

[0169] (9);

[0170] in, This refers to the orientation of the surge arrester's body coordinate system relative to the world coordinate system. In the coordinate system of the surge arrester body The unit vector along the axial direction, and ; In the coordinate system of the surge arrester body The unit vector along the axis, and let (pointing to the outside of the nut) express identity matrix In the world coordinate system The unit vector along the axial direction; In the coordinate system of the surge arrester body The unit vector along the axial direction, and ; Represents the coordinate system of the surge arrester body. The homogeneous transformation matrix to the world coordinate system;

[0171] A8.2: In the coordinate system of the surge arrester body Based on the optimal grasping posture, the clamping postures of two live-line cooperative robotic arms are preset with the lightning arrester body coordinate system as the reference.

[0172] (10);

[0173] (11);

[0174] in, , These represent the coordinate systems of the left and right live-line working collaborative robotic arms relative to the surge arrester body. The homogeneous transformation matrix, that is, in the lightning arrester body coordinate system The clamping posture of the two live-operation collaborative robotic arms; and This indicates the coordinate system of the left and right live-line working collaborative robotic arms relative to the lightning arrester body. The orientation and direction; and Let be the position vectors, representing the left and right end effectors of the live-line working collaborative robotic arm in the lightning arrester's body coordinate system, respectively. Spatial location; subscript , They represent Left (left arm) and Right (right arm) respectively.

[0175] A8.3: Based on the coordinate system of the surge arrester body Homogeneous transformation matrix to world coordinate system The clamping poses of the two live-line cooperative manipulators are transformed to the world coordinate system to obtain the target pose matrix of the end effectors of the two live-line cooperative manipulators in the world coordinate system.

[0176] (12);

[0177] (13);

[0178] in, , These are the target pose matrices of the left and right end effectors of the live-line cooperative robotic arms in the world coordinate system;

[0179] A8.4: Based on the target pose matrix of the end effectors of the two live-line cooperative manipulators in the world coordinate system, set the relative constraints between the two live-line cooperative manipulators;

[0180] (14);

[0181] in, The homogeneous transformation matrix represents the end effector of the right live-line working collaborative robot arm relative to the end effector of the left live-line working collaborative robot arm, used to describe the relative pose relationship between the two live-line working collaborative robot arms; const is a constant, indicating that the relative pose relationship remains constant throughout the operation to form an anti-torsional / anti-rotation support;

[0182] A8.5: For safe approach, based on the target pose matrices of the end effectors of the two live-operation collaborative robotic arms in the world coordinate system, in the unit vector... The direction is adjusted to allow for a predetermined approach distance and the axis is aligned to obtain the pre-approach pose;

[0183] (15);

[0184] (16);

[0185] in, This represents the target pose matrix of the left or right end effector of a live-line collaborative robotic arm in the world coordinate system. This represents the position vector of the end effector of the left or right live-line collaborative robotic arm. This represents the attitude matrix of the end effector of the left or right live-line collaborative robotic arm. This represents the target position vector of the end effector of the live-line working collaborative robotic arm in the world coordinate system during the pre-approach phase. This represents the attitude matrix of the end effector of a collaborative robotic arm used for live-line work during the pre-approach phase. To get closer; For the attitude alignment function, based on the lightning arrester body coordinate system Homogeneous transformation matrix to world coordinate system With unit vector Calculate the required attitude matrix for the end effector of the live-line working collaborative robotic arm. The unit vector of the working axis of the end effector of the live-line working collaborative robotic arm with the axial direction of the surge arrester. Alignment, the above constitutes the key target frame of the plan;

[0186] A8.6: Controlling the end effectors of two live-working collaborative robotic arms from their initial positions. sequentially approaching the pre-position With target pose Move until the distance to the surface of the surge arrester is reached. Reaching the threshold And throughout the movement, the two live-operation collaborative robotic arms must maintain relative constraints. This represents the initial attitude rotation matrix for the end effectors of two live-operation collaborative robotic arms. Let be the initial position vector of the end effectors of two live-working collaborative robotic arms;

[0187] Position and orientation interpolation during motion is:

[0188] (17);

[0189] (18);

[0190] in, Indicates the current moment. For normalized time variables, and , To plan the total time, This represents a normalized time interpolation function; Indicates at time The position vector of the end effector of the collaborative robotic arm for live-line working in the world coordinate system; The location of the target trajectory; This indicates the time of the end effector of the live-line working collaborative robotic arm. The posture quaternion; and These are the quaternions corresponding to the initial attitude and the target attitude, respectively; Represents a spherical linear interpolation function;

[0191] To obtain a smooth and controllable end-time process, this embodiment uses a fifth-order polynomial time-scaled function for the Cartesian trajectory:

[0192] (19);

[0193] Apply speed limit:

[0194] (20);

[0195] in, For linear velocity, The linear velocity modulus reflects the motion speed of the end effector of the live-line working collaborative robotic arm in Cartesian space. Angular velocity, The modulus of angular velocity reflects the magnitude of the angular velocity of the end effector of the live-line working collaborative robotic arm as a reflection of its attitude change. This is the upper limit of the linear velocity at the end. This is the upper limit of the terminal angular velocity;

[0196] A8.7: When the surface distance between the end effectors of two live-line working collaborative robotic arms and the surge arrester... Entry threshold Inside, the automatic speed reduction is implemented, and the linear velocity of the end effectors of the two live-operation collaborative robotic arms decreases with distance until the target position is reached. ;

[0197] (twenty one);

[0198] in, Indicates at time Linear velocity of the end effector of a collaborative robotic arm for live-line work. At normal speed, This is the deceleration coefficient; It is a decreasing factor that monotonically changes with distance, used to achieve natural deceleration according to distance and avoid impact and overshoot when getting close;

[0199] A8.8: After reaching the target position, the end effectors of the two live-line cooperative robotic arms attach and clamp the surge arrester and its nut, and make fine-tuning through superimposed force or torque closed loop during the attachment and clamping of the surge arrester and its nut.

[0200] The correction amount during the fine-tuning process is:

[0201] (twenty two);

[0202] (twenty three);

[0203] in, This represents the position correction vector. Represents the attitude correction matrix. Represents the desired force vector. The measured force vector, This is the torque error vector. Here is the position and attitude stiffness matrix. For vector antisymmetric matrix operators; For the Lie group exponent mapping, the small angle approximation is: This correction is superimposed on the desired trajectory with the control cycle to ensure low-force fit and stable clamping. It is superimposed on the desired trajectory in each control cycle to complete low-force fit and stable clamping.

[0204] A8.9: The nuts of the surge arrester are loosened or removed using an electrically operated disassembly tool on a live-line working collaborative robotic arm, and the torque is monitored in real time. ,when The machine will stop or reverse when the angular displacement reaches the set amount. To determine the upper limit threshold of the protection torque during the unscrewing process;

[0205] When entering the nut loosening / unloosening stage, ensure that the electric disassembly tool axis is collinear with the bolt axis, and that the angular position trajectory of the electric disassembly tool satisfies:

[0206] (twenty four);

[0207] in, This refers to the angular position of the electric disassembly tool around the bolt axis during the unscrewing process. This is the initial angular position when entering the unscrewing stage. The angular velocity of the electric disassembly tool about the bolt axis. For time variables in integration, The angular acceleration of the electric disassembly tool about the bolt axis. This is the upper limit of the permissible angular velocity during the unscrewing phase. This represents the upper limit of angular acceleration allowed during the unscrewing phase.

[0208] Another live-line working collaborative robotic arm maintains the relative constraint between the two live-line working collaborative robotic arms and uses the working gripper to hold the surge arrester, providing anti-torsional support and preventing relative slippage;

[0209] A8.10: During this process, the angular position trajectory generated during the nut tightening and loosening stage in A8.9 is used. With end position trajectory Discretize to obtain a series of Cartesian keyframes The corresponding joint space point set is obtained by solving the inverse kinematics. Then, the resulting joint space point set Collision and singularity detection and time parameterization are performed to generate robotic arm motion commands that satisfy joint constraints and to execute the operation task using two live-working collaborative robotic arms.

[0210] (25);

[0211] (26);

[0212] in, Indicates the first Each joint at any time The position of the joint angle; For the joint number; and No. Each joint at any time Angular velocity and angular acceleration; Indicates the first The maximum permissible angular velocity of each joint; Indicates the first The maximum permissible angular acceleration of each joint;

[0213] If necessary, use S-shaped acceleration limiting to reduce vibration;

[0214] A8.11: After the operation task is completed, follow... The direction was withdrawn to the pre-approach position. And return to the initial pose;

[0215] If visual confidence drops sharply, force / torque exceeds limits, or the solution fails at any stage, the system will revert to the pre-approach pose and request VR to take over, thus completing the entire trajectory planning process.

[0216] During autonomous operation mode, a task completion judgment and status monitoring step is set up. The process ends when the task is successfully completed. If any of the following situations occur and continue for a preset time, it is judged as an abnormality and a prompt or trigger is issued to switch to human-machine remote operation mode:

[0217] (1) The failure of color frame or depth frame alignment in A1 leads to unreliable visualization input; the confidence of detection or segmentation results in A3 is lower than the threshold; the point cloud quality or point cloud “efficiency” in A6 is insufficient, resulting in no effective grabbing candidates under GPD centroid constraints.

[0218] (2) Trajectory planning or inverse kinematics solution fails in A8.4; pre-approach or approach phase cannot be completed or progress is abnormal in A8.5 and A8.6; position and attitude interpolation is abnormal in A8.6 approach or fit phase; electric disassembly tool axis and bolt axis are not aligned in A8.9.

[0219] like Figure 5 As shown, in the human-machine remote operation mode, the control method of the dual-arm live-line working robot system is as follows:

[0220] B1: Trigger the control mode switching logic. The first industrial control computer will stop sending robotic arm movement commands and simultaneously send a takeover prompt to the remote operator interface of the ground control terminal. After taking over the authority, switch to the human-machine remote operation mode.

[0221] B2: Real-time color frames are captured by a depth camera, and auxiliary color frames are acquired by two RGB cameras and sent to the first industrial control computer. The color frames captured in real time by the depth camera and the auxiliary color frames acquired by the two RGB cameras are then compressed and encoded and sent back to the independent display screen of the ground control terminal for visualization.

[0222] In this embodiment, the depth frame is used only for local 3D calculations in autonomous operation mode, and is not included in visualization feedback by default in all modes. The auxiliary color frame is also converted into a BGR matrix for detailed observation and calibration during close-range operations;

[0223] To reduce backhaul bandwidth and end-to-end latency, this invention enables image data to bypass local storage in the first industrial control computer and be directly encoded and transmitted. All color frames used for visualization (including global depth camera color and two-channel end RGB) uniformly adopt... After JPEG encoding using OpenCV (quality factor approximately 95), the bandwidth is empirically estimated as follows:

[0224] (27);

[0225] in, This refers to the bandwidth of the compressed video stream, and the compression rate is... and with For example, during the transmission phase, the first industrial control computer at the high-altitude end starts the MJPEG service based on Flask (a web framework for building streaming media services) and continuously outputs the video. To ensure the stability of the transmission link, the high-altitude end decouples video encoding and data transmission tasks, schedules them in a non-blocking manner, supports reconnection after disconnection, and ensures continuous output of the video stream even in weak network environments. The video stream is then transmitted back to an independent display screen at the ground control terminal in HTTP-MJPEG format for real-time display.

[0226] A hybrid architecture is adopted for communication: the visual data return channel, composed of WebSocket and HTTP-MJPEG, and the control command transmission channel based on Socket operate independently and in parallel, without blocking each other, ensuring low latency and stability under human-machine remote operation. Figure 6 The main processing chain of the visual perception module of the core global depth camera is shown.

[0227] B3: The second industrial control computer creates a virtual scene based on the color frames captured in real time by the depth camera and the auxiliary color frames obtained by the two RGB cameras, and sends it to the head-mounted VR device.

[0228] The ground-based system reads MJPEG segments frame by frame via a C# client, decodes them into Texture2D objects (Unity engine objects carrying image data), and displays them as textures in the Unity scene. The entire transmission process employs asynchronous I / O and multi-threading decoupling, ensuring that acquisition, compression, transmission, and rendering do not interfere with each other, thus maintaining a stable end-to-end latency of approximately [missing information]. This meets the needs of immersive visual feedback in VR environments.

[0229] After decoding on the Unity side, the image is precisely fitted onto the camera view of the virtual scene. With the help of model intrinsic parameter mapping, it is possible to achieve real-time mapping of the real scene with the three-dimensional posture and displacement in virtual reality, so that the operator can obtain stable, low-latency, and spatially consistent first-person visual feedback in the VR environment.

[0230] First, by adding an XR Origin object to the Unity scene, the position and orientation of the origin in the virtual world are defined, and a transformation channel from physical space to Unity world space is provided. Based on the position of the headset, which serves as the origin of the real-world tracking reference frame, the specific pose of the controllers has a reference point. The XR Origin object, as the origin of the Unity world, can be set to coincide with the origin of the tracking reference frame, facilitating the subsequent acquisition of controller position data.

[0231] The TrackedPose Driver component in the Left / Right Controller of the XR Origin object in the scene maps the position information of real-world devices (such as VR controllers) to virtual devices in Unity, enabling the virtual scene in Unity to follow the movement of the controllers in the real scene. At the same time, this component supports using C# scripts to obtain the controller pose in the Unity world coordinate system in real time.

[0232] B4: The operator wears a VR headset and outputs the handheld pose information through the VR handheld controller. The handheld pose information is sent to the second industrial control computer via TCP Socket and parsed to obtain control commands.

[0233] B4.1: Initialize the VR controller and the two real live-lined collaborative robotic arms in the high-altitude operation execution end of the virtual scene;

[0234] B4.2: The operator outputs the handheld pose information in the Unity world coordinate system through the VR handheld controller and sends it to the second industrial control computer to convert the handheld pose information in the Unity world coordinate system to the robot arm base coordinate system;

[0235] In this embodiment, the Unity world coordinate system uses the XR Origin object created in the virtual scene as its origin. , The axis points to the right side of the scene. The axis points upwards. The axis points forward;

[0236] The base coordinate system of the robotic arm is centered at the base center of the live-line working collaborative robotic arm. , The axis pointing in front of the scene is considered positive. The axis pointing to the left of the scene is considered positive. The axis pointing upwards is positive;

[0237] The obtained handle pose information is in the Unity world coordinate system. It needs to be transformed to the robot arm base coordinate system through a rotation matrix. Since the pose in Unity exists in the form of quaternions, mathematical operations should be performed in the form of quaternions.

[0238] The basic form of quaternions is:

[0239] (28);

[0240] in, Represents quaternions, It is the real part of the quaternion. This is the imaginary part of the quaternion. It is a virtual unit;

[0241] Generally, a quaternion is used. To represent a rotation, we call it the rotation unit quaternion. The rotation unit quaternion can be used to represent a vector using the following formula: Rotate:

[0242] (29);

[0243] in, The vector after rotation. It is a vector Defined as a pure quaternion (real part is 0); It is a rotational unit quaternion Conjugate;

[0244] Since quaternion rotation is too abstract to be directly and intuitively calculated, this embodiment first solves for the rotation matrix in Euler angle form, and then controls the VR controller to move along the bottom edge of the Unity world coordinate system. The axis is moved and the actual direction of movement of the live-line working collaborative robot arm along the robot arm's base coordinate system is viewed. This method is then used for further processing. axis, Alignment of axes, thereby obtaining the rotation matrix in Euler angle form;

[0245] To use the Euler angle form of a rotation matrix on a quaternion, a mathematical transformation is required first, for rotations around a unit vector. Rotation angle In the case of a rotating unit quaternion It can be equivalently represented as:

[0246] (30);

[0247] in, Let be the unit vector along the axis of rotation. for In the Unity world coordinate system, at the lower edge Components in the axial direction, for In the Unity world coordinate system, at the lower edge Components in the axial direction, for In the Unity world coordinate system, at the lower edge Components along the axial direction; Let be the rotation angle about the axis of rotation, then the fundamental form of the quaternion satisfies:

[0248] , , , (31);

[0249] If rotation is used, Euler angles are employed. express, These are the lower edges of the Unity world coordinate system. The rotation angle of the axis, then its corresponding unit quaternion can be obtained by... - - Sequential axis rotation quaternion combinations yield:

[0250] (32);

[0251] in, These represent the lower edge of the Unity world coordinate system. Axis rotation The quaternion of the split axis rotation has the following specific form:

[0252] (33);

[0253] (34);

[0254] (35);

[0255] Then, quaternion multiplication is used to calculate the composite rotation, such as an Euler angle. For those revolving around the Unity world coordinate system Axis rotation (Right now The unit quaternion of rotation is:

[0256] (36);

[0257] Right now:

[0258] (37);

[0259] B4.3: Convert the handle pose information in the robot arm base coordinate system from quaternion form to Euler angle increments, so as to intuitively present the handle pose information in the virtual scene;

[0260] After obtaining the handle pose information in the robot arm's base coordinate system, since robot arm control commands are usually input in Euler angles, a further step is needed to convert the rotated Unity quaternion-form handle pose information into Euler angles. The order of the robot arm Euler angles used in this embodiment is as follows: - - (First, rotate in the robot arm's base coordinate system) Axis, then winding Axis, then winding (Axis rotation). Its solution sequence differs from the normal solution sequence in the specific solution process:

[0261] Let the quaternion be represented as ,in For the real part, The imaginary part has its corresponding direction cosine matrix. The expression is as follows:

[0262] (38);

[0263] Through the cosine matrix of this direction Further calculations can be performed to sequentially calculate the coordinates around the robot arm's base coordinate system. Rotation angle of the axis , around Rotation angle of the axis , around Rotation angle of the axis :

[0264] (39);

[0265] (40);

[0266] (41);

[0267] Through the above mathematical transformations, a precise mapping from quaternions to Euler angles can be achieved, which can be effectively used for attitude display, control logic judgment, and intuitive presentation of rotation angles in human-computer interaction interfaces.

[0268] B4.4: Perform master-slave operation space mapping from the Unity world coordinate system to the robot arm base coordinate system, and then obtain the actual control commands for the live-line working collaborative robot arm based on the handle pose information output by the VR handle controller. This enables the actual live-line working collaborative robot arm to perform operation actions according to the handle pose information output by the VR handle controller (to ensure that the control input from the ground control operation end can be accurately and smoothly converted into the motion commands of the live-line working collaborative robot arm at the high-altitude operation execution end).

[0269] The master-slave operation space mapping is specifically as follows:

[0270] First, the range of motion of the end effector of the live-line working collaborative robotic arm is obtained, which includes the range in the robotic arm's base coordinate system. The maximum and minimum position values ​​that can be achieved in the three directions of the axis;

[0271] Based on the motion range of the end effector of the live-line working collaborative robotic arm, a mapping algorithm is designed to convert the handle pose information input by the user at the ground control end into the corresponding position and attitude of the end effector of the live-line working collaborative robotic arm at the aerial work execution end. In the position mapping, the maximum proportional coefficient between the ground control end and the aerial work execution end in each direction is first calculated:

[0272] (42);

[0273] in, and These represent the maximum and minimum values ​​of the position component of the end effector of a live-line working collaborative robotic arm in a certain direction; and These represent the maximum and minimum values ​​of the position component of the ground control operator in that direction, respectively.

[0274] Subsequently, based on the maximum proportional coefficient, the position mapping formula between the ground control terminal and the end effector of the live-line working collaborative robotic arm can be written as:

[0275] (43);

[0276] in, , , These represent the positions of the end effector of the live-line working collaborative robotic arm along the lower edge of the robotic arm's base coordinate system. The desired target position coordinates along the axis. , , These represent the VR controllers on the ground control end in the Unity world coordinate system. Current position coordinates along the axis. , , The end effector of the live-line collaborative robotic arm is currently positioned at the lower edge of the robotic arm's base coordinate system. The actual position coordinates along the axis; , , These represent the positions of the ground control terminal and the end effector of the live-line working collaborative robotic arm in the lower coordinate system of the robotic arm. The mapping scaling factor along the axis is used to adjust the corresponding scaling factor of pose changes at both ends, and can be dynamically adjusted according to task requirements. When the scaling factor is at its maximum value, full-coverage mapping between the operating end and the execution end space is achieved; when the scaling factor is at a smaller value, fine-grained control is achieved.

[0277] Considering that the starting origin of the VR controller in the virtual environment may be offset from the actual world coordinate system of the live-line collaborative robotic arm, especially when devices such as Meta Quest 3 are restarted or XR Origin is initialized, there is an absolute pose error.

[0278] Therefore, to improve the accuracy and stability of teleoperation, the system adopts inter-frame incremental commands based on the robotic arm's base coordinate system in the reference mapping of (43), avoiding the cumulative error caused by global initial deviation. Specifically, the current frame position of the VR controller is collected in real time. Position relative to the previous frame Calculate displacement increment :

[0279] (44);

[0280] in, To be in the Unity world coordinate system Linear displacement increment in the axial direction. To be in the Unity world coordinate system Linear displacement increment in the axial direction. To be in the Unity world coordinate system Linear displacement increment in the axial direction;

[0281] Meanwhile, the angle increment is calculated using quaternions, and the current frame rotation of the VR controller is set as a quaternion. The previous frame rotated to The incremental quaternion is:

[0282] (45);

[0283] Increment the quaternion Perform mathematical conversion to Euler angle increments Ultimately, the control commands for the actual live-line working collaborative robotic arm are constructed as a 6-dimensional vector in incremental form:

[0284] (46);

[0285] in, These represent the VR controller's position in the Unity world coordinate system. Displacement increment in the axial direction, These respectively represent the VR controller's rotation based on the Unity world coordinate system. The rotational increments of the axes correspond to the three-dimensional displacement and three-axis rotational increments of the end effector, respectively. The sensitivity factor is used to adjust the position mapping size (in the Unity world, the position is usually in meters by default, while the robotic arm position command is in millimeters by default).

[0286] Effective displacement of the end effector:

[0287] (29);

[0288] Effective rotation of the end effector:

[0289] (30);

[0290] in, This represents the three-dimensional displacement modulus of the end effector. These represent the positions of the end effector in the robot arm's base coordinate system. The displacement increment in the direction; The modulus representing the change in the three-axis rotation angle of the end effector. These represent the end effector's coordinates around the robot arm's base coordinate system. The rotation angle increments (roll, pitch, and yaw angles) of the three coordinate axes; the control cycle is set to... The position change is lower than Or the change in posture is lower than Instructions will be filtered out to suppress noise input caused by slight hand tremors.

[0291] B5: The control commands of the live-line working collaborative robotic arm are sent to the first industrial control computer at the high-altitude operation execution end. Using the inverse kinematics algorithm, the target angles of each joint of the live-line working collaborative robotic arm are calculated to form the motion trajectory of the end effector of the live-line working collaborative robotic arm. The live-line working collaborative robotic arm is then driven to perform the corresponding operation tasks, realizing real-time tracking of the ground control operation end, thereby completing posture synchronization and remote operation control.

[0292] In this embodiment, the second industrial control computer parses and formats the received control commands and forwards them to the first industrial control computer. After receiving the commands, the first industrial control computer unpacks them and uses an inverse kinematics algorithm to convert the control commands (pose increments) into the motion trajectory of the end effector of the live-line working collaborative robotic arm.

[0293] In addition, in the human-machine remote operation mode, the button information of the VR controller is obtained in real time, including the opening and closing commands of the work gripper, pause, operation sensitivity adjustment and posture control only.

[0294] In this embodiment, the opening and closing commands of the gripper are independently analyzed. This separate design avoids mutual interference between posture control and gripping action, enabling the robot to maintain stable movement and precise gripping at the same time.

[0295] When performing grasping tasks, the collaborative robotic arm for live-line work requires real-time cross-platform communication and synchronous control. The system uses a Socket communication mechanism to complete the command interaction between the Unity virtual scene and the Python control terminal. This method enables real-time sending of grasping commands from the virtual end, and the robotic arm receives, parses, and executes the corresponding grasping actions, ensuring the timeliness and synchronization of the grasping process. To achieve synchronous following of the virtual robotic arm to the real robotic arm, the system establishes an independent joint status feedback channel between the first industrial control computer and the Unity client. This channel periodically reads the angles of each joint of the real robotic arm at a frequency of 50-100Hz, encapsulates them into lightweight CSV format data frames, and continuously sends them via a TCP link. After receiving and parsing this data, the Unity client directly drives the corresponding joints of the virtual robotic arm, thereby achieving low-latency posture synchronization and allowing operators to more clearly view the status of each joint of the robotic arm.

[0296] This invention also integrates multiple modes such as button pause, operation sensitivity adjustment, and posture-only control to improve the stability and security of system operation, allowing users to flexibly control system status and response rate during task grasping. This function can be switched solely through a combination of controller buttons without relying on additional hardware interaction. The invention triggers different control logics by listening to button inputs from the left and right VR controllers.

[0297] The pause is achieved by pressing the X button on the left handle to switch the system status flag. This flag will determine whether to skip the processing of position and attitude data in each frame of the running logic, thereby achieving the effect of pausing the system response. When the pause is lifted, the invention resets the pose increment array by resetting the last frame of the pause, thereby avoiding the problem of excessive pose increment before and after the pause, which may cause the robotic arm to shake violently or directly report an error.

[0298] The sensitivity can be adjusted dynamically using a combination of the Trigger key and button. At the same time, the sensitivity coefficient can be adjusted to 0 so that the system only responds to the control input of the posture, which is suitable for operation tasks in pure rotation grasping or space-constrained environments.

Claims

1. A dual-arm live-line work robot system, characterized by, The ground control operation end and the high-altitude operation execution end are provided. The ground control operation end is used to issue operation instructions to the high-altitude operation execution end in the man-machine remote operation mode. The high-altitude operation execution end is arranged on the live working platform, and automatically executes the operation task on the lightning arrester in the autonomous operation mode, and executes the operation task on the lightning arrester according to the operation instruction issued by the operator in the man-machine remote operation mode. The autonomous operation mode specifically comprises the following steps: A1: real-time collection of color frames and depth frames by a depth camera and sending of the color frames and the depth frames to a first industrial computer, alignment of the color frames and the depth frames collected by the depth camera by the first industrial computer, and taking the aligned color frames and depth frames as RGB-D image data; A2: compression encoding of the color frames in the RGB-D image data and return of the color frames to an independent display screen of the ground control operation end for visualization; A3: input of the color frames in the RGB-D image data into a YOLOv11-Seg model in the first industrial computer for identification and segmentation of the lightning arrester and its nut, output of a detection result and a confidence of each target, and filtering of the detection result with a confidence lower than a set value; the detection result is a bounding box position, a category and an instance mask in an image coordinate system; The image coordinate system with the upper left corner of the color frame as the origin , The axis is positive along the horizontal direction to the right, The axis is positive along the vertical direction downward; A4: determining, according to the instance mask and the depth frame, coordinates of a target pixel point in the instance mask in an image coordinate system and an axial distance of the target pixel point in the camera coordinate system; the camera coordinate system , with the depth camera optical center as the origin , , the axis is positive along the optical axis pointing forward, , the axis is positive horizontally to the right, , the axis is positive vertically downward; A5: axial distance of the target pixel point in the camera coordinate system , the coordinates of the target pixel point are back-projected from the image coordinate system to the camera coordinate system by using a pinhole model to obtain a three-dimensional position vector of the target pixel point in the camera coordinate system ; A6: a three-dimensional position vector of the target pixel point in the camera coordinate system A point cloud is generated, a feasible grasp is evaluated on the point cloud using the GPD, and a centroid constraint is combined to screen, to obtain an optimal grasp pose wherein represents a rotation matrix of the optimal grasp pose in the camera coordinate system, represents a translation vector of the optimal grasp pose in the camera coordinate system, represents a set of rigid body poses in a three-dimensional space; A7: mapping the optimal grasp pose in the camera coordinate system to the world coordinate system; the world coordinate system , with the center of the live working platform as the origin , the axis points to the front of the lightning arrester, the axis points to the left side of the live working platform, the axis is perpendicular to the live working platform upwards, as a global reference system for double-arm operation and path planning; （3）； （4）； wherein, is a position vector of the optimal grasp pose in the world coordinate system, is a pose matrix of the optimal grasp pose in the world coordinate system, is a camera extrinsic, denotes a rotation matrix of the camera coordinate system relative to the world coordinate system, denotes a position vector of the depth camera optical center in the world coordinate system; A8: trajectory planning of the live working collaborative robot arm according to an optimal grasping pose in a world coordinate system, so that the live working collaborative robot arm executes the operation task on the lightning arrester; The man-machine remote operation mode specifically comprises the following steps: B1: when an abnormality occurs in the autonomous operation mode, a control mode switching logic is triggered, the first industrial computer stops sending the robot arm motion instruction, and sends a takeover prompt to a remote operator interface of the ground control operation end, after the takeover permission is obtained, the mode is switched to the man-machine remote operation mode; B2: real-time collection of color frames by a depth camera, real-time acquisition of auxiliary color frames by two RGB cameras and sending of the color frames and the auxiliary color frames to a first industrial computer, and compression encoding of the color frames and the auxiliary color frames and return of the color frames and the auxiliary color frames to an independent display screen of the ground control operation end for visualization; B3: creation of a virtual scene by the second industrial computer according to the color frames and the auxiliary color frames and sending of the virtual scene to a head-mounted VR device; B4: the operator wears the head-mounted VR device, and outputs a handle pose information through a VR handle controller, the handle pose information is sent to the second industrial computer through a TCP Socket, and a real live working collaborative robot arm control instruction is obtained by analysis; B5: the real live working collaborative robot arm control instruction is sent to the first industrial computer of the high-altitude operation execution end, a target angle of each joint of the live working collaborative robot arm is calculated by using an inverse kinematics algorithm to form a motion trajectory of a live working collaborative robot arm end effector, and the live working collaborative robot arm is driven to execute a corresponding operation task.

2. The dual-arm live working robot system of claim 1, wherein, The high-altitude operation execution end comprises two live working collaborative robot arms, a first industrial computer, two RGB cameras, an end effector, a depth camera and a platform support. The platform support is placed on the live-line work platform and used for fixing a depth camera, the depth camera is used for acquiring a video data stream of a work area and transmitting to a first industrial computer, the video data stream includes color frames and depth frames at different time points; The two live-line work collaborative mechanical arms are respectively provided with an RGB camera and an end effector at the tail end, the end effector includes a work clamp jaw and an electric dismounting tool, the RGB camera is used for acquiring a close-range work video of the arrester and nut area and transmitting to the first industrial computer, the work clamp jaw and the electric dismounting tool are used for performing operations on the arrester and nut; in the autonomous operation mode, the first industrial computer receives the video data stream of the work area and processes, returns the color frames in the processed video data stream to the ground control operation end, simultaneously detects and segments the arrester and nut based on the color frames to obtain instance masks of the arrester and nut in the image coordinate system, determines the coordinates and axial distances of target pixel points in the instance masks based on the depth frames, and then solves the optimal grasping poses of the two live-line work collaborative mechanical arms; trajectory planning of the live-line work collaborative mechanical arms is performed according to the optimal grasping poses, executable mechanical arm motion instructions are generated, and the two live-line work collaborative mechanical arms are driven to perform operation tasks on the arrester; in the man-machine remote operation mode, the first industrial computer receives the color frames in the video data stream of the work area acquired by the depth camera, simultaneously receives and processes auxiliary color frames in the close-range work video of the arrester and nut area acquired by the two RGB cameras, and then forwards to the ground control operation end through a wired network; the control instructions of the end effector of the live-line work collaborative mechanical arm sent by the high-altitude work execution end are received, the kinematics algorithm is used to calculate the motion trajectories of the joints of the live-line work collaborative mechanical arm, and the two live-line work collaborative mechanical arms are driven to perform operation tasks on the arrester; the key instruction sent by the high-altitude work execution end is received and the two live-line work collaborative mechanical arms are adjusted according to the key instruction.

3. The dual-arm live working robot system of claim 1, wherein, The ground control operation end includes a VR interaction device, an independent display screen and a second industrial computer; In the autonomous operation mode, the second industrial computer visualizes the color frames in the processed video data stream through the independent display screen; in the man-machine remote operation mode, the color frames in the video data stream of the work area acquired by the depth camera and the auxiliary color frames in the close-range work video of the arrester and nut area acquired by the RGB camera are visualized through the independent display screen, and a virtual scene is generated and sent to the VR interaction device including a head-mounted VR device and a VR handle controller; an operator wears the head-mounted VR device and uses the VR handle controller to output handle pose information and key information, the second industrial computer analyzes the handle pose information and the key information, converts them into control instructions and key instructions of the end of the live-line work collaborative mechanical arm respectively and sends them to the high-altitude work execution end through a network.

4. The dual-arm live working robot system of claim 1, wherein, The A4 is specifically: in the instance mask, the coordinate of a target pixel point is marked as , and a corresponding depth is ; an axial distance in a camera coordinate system is obtained , indicating an actual distance of the target pixel point along an optical axis direction relative to a depth camera optical center, wherein indicates a coordinate of the target pixel point in a horizontal direction of an image coordinate system, indicates a coordinate in a vertical direction of the image coordinate system.

5. The dual-arm live-line work machine system of claim 1, wherein, The conversion method of the A5 is: （1）； （2）； wherein, denotes the component of the target pixel in the camera coordinate system along the x-axis, denotes the component of the target pixel in the camera coordinate system along the y-axis, denotes the component of the target pixel in the camera coordinate system along the x-axis, denotes the component of the target pixel in the camera coordinate system along the y-axis, denotes the camera intrinsic matrix, , is the camera intrinsic, denotes the equivalent focal length in the horizontal direction of the image in the camera intrinsic, denotes the equivalent focal length in the vertical direction of the image in the camera intrinsic, denotes the horizontal shift of the optical axis on the image, denotes the vertical shift of the optical axis on the image.

6. The dual-arm live-line work machine system of claim 1, wherein, The A8 specifically includes the following steps: A8.1: Define the arrester body coordinate system and solve the homogeneous transformation matrix from the arrester body coordinate system to the world coordinate system ; After obtaining the optimal grasping pose in the world coordinate system, a unit vector is obtained by point cloud cylinder fitting , which points to the outside of the nut, and the arrester body coordinate system , which takes the geometric center of the arrester as the origin , , the axis points to the upper end of the arrester along the longitudinal axis direction of the arrester, consistent with the unit vector ; , the axis is perpendicular to the arrester axis and points to the outside of the nut; , the axis is determined by the right-hand rule; The homogeneous transformation matrix from the arrester body coordinate system to the world coordinate system is: （9）； wherein is the orientation of the lightning arrester body coordinate system with respect to the world coordinate system, is the unit vector in the direction of the axis in the lightning arrester body coordinate system, and ; is the unit vector in the direction of the axis in the lightning arrester body coordinate system, and let , denotes the identity matrix, is the unit vector in the direction of the axis in the world coordinate system; is the unit vector in the direction of the axis in the lightning arrester body coordinate system, and ; denotes the homogeneous transformation matrix from the lightning arrester body coordinate system to the world coordinate system; A8.2: In the lightning arrester body coordinate system According to the optimal grasping pose, the clamping poses of the two live working collaborative manipulators are preset in the lightning arrester body coordinate system. （10）； （11）； wherein, , and represent the homogeneous transformation matrices of the left and right live working collaborative robot arm end effectors with respect to the arrester body coordinate system , i.e. the clamping poses of the two live working collaborative robot arms within the arrester body coordinate system ; and represent the orientation directions of the left and right live working collaborative robot arm end effectors with respect to the arrester body coordinate system ; and are position vectors representing the spatial positions of the left and right live working collaborative robot arm end effectors within the arrester body coordinate system ; A8.3: According to the homogeneous transformation matrix from the lightning arrester body coordinate system to the world coordinate system , the clamping poses of the two live-line operation collaborative mechanical arms are converted to the world coordinate system to obtain the target pose matrix of the two live-line operation collaborative mechanical arms in the world coordinate system. （12）； （13）； in, , These are the target pose matrices of the left and right end effectors of the live-line cooperative robotic arms in the world coordinate system; A8.4: According to the target pose matrix of the two live-line operation collaborative manipulator end effectors in the world coordinate system, the two live-line operation collaborative manipulators are set to maintain relative constraints; （14）； wherein, H represents a homogeneous transformation matrix of the right live work collaborative robotic arm end effector with respect to the left live work collaborative robotic arm end effector; const is a constant; A8.5: respectively in two live-line work cooperative type robot end effector in the world coordinate system under the target pose matrix basis, in the unit vector The approaching distance is set in the direction of retreat, and the axis is aligned to obtain the pre-approaching pose. （15）； （16）； wherein, represents a target pose matrix of the left or right live-line work collaborative robot arm end effector in the world coordinate system, represents a position vector of the left or right live-line work collaborative robot arm end effector, represents a pose matrix of the left or right live-line work collaborative robot arm end effector; represents a target position vector of the pre-approaching stage live-line work collaborative robot arm end effector in the world coordinate system; represents a pose matrix of the pre-approaching stage live-line work collaborative robot arm end effector; is an approaching distance; is a pose alignment function; A8.6: Control the two live working collaborative robotic arm end effectors from the initial pose sequentially to the pre-approach pose to the target pose until the distance to the arrester surface reaches a threshold and the two live working collaborative robotic arms always satisfy the relative constraint during the motion, is the initial pose rotation matrix for the two live working collaborative robotic arm end effectors, is the initial position vector for the two live working collaborative robotic arm end effectors; The position and attitude interpolation in the movement process is: （17）； （18）； wherein, denotes the current time instant, is a normalized time variable, and , is the total planning time, denotes a normalized time interpolation function; denotes the position vector of the live-line work collaborative robot end effector in the world coordinate frame at time instant ; is the trajectory target position; denotes the pose quaternion of the live-line work collaborative robot end effector at time instant ; and are the quaternions corresponding to the initial pose and the target pose, respectively; denotes a spherical linear interpolation function; （19）； The upper limit of the applied acceleration is: （20）； wherein, is a linear velocity, is a linear velocity's magnitude; is an angular velocity, is an angular velocity's magnitude; is an upper limit on the end linear velocity, is an upper limit on the end angular velocity; A8.7: When two live-line collaborative robotic arm end effectors are at a distance from the surface of the arrester Entry threshold Inside, automatic speed reduction is performed, and the linear speed of the two live-line collaborative robotic arm end effectors decreases by distance until the target pose is reached ; （21）； wherein, represents the linear velocity of the live-line work cooperative manipulator end effector at time represents the linear velocity of the live-line work cooperative manipulator end effector at time is a constant motion velocity, is a deceleration coefficient; is a monotonically decreasing factor that varies with distance; A8.8: After reaching the target pose, the two live-line operation collaborative manipulator end effectors are attached and clamped to the arrester and its nut, and in the process of attaching and clamping the arrester and its nut, force or torque closed loop fine tuning is performed by superposition; The correction amount in the fine tuning process is: （22）； （23）； wherein, denotes a position correction vector, denotes a pose correction matrix, denotes a desired force vector, is a measured force vector, is a moment error vector, is a position and pose stiffness matrix, is the skew-symmetric matrix operator of a vector; is the Lie group exponential map; A8.9: Unscrewing the nuts of the surge arrester by electrically powered unscrewing tool on live-line collaborative robot arm, and real-time monitoring of torque When Or the angular displacement reaches the set amount, stop or reverse, wherein Is the upper threshold of the protection torque in the unscrewing process; When entering the nut disassembly stage, the motorized disassembly tool shaft is collinear with the bolt shaft, and the angular pose trajectory of the motorized disassembly tool satisfies: （24）； wherein is the angular position of the electric dismounting tool around the bolt axis during the screwing-off process, is the initial angular position at the beginning of the screwing-off phase, is the angular velocity of the electric dismounting tool around the bolt axis, is the time variable in the integral, is the angular acceleration of the electric dismounting tool around the bolt axis, is the upper limit of the angular velocity allowed in the screwing-off phase, is the upper limit of the angular acceleration allowed in the screwing-off phase; The other live-line operation collaborative manipulator maintains the relative constraints of the two live-line operation collaborative manipulators, and clamps the arrester by using the operation gripper; A8.10: The angular position trajectory generated in accordance with the nut unscrewing stage in A8.9 with the end position trajectory , a series of Cartesian key frames are obtained by discretization , and the corresponding joint space point set is obtained by inverse kinematics solution , and then the obtained joint space point set is subjected to collision and singularity detection and time parameterization to generate mechanical arm motion instructions satisfying joint constraints and execute operation tasks using two live working collaborative mechanical arms; （25）； （26）； in, Indicates the first Each joint at any time The position of the joint angle; For the joint number; and No. Each joint at any time Angular velocity and angular acceleration; Indicates the first The maximum permissible angular velocity of each joint; Indicates the first The maximum permissible angular acceleration of each joint; A8.11: After the operation task is completed, the two live working collaborative manipulators are withdrawn in the direction to the pre-approach pose and return to the initial pose.

7. The dual-arm live-line work machine system of claim 1, wherein, The B4 specifically comprises the following steps: B4.1: Initialize the VR handle controller in the virtual scene and the real two live-line operation collaborative manipulators in the high-altitude operation execution end; B4.2: The operator outputs the handle pose information in the Unity world coordinate system through the VR handle controller and sends it to the second industrial computer, and converts the handle pose information in the Unity world coordinate system to the manipulator base coordinate system; The Unity world coordinate system has its origin at the XR Origin object established in the virtual scene , The X axis points to the right of the scene, The Y axis points upwards, The Z axis points forward; The base coordinate system of the mechanical arm takes the center of the base of the live-line work collaborative mechanical arm as the origin , The axis points forward to the scene and is positive, The axis points to the left of the scene and is positive, The axis points upward and is positive; B4.3: Convert the handle pose information in the manipulator base coordinate system from quaternion form to Euler angle increment; Specifically, let the quaternion be represented as where is the real part, is the imaginary part, and the corresponding direction cosine matrix is expressed as follows: （38）； The direction cosine matrix is calculated in order The rotation angle around the axis is calculated in order The rotation angle around the axis is calculated in order The rotation angle around the axis is calculated in order （39）； （40）； （41）； B4.4: Perform master-slave operation space mapping from the Unity world coordinate system to the manipulator base coordinate system, and then obtain the control command of the real live-line operation collaborative manipulator according to the handle pose information output by the VR handle controller, so that the real live-line operation collaborative manipulator executes the operation action according to the handle pose information output by the VR handle controller; Specifically: real-time acquisition of the current frame position of the VR handle controller and the last frame position Calculate displacement increment : （44）； wherein, is a linear displacement increment along the axis direction in the Unity world coordinate system, is a linear displacement increment along the axis direction in the Unity world coordinate system, is a linear displacement increment along the axis direction in the Unity world coordinate system. Meanwhile, the angle increment is calculated by quaternion, and the current frame rotation of the VR handle controller is set as a quaternion , the last frame rotation is , and the incremental quaternion is （45）； The quaternion increment is mathematically converted into an Euler angle increment The final real live live charged work collaborative robot control instruction is constructed as a 6-dimensional vector in the form of increment : （46）； wherein, respectively represent the displacement increment of the VR handle controller along the x, y, and z axes in the Unity world coordinate system, respectively represent the rotation increment of the VR handle controller around the x, y, and z axes based on the Unity world coordinate system, respectively represent the displacement increment of the VR handle controller along the x, y, and z axes in the Unity world coordinate system, respectively represent the rotation increment of the VR handle controller around the x, y, and z axes based on the Unity world coordinate system, are sensitivity coefficients; The effective displacement of the end effector: （29）； The effective rotation of the end effector: （30）； wherein, denotes the three-dimensional displacement module length of the end effector, denotes the displacement increment of the end effector along the directions in the base coordinate system of the robot arm, respectively; denotes the three-axis rotation angle change amount module length of the end effector, denotes the rotation angle of the end effector around the three coordinate axes of the base coordinate system of the robot arm, respectively.

8. The dual-arm live-line work machine system of claim 1, wherein, In the human-machine remote operation mode, the key information of the VR handle controller is obtained in real time, including the operation gripper opening and closing instruction, the pause and the operation sensitivity adjustment; The operation gripper opening and closing instruction is used to adjust the opening and closing of the operation gripper; The pause is used to pause the action of the live-line operation collaborative manipulator; The sensitivity adjustment is for dynamically adjusting a control sensitivity coefficient ; and when the sensitivity coefficient is 0, only a posture control input is responded to.

Images