A robot autonomous inspection and live working cooperation method

Abstract

The embodiment of the application relates to the technical field of robots, and particularly relates to a robot autonomous inspection and live working cooperation method, which comprises the following steps: a robot is loaded with a sensor, a high-precision map containing three-dimensional geometry and semantic information is constructed in a multi-modal close-coupling mode; path planning is carried out based on the map and navigation is performed to a target screen cabinet; a target element is identified and its image coordinates are determined; a relative pose of a mechanical arm end and the target element is obtained through pose solution; a natural language instruction is acquired; the instruction is disassembled into an atomic action sequence in combination with semantic information and the relative pose; an end tool is switched according to the sequence to execute live operation, and a remaining action sequence is dynamically adjusted; force sense data and state data are output; and remote operation and safety self-correction of force sense guidance are carried out according to the force sense data. Through the fusion of a high-precision semantic map, natural language task disassembly and force sense guidance remote operation, the robot autonomous inspection and fine live working in an extra-high voltage converter station screen cabinet are realized.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method for autonomous robot inspection and live-line work collaboration. Background Technology

[0002] Currently, the daily operation and maintenance of UHVDC converter station cabinets mainly includes two aspects: inspection and equipment operation. For inspection, existing indoor track-mounted inspection robots can move along a fixed track to the cabinet, identify the status of pressure plates, indicator lights, and switch handles by capturing images of the cabinet panels, and upload the inspection data. However, their function is limited to inspection and they cannot perform live operations on internal cabinet components such as faulty boards and pressure plates. For live-line work, distribution line live-line work robots typically use insulated bucket trucks or robot platforms to reach the work point. The robotic arm holds specialized tools, and operators remotely control the operation via video to perform tasks such as line connection and lead disconnection. However, the insulation protection of these robots is mainly for 10kV to 35kV distribution lines, which cannot meet the 2kV and above withstand voltage requirements of UHVDC converter station secondary cabinets, and the creepage distance of the robotic arm joints is insufficient. In addition, traditional teleoperated robots lack force and tactile feedback. Operators can only observe the site through a monitor and remotely control the robotic arm through a joystick. They cannot sense the contact force between the end effector and the cabinet components, resulting in high operational risks. At the same time, the end tools of existing live-line working devices have limited functions and cannot be quickly switched, making it difficult to adapt to diverse operational tasks within the cabinet, such as board insertion and removal, pressure plate deployment and retraction, button pressing, and fiber optic insertion and removal.

[0003] Therefore, how to achieve integrated collaboration between autonomous inspection and live-line work of robots in UHV converter station cabinets has become an urgent technical problem to be solved. Summary of the Invention

[0004] To address this issue, the present invention provides a collaborative method for autonomous robot inspection and live-line work, which overcomes the problem in the prior art that robots cannot perform safe and precise live-line operations on the internal components of UHV converter station cabinets.

[0005] To achieve the above objectives, the present invention provides a method for autonomous robot inspection and live-line work collaboration, comprising: Step S1: A high-precision map containing 3D geometric information and semantic annotation information is constructed and stored using a combination of offline pre-construction and online updating in a multimodal tightly coupled manner. Step S2: Based on the three-dimensional geometric information in the high-precision map, perform path planning to navigate to the target cabinet, identify the target element on the panel of the target cabinet and determine its image coordinates, and obtain the relative pose between the end effector of the robot's robotic arm and the target element based on the image coordinates through pose calculation. Step S3: Obtain the natural language command issued by the user, combine the semantic annotation information in the high-precision map and the relative pose, decompose the natural language command into an atomic action sequence, switch the end tool to perform the live operation according to the atomic action sequence, and dynamically adjust the remaining unexecuted atomic action sequence according to real-time perception and operation feedback during the live operation, while outputting force data and status data during the live operation in real time. Step S4: Perform force-guided teleoperation and safety self-correction based on the force data, upload the status data, and generate a fault report.

[0006] Further, in step S1, a high-precision map containing three-dimensional geometric information and semantic annotation information is constructed, including: The robot is equipped with multiple types of environmental perception sensors and constructs a three-dimensional environmental map through a multimodal tight coupling method; Semantic annotation is performed on the three-dimensional environment map to obtain semantic annotation information, which includes the spatial location information of the cabinet and the information of the passable area.

[0007] Furthermore, a 3D environment map is constructed through a multimodal tightly coupled approach, including: The robot is equipped with a lidar, a vision camera, and an inertial measurement unit to collect 3D point cloud data, image data, and inertial data, respectively. The image data is input into a semantic segmentation network to generate a pixel-level semantic label map; The three-dimensional point cloud data collected by the lidar is spatiotemporally aligned with the semantic tag map, and the semantic tags are mapped to each point in the three-dimensional point cloud through coordinate transformation to generate a colored point cloud with semantic tags. The inertial data and the semantically labeled color point cloud are input into the graph optimization model. By minimizing the reprojection error and the inertial pre-integration residual, a three-dimensional environment map is constructed.

[0008] Further, in step S2, obtaining the relative pose between the robot's end effector and the target element includes: Step S21: Based on the three-dimensional geometric information in the high-precision map, perform path planning that combines global and local methods, and control the robot to autonomously move to the predetermined working position of the target cabinet. Step S22: Acquire the panel image of the target cabinet, identify the target element in the panel image, and determine the image coordinates of the target element in the image; Step S23: Calculate the relative pose between the end effector of the robotic arm and the target element based on the mapping relationship between image coordinates and spatial coordinates.

[0009] Further, calculating the relative pose between the robotic arm end effector and the target element includes: Establish the coordinate transformation relationship between the camera coordinate system for acquiring images, the robot coordinate system where the end effector of the robotic arm is located, and the pre-calibrated spatial coordinate system; Based on the image coordinates of the target element and the coordinate transformation relationship, the target element is mapped from the image space to the three-dimensional physical space to obtain the spatial coordinates of the target element; Based on the spatial coordinates of the target element and the current spatial coordinates of the robotic arm end effector, the relative pose between the robotic arm end effector and the target element is calculated.

[0010] Further, in step S3, the natural language instruction is decomposed into a sequence of atomic actions, including: Semantic parsing is performed on the acquired natural language instructions to extract the operation goals and intentions contained in the natural language instructions; Based on the operation objective and operation intent, combined with the semantic annotation information in the high-precision map and the relative pose, the natural language instruction is decomposed into a sequence of atomic actions to be executed sequentially.

[0011] Further, in step S3, the remaining unexecuted atomic action sequences are adjusted, including: The robot's status data during the execution of its current action is monitored in real time, and the status data is compared with a preset safety threshold to determine whether there is any abnormality in the current operation. When an anomaly is detected in the current operation, the sequence of atomic actions that have not yet been executed is adjusted according to the type of anomaly in the current operation.

[0012] Further, in step S4, force-guided teleoperation and safety self-correction are performed based on the force data, including: Step S41: Monitor the force data generated by the end effector of the robotic arm during the live operation in real time, and compare the force data with a preset safety force threshold. Step S42: When the force data exceeds the safety force threshold, a safety self-correction mechanism is triggered; Step S43, the safety self-correction mechanism includes immediately pausing the currently executing atomic action and locking the motion state of the robotic arm; Step S44: Calculate the safe retraction path of the robotic arm end effector based on the direction and magnitude of the contact force in the force data. Step S45: Control the end effector of the robotic arm to move along the safe retraction path until the force data drops below the safe force threshold, thereby releasing the abnormal contact state; Step S46: After the abnormal contact state is resolved, read the current state data of the robot; Step S47: Determine whether the conditions for automatic recovery operation are met based on the current state data of the robot. If the conditions for automatic recovery operation are met, the remaining unfinished atomic action sequence will be automatically resumed. If the conditions for automatic recovery operation are not met, switch to remote operation mode.

[0013] Further, in step S44, the safe retraction path of the robotic arm's end effector is calculated, including: The current pose of the robotic arm end effector, the spatial pose of the target element, and the direction and magnitude of the contact force in the force data are obtained at the moment when the safety self-correction mechanism is triggered. The initial retraction direction is determined based on the opposite direction of the contact force, and the initial retraction step size is calculated in conjunction with the magnitude of the contact force. Starting from the current pose of the end effector of the robotic arm, candidate retraction path points are generated along the initial retraction direction according to the initial retraction step size; Map the candidate backtracking path points onto the high-precision map, and query whether the candidate backtracking path points collide or interfere with the cabinet panels or target components in the high-precision map. If a collision or interference occurs, adjust the backtracking direction and step size, regenerate candidate backtracking path points, and continue until a collision-free backtracking path point is obtained. The path between the collision-free backtracking path point and the current pose is determined as the safe backtracking path, and the sequence of changes in the length and direction of the safe backtracking path is recorded.

[0014] Further, in step S4, the status data is uploaded to generate a fault report, including: The real-time output status data and force data during the live operation are packaged and uploaded to the remote monitoring system via wireless communication. The remote monitoring system performs time-series analysis on the received status data and, in conjunction with the time markers that trigger the security self-correction mechanism, determines the time point and operation stage of the anomaly. Based on the time point of the anomaly, the corresponding environmental semantic context information is extracted from the high-precision map. The environmental semantic context information includes the current operating cabinet identifier, the target component type, and the spatial pose of the robotic arm end effector when the anomaly occurs. The time of occurrence of the anomaly, the anomaly type, the environmental semantic context information, and the execution result of the security self-correction mechanism are associated and stored to generate a fault report.

[0015] Compared with existing technologies, the beneficial effects of this invention are that it constructs a high-precision map containing three-dimensional geometric information and semantic annotation information through multimodal sensor fusion and semantic SLAM technology. This enables accurate positioning and semantic understanding of converter station cabinets and passage areas in environments where GPS signals are denied. It provides accurate spatial prior knowledge for subsequent robot autonomous navigation, target recognition and live operation, and solves the problem that existing track-type inspection robots can only move along fixed tracks and cannot adapt to unstructured environments. This significantly improves the robot's environmental perception and operational adaptability in complex power scenarios.

[0016] Furthermore, by integrating high-precision semantic map navigation and visual servo positioning technology, this invention enables robots to perform autonomous path planning and sub-millimeter-level relative pose calculation for tiny targets on the panel of the cabinet in the absence of GPS. This solves the problems in the prior art where robots cannot autonomously move to non-fixed workstations and cannot accurately locate components inside the cabinet. It provides precise pose guidance for the subsequent robotic arm to perform refined live operations, significantly improving the robot's operational accuracy and autonomy in complex electromagnetic environments.

[0017] Furthermore, by integrating natural language semantic parsing with a real-time perception feedback mechanism, this invention enables the autonomous decomposition and dynamic adjustment of complex tasks into atomic action sequences. This solves the problems of robots lacking autonomous task planning capabilities and imperfect human-machine collaborative safety mechanisms in existing technologies, and significantly improves the robot's task understanding ability, anomaly response capability, and operational safety in complex scenarios at ultra-high voltage converter stations.

[0018] Furthermore, this invention achieves smooth contact with cabinet components by dynamically adjusting the stiffness and damping of the end tool according to the target component type and real-time contact force through adaptive impedance control. This effectively avoids impact damage to precision circuit boards, pressure plates, and other components during live operation, significantly improving operational safety and component lifespan.

[0019] Furthermore, this invention achieves autonomous safety protection and flexible human-machine collaborative control of the robot in abnormal states by using a six-dimensional force sensor to monitor contact force in real time, combining high-precision map planning for collision-free safe retreat path, and intelligent switching between automatic recovery and remote operation modes, which significantly improves the safety and reliability of live-line work at UHV converter stations.

[0020] Furthermore, this invention generates a fault report containing location annotations and image evidence by associating and storing operational status data, force data, and environmental semantic context information in a high-precision map. This solves the problems of incomplete fault information recording and lack of environmental context association in the prior art, realizes closed-loop operation and maintenance data accumulation and continuous model optimization, and provides complete data support for subsequent fault analysis, equipment maintenance, and iterative training of large models.

[0021] Furthermore, this invention achieves task allocation, path conflict resolution, and resource sharing among multiple robots through a centralized collaborative scheduler and a spatiotemporal occupancy map. This solves the problem of collisions and resource competition that easily occur when multiple robots are working in a confined cabinet space, and significantly improves the overall efficiency and safety of intelligent operation and maintenance of converter stations. Attached Figure Description

[0022] Figure 1 This is a flowchart of the robot autonomous inspection and live-line work collaboration method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the relative pose between the robot's end effector and the target component in the robot autonomous inspection and live-line work collaborative method according to an embodiment of this application. Detailed Implementation

[0023] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0024] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0025] Please see Figures 1-2 As shown, Figure 1 This is a flowchart of the robot autonomous inspection and live-line work collaboration method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the relative pose between the robot's end effector and the target component in the robot autonomous inspection and live-line work collaborative method according to an embodiment of this application.

[0026] The technical solution provided in this application includes the following steps: Step S1: A high-precision map containing 3D geometric information and semantic annotation information is constructed and stored using a combination of offline pre-construction and online updating in a multimodal tightly coupled manner. Step S2: Based on the three-dimensional geometric information in the high-precision map, perform path planning to navigate to the target cabinet, identify the target element on the panel of the target cabinet and determine its image coordinates, and obtain the relative pose between the end effector of the robot's robotic arm and the target element based on the image coordinates through pose calculation. Step S3: Obtain the natural language command issued by the user, combine the semantic annotation information in the high-precision map and the relative pose, decompose the natural language command into an atomic action sequence, switch the end tool to perform the live operation according to the atomic action sequence, and dynamically adjust the remaining unexecuted atomic action sequence according to real-time perception and operation feedback during the live operation, while outputting force data and status data during the live operation in real time. Step S4: Perform force-guided teleoperation and safety self-correction based on the force data, upload the status data, and generate a fault report.

[0027] In this embodiment of the invention, in step S1, a high-precision map is established by combining offline pre-construction and online updating: during the initial deployment or when the environment changes, the robot carries sensors to traverse the target area in a multimodal tightly coupled manner, constructs and stores a prior high-precision map containing three-dimensional geometric information and semantic annotation information; during each operation, the robot loads the prior high-precision map, perceives it in real time through multimodal sensors and registers it with the prior map to achieve relocation, and at the same time dynamically updates the local map according to environmental changes during the operation.

[0028] In this embodiment of the invention, the preset safety force threshold is determined by statistically analyzing historical operation data of the target component. Specifically, this includes: collecting contact force data from multiple manual or robotic operations on the component (such as circuit boards or pressure plates), and taking 1.2 times the maximum normal operating force as the safety force threshold; or setting it directly based on the maximum allowable operating force parameter provided by the component manufacturer, for example, if the maximum allowable pulling force for circuit board insertion and removal is 50N, then the safety force threshold is set to 50N. The preset safety threshold includes a force threshold, a visual deviation threshold, an electromagnetic threshold, and a joint torque threshold. The force threshold is determined using the aforementioned method; the visual deviation threshold is set based on the positioning accuracy requirements of the robotic arm end effector, for example, if the required positioning accuracy is ±0.5mm, then the visual deviation threshold is set to 0.5mm; the electromagnetic threshold is set based on the electromagnetic environment test data from the UHV converter station, taking 1.5 times the maximum electromagnetic intensity of the normal operating environment as the threshold; the joint torque threshold is set based on a certain proportion of the maximum allowable output torque of each joint of the robot, for example, taking 80% of the maximum allowable torque as the safety threshold.

[0029] The initial step size for retraction is determined by multiplying the contact force magnitude by a preset step size coefficient. The specific formula is: Initial step size = Contact force magnitude × Step size coefficient. The step size coefficient is determined through offline calibration. The calibration method involves: repeatedly testing the retraction success rate under different step sizes in a collision-free environment, and selecting the step size coefficient with the highest success rate as the calibration value; or using a fixed step size method, setting a fixed initial step size based on the size of the target component. For example, for a standard board, the initial step size is fixed at 5mm, and for a pressure plate, it is fixed at 2mm. The conditions for automatic recovery operation include whether multiple sub-conditions are met simultaneously. The judgment criteria for each sub-condition are determined through offline testing. The specific method includes: First, collecting state data from multiple successful automatic recovery operations after the robot has safely retracted, including end-effector pose deviation, visual positioning deviation, and joint angle deviation; then, calculating the statistical normal range of each state data, and taking the mean ± 3 times the standard deviation as the judgment criteria for meeting the conditions; finally, combining the judgment criteria of each sub-condition as the conditions for automatic recovery operation. For example, the conditions are set as follows: when the positional deviation of the robotic arm end effector is ≤2mm, the visual positioning deviation is ≤0.3mm, and the joint angle deviation is ≤1°, the conditions for automatic recovery operation are met.

[0030] Specifically, in step S1, a high-precision map containing 3D geometric information and semantic annotation information is constructed, including: The robot is equipped with multiple types of environmental perception sensors and constructs a three-dimensional environmental map through a multimodal tight coupling method; Semantic annotation is performed on the three-dimensional environment map to obtain semantic annotation information, which includes the spatial location information of the cabinet and the information of the passable area.

[0031] Specifically, constructing a 3D environment map through a multimodal tightly coupled approach includes: The robot is equipped with a lidar, a vision camera, and an inertial measurement unit to collect 3D point cloud data, image data, and inertial data, respectively. The image data is input into a semantic segmentation network to generate a pixel-level semantic label map; The three-dimensional point cloud data collected by the lidar is spatiotemporally aligned with the semantic tag map, and the semantic tags are mapped to each point in the three-dimensional point cloud through coordinate transformation to generate a colored point cloud with semantic tags. The inertial data and the semantically labeled color point cloud are input into the graph optimization model. By minimizing the reprojection error and the inertial pre-integration residual, a three-dimensional environment map is constructed.

[0032] Specifically, in this embodiment, in step S1, the high-precision map is built using a combination of offline pre-construction and online updating, specifically including: Step S11: When the robot is first deployed to the target converter station or when the converter station layout changes, the robot, equipped with multiple environmental perception sensors such as LiDAR, vision camera, and inertial measurement unit, traverses the target area along a preset inspection path. During the traversal, environmental data is collected through a multimodal tightly coupled approach: image data collected by the vision camera is input into a semantic segmentation network to generate a pixel-level semantic label map; the 3D point cloud data collected by the LiDAR is spatiotemporally aligned with the semantic label map, and the semantic labels are mapped to each point in the 3D point cloud through coordinate transformation to generate a colored point cloud with semantic labels; the inertial data collected by the inertial measurement unit and the colored point cloud with semantic labels are input into a graph optimization model, and a 3D environmental map is constructed by minimizing the reprojection error and the inertial pre-integration residual. The constructed 2D environmental map is semantically annotated to obtain semantic annotation information, which includes the spatial location information of the cabinet, the distribution information of the cabinet panel components, and the information of the passable area. The aforementioned 3D environment map, which includes 3D geometric information and semantic annotation information, is used as a priori high-precision map and stored in the robot's local memory or a cloud server.

[0033] Step S12: During each inspection or live-line operation, the robot loads the prior high-precision map from local memory or the cloud. During robot movement, it collects LiDAR point clouds, visual images, and inertial data in real time, performs multimodal registration with the prior high-precision map, calculates the robot's current precise pose on the map, and achieves relocalization. Simultaneously, it compares the real-time perceived environmental information with the prior map. When environmental changes are detected (including changes in the opening / closing status of cabinet doors, the appearance of temporary obstacles, and changes in passable areas), the corresponding area in the prior map is locally updated to generate a dynamic environmental map, which supports real-time path planning and obstacle avoidance in subsequent step S2.

[0034] In this embodiment of the invention, the robot is equipped with multiple environmental perception sensors, including LiDAR, depth camera, and inertial measurement unit (IMU), and constructs a high-precision map containing 3D geometric information and semantic annotation information through a multimodal tight coupling method. First, the robot, equipped with LiDAR, vision camera, and IMU, collects 3D point cloud data, image data, and inertial data, respectively. Specifically, the LiDAR is used to acquire accurate 3D point cloud information of the cabinet and its surrounding environment, the vision camera is used to acquire high-resolution images of the cabinet panel, and the IMU is used to acquire the robot's motion state information in real time. Second, the image data is input into a semantic segmentation network to generate pixel-level semantic label maps. This semantic segmentation network can employ a deep learning-based semantic segmentation model, capable of pixel-level classification of targets in the image, such as cabinets, pressure plates, indicator lights, buttons, and board slots, and outputs a semantic category label corresponding to each pixel.

[0035] Then, the 3D point cloud data collected by the LiDAR is spatiotemporally aligned with the semantic tag map. Semantic tags are mapped to each point in the 3D point cloud through coordinate transformation, generating a colored point cloud with semantic tags. Specifically, using the calibration parameters between the LiDAR and the camera, the semantic tags in the image coordinate system are mapped to the 3D point cloud in the LiDAR coordinate system through perspective projection transformation, ensuring that each 3D point has corresponding semantic information. Finally, the inertial data and the colored point cloud with semantic tags are input into a graph optimization model. By minimizing the reprojection error and the inertial pre-integration residual, a 3D environment map is constructed. This graph optimization model uses a tightly coupled LiDAR-vision SLAM algorithm, which can optimize the robot's pose estimation and construct a high-precision 3D environment map in real time in indoor environments where GPS signals are denied. The constructed 3D environment map not only contains geometric information about the environment but also includes information such as cabinet type, cabinet number, panel component distribution location, and restricted areas through semantic annotation, providing prior knowledge for subsequent autonomous navigation and target recognition.

[0036] This invention constructs a high-precision map containing three-dimensional geometric information and semantic annotation information through multimodal sensor fusion and semantic SLAM technology. It achieves accurate positioning and semantic understanding of converter station cabinets and passage areas in environments where GPS signals are denied. This provides accurate spatial prior knowledge for subsequent robot autonomous navigation, target recognition and live operation. It solves the problem that existing track-type inspection robots can only move along fixed tracks and cannot adapt to unstructured environments, and significantly improves the robot's environmental perception and operational adaptability in complex power scenarios.

[0037] Specifically, in step S2, obtaining the relative pose between the robot's end effector and the target element includes: Step S21: Based on the three-dimensional geometric information in the high-precision map, perform path planning that combines global and local methods, and control the robot to autonomously move to the predetermined working position of the target cabinet. Step S22: Acquire the panel image of the target cabinet, identify the target element in the panel image, and determine the image coordinates of the target element in the image; Step S23: Calculate the relative pose between the end effector of the robotic arm and the target element based on the mapping relationship between image coordinates and spatial coordinates.

[0038] Specifically, calculating the relative pose between the robotic arm's end effector and the target element includes: Establish the coordinate transformation relationship between the camera coordinate system for acquiring images, the robot coordinate system where the end effector of the robotic arm is located, and the pre-calibrated spatial coordinate system; Based on the image coordinates of the target element and the coordinate transformation relationship, the target element is mapped from the image space to the three-dimensional physical space to obtain the spatial coordinates of the target element; Based on the spatial coordinates of the target element and the current spatial coordinates of the robotic arm end effector, the relative pose between the robotic arm end effector and the target element is calculated.

[0039] In this embodiment of the invention, the robot performs autonomous navigation and target localization based on the high-precision map constructed in step S1. First, in step S21, the robot performs global and local path planning based on the three-dimensional geometric information in the high-precision map, controlling the robot to autonomously move to the predetermined working position of the target cabinet. Specifically, the robot uses a global path planning algorithm (such as the A* algorithm) to plan the globally optimal path from its current position to the target cabinet in the high-precision map, while simultaneously combining a local path planning algorithm (such as the DWA algorithm) to avoid obstacles in the environment in real time, ensuring that the robot moves safely and accurately to the predetermined position in front of the target cabinet suitable for the robotic arm to deploy its work. Second, in step S22, after the robot reaches the predetermined working position, it acquires panel images of the target cabinet through its onboard vision camera and identifies target components in the panel images, determining the image coordinates of the target components in the image. Specifically, an improved target detection algorithm is used to identify small targets such as pressure plates, indicator lights, buttons, and card slots in the cabinet panel image, outputting the pixel coordinates of each target component in the image coordinate system.

[0040] Then, in step S23, the robot calculates the relative pose between the robotic arm end effector and the target element based on the mapping relationship between image coordinates and spatial coordinates. Specifically, firstly, a coordinate transformation relationship is established between the camera coordinate system for acquiring images, the robot coordinate system where the robotic arm end effector is located, and the pre-calibrated spatial coordinate system. The relationship between the camera coordinate system and the robot coordinate system is obtained through hand-eye calibration, and the relationship between the robot coordinate system and the world coordinate system is obtained through the positioning information in the high-precision map constructed in step S1. Secondly, based on the image coordinates of the target element and the coordinate transformation relationship, the target element is mapped from image space to three-dimensional physical space to obtain the spatial coordinates of the target element. Specifically, the image coordinates are converted into direction vectors in the camera coordinate system using the camera's intrinsic parameter model, and then combined with depth information or through multi-view triangulation methods to determine the precise position of the target element in three-dimensional space. Finally, based on the spatial coordinates of the target element and the current spatial coordinates of the robotic arm end effector, the relative pose between the robotic arm end effector and the target element is calculated, including the three-dimensional position deviation and attitude deviation between the two, providing pose guidance for the subsequent precise live-line operation of the robotic arm.

[0041] This invention integrates high-precision semantic map navigation and visual servo positioning technology to achieve autonomous path planning for robots in GPS-free environments and sub-millimeter-level relative pose calculation for tiny targets on cabinet panels. It solves the problems in existing technologies where robots cannot autonomously move to non-fixed workstations and cannot accurately locate components inside cabinets. This provides precise pose guidance for subsequent refined live-line operations by robotic arms, significantly improving the robot's operational accuracy and autonomy in complex electromagnetic environments.

[0042] Specifically, in step S3, the natural language instruction is broken down into a sequence of atomic actions, including: Semantic parsing is performed on the acquired natural language instructions to extract the operation goals and intentions contained in the natural language instructions; Based on the operation objective and operation intent, combined with the semantic annotation information in the high-precision map and the relative pose, the natural language instruction is decomposed into a sequence of atomic actions to be executed sequentially.

[0043] Specifically, in step S3, the remaining unexecuted atomic action sequences are adjusted, including: The robot's status data during the execution of its current action is monitored in real time, and the status data is compared with a preset safety threshold to determine whether there is any abnormality in the current operation. When an anomaly is detected in the current operation, the sequence of atomic actions that have not yet been executed is adjusted according to the type of anomaly in the current operation.

[0044] In this embodiment of the invention, after the robot accurately locates the target component, it enters the task planning and execution phase. First, in step S3, the robot acquires the natural language command issued by the user and decomposes the command into a sequence of atomic actions. Specifically, the robot performs semantic parsing on the acquired natural language command, extracting the operation target and operation intent contained within it. For example, when the user inputs the natural language command "Replace faulty board F2 in control cabinet A03," the robot identifies the operation target as "board F2" and the operation intent as "replace" through a semantic parsing model. Then, based on the operation target and operation intent, combined with the semantic annotation information in the high-precision map constructed in step S1 and the relative pose calculated in step S2, the robot decomposes the natural language command into a sequence of atomic actions to be executed sequentially. For example, the complex task of "replace board F2" is decomposed into a series of atomic actions such as "navigate to control cabinet A03," "locate the position of board F2," "identify the status of board F2," "adsorb and grasp board F2," "unimpeded removal of the existing board," "insertion of the new board," and "verification of the replacement result." Each atomic action in the atomic action sequence is the smallest operational unit that the robot can directly execute.

[0045] Secondly, during the execution of live operations, the robot dynamically adjusts the remaining unexecuted atomic action sequences based on real-time perception and operational feedback. Specifically, in step S3, adjusting the remaining unexecuted atomic action sequences includes the following steps: the robot monitors the state data during the execution of the current action in real time, the state data including one or more of collision force data, electromagnetic field strength data, visual positioning deviation data, and joint torque data; the state data is compared with a preset safety threshold to determine whether there is an anomaly in the current operation, such as whether the contact force exceeds the preset force threshold or whether the visual positioning deviation exceeds the allowable range; when an anomaly is determined in the current operation, the robot adjusts the unexecuted atomic action sequences according to the type of anomaly in the current operation, the adjustment including replanning the remaining actions, skipping unexecuted actions, or executing a preset safety retreat strategy. For example, when the robot is performing the "adsorb and grasp board F2" action, if the negative pressure sensor detects that the suction cup negative pressure value is lower than the preset threshold, it determines that there is an anti-drop anomaly, the robot pauses the current action, adjusts the unexecuted subsequent actions such as "uniformly pull out the existing board" to "retry adsorption" or "switch to a backup adsorption tool", and triggers an audible and visual alarm.

[0046] This invention integrates natural language semantic parsing with a real-time perception feedback mechanism to achieve autonomous decomposition and dynamic adjustment of complex tasks into atomic action sequences. It solves the problems of robots lacking autonomous task planning capabilities and imperfect human-machine collaborative safety mechanisms in existing technologies, and significantly improves the robot's task understanding ability, anomaly response capability, and operational safety in complex scenarios of UHV converter stations.

[0047] Specifically, during live-line operations, the impedance parameters of the end-effector are dynamically adjusted based on the target component type and the current contact force, so that the end-effector maintains stable contact while avoiding impact damage to the component.

[0048] In this embodiment of the invention, during live-line operations, the robot dynamically adjusts the impedance parameters of the end effector based on the target component type and the current contact force. The impedance parameters include stiffness coefficient, damping coefficient, and inertia coefficient. Specifically, the robot first identifies the type of the target component being operated on (e.g., a board, pressure plate, button, or fiber optic connector), and retrieves the corresponding basic impedance parameters from a preset impedance parameter library. Then, based on the contact force data collected in real-time by a six-dimensional force sensor, the robot adjusts the target contact force of the end effector in real-time using an impedance control algorithm. When the detected contact force exceeds the target force threshold for the current operation stage, the stiffness coefficient is reduced to soften the contact; when the contact force is insufficient, the stiffness coefficient is appropriately increased to ensure operational effectiveness. For example, when performing board insertion / removal operations, the axial stiffness coefficient is set to a lower value (e.g., 200 N / m) to allow the end effector some flexibility in the insertion / removal direction, avoiding damage to the board connector caused by hard insertion / removal. When performing pressure plate insertion / removal operations, the torsional stiffness coefficient is set to a higher value (e.g., 500 N·m / rad) to ensure accurate transmission of the insertion / removal torque. Through adaptive impedance control, this embodiment achieves compliant contact between the end tool and the cabinet components, solving the problem that rigid operation in the prior art is prone to damaging precision components.

[0049] This invention achieves smooth contact with cabinet components by dynamically adjusting the stiffness and damping of the end tool according to the target component type and real-time contact force through adaptive impedance control. This effectively avoids impact damage to precision circuit boards, pressure plates and other components during live operation, and significantly improves operational safety and component lifespan.

[0050] Specifically, in step S4, force-guided teleoperation and safety self-correction based on the force data include: Step S41: Monitor the force data generated by the end effector of the robotic arm during the live operation in real time, and compare the force data with a preset safety force threshold. Step S42: When the force data exceeds the safety force threshold, a safety self-correction mechanism is triggered; Step S43, the safety self-correction mechanism includes immediately pausing the currently executing atomic action and locking the motion state of the robotic arm; Step S44: Calculate the safe retraction path of the robotic arm end effector based on the direction and magnitude of the contact force in the force data. Step S45: Control the end effector of the robotic arm to move along the safe retraction path until the force data drops below the safe force threshold, thereby releasing the abnormal contact state; Step S46: After the abnormal contact state is resolved, read the current state data of the robot; Step S47: Determine whether the conditions for automatic recovery operation are met based on the current state data of the robot. If the conditions for automatic recovery operation are met, the remaining unfinished atomic action sequence will be automatically resumed. If the conditions for automatic recovery operation are not met, switch to remote operation mode.

[0051] Specifically, in step S44, the safe retraction path of the robotic arm's end effector is calculated, including: The current pose of the robotic arm end effector, the spatial pose of the target element, and the direction and magnitude of the contact force in the force data are obtained at the moment when the safety self-correction mechanism is triggered. The initial retraction direction is determined based on the opposite direction of the contact force, and the initial retraction step size is calculated in conjunction with the magnitude of the contact force. Starting from the current pose of the end effector of the robotic arm, candidate retraction path points are generated along the initial retraction direction according to the initial retraction step size; Map the candidate backtracking path points onto the high-precision map, and query whether the candidate backtracking path points collide or interfere with the cabinet panels or target components in the high-precision map. If a collision or interference occurs, adjust the backtracking direction and step size, regenerate candidate backtracking path points, and continue until a collision-free backtracking path point is obtained. The path between the collision-free backtracking path point and the current pose is determined as the safe backtracking path, and the sequence of changes in the length and direction of the safe backtracking path is recorded.

[0052] In this embodiment of the invention, the robot monitors the force data generated by the end effector of the robotic arm during live-line operations in real time and compares the force data with a preset safety force threshold. The force data is collected in real time by a six-dimensional force sensor installed at the end effector of the robotic arm, including force components in three directions and torque components in three directions. When the force data exceeds the safety force threshold, a safety self-correction mechanism is triggered. For example, when the end effector of the robotic arm is performing a board insertion / removal operation, if the insertion / removal resistance abnormally increases beyond a preset threshold, it is determined that there is a risk of jamming or misoperation, and safety self-correction is immediately triggered. The safety self-correction mechanism includes immediately pausing the currently executing atomic action and locking the movement state of the robotic arm to prevent damage to equipment or personnel caused by misoperation under abnormal conditions.

[0053] Based on the direction and magnitude of the contact force in the force data, a safe retraction path for the robotic arm's end effector is calculated. Specifically, firstly, the current pose of the robotic arm's end effector, the spatial pose of the target element, and the direction and magnitude of the contact force in the force data are obtained at the moment the safety self-correction mechanism is triggered. Then, the initial retraction direction is determined based on the opposite direction of the contact force, and the initial retraction step size is calculated in conjunction with the magnitude of the contact force; for example, the larger the contact force, the larger the retraction step size. Next, starting from the current pose of the robotic arm's end effector, candidate retraction path points are generated along the initial retraction direction and according to the initial retraction step size. Then, the candidate retraction path points are mapped to the high-precision map, and it is checked whether the candidate retraction path points collide or interfere with the panel or target element in the high-precision map. If a collision or interference occurs, the retraction direction and step size are adjusted, and candidate retraction path points are regenerated until a collision-free retraction path point is obtained. Finally, the path between the collision-free retraction path point and the current pose is determined as the safe retraction path, and the length and direction change sequence of the safe retraction path are recorded.

[0054] The robot arm's end effector moves along the safe retraction path until the force feedback data drops below the safe force threshold, thus releasing the abnormal contact state. During the retraction process, the robot continuously monitors the force feedback data to ensure that the robot arm's end effector is no longer in abnormal contact with the cabinet and target components. After the abnormal contact state is released, the robot's current state data is read, including the current pose of the robot arm's end effector, the angles of each joint, and the progress of the operation. Based on the robot's current state data, it is determined whether the conditions for automatic recovery are met. If the conditions for automatic recovery are met (e.g., the robot arm's end effector has safely retracted to the predetermined position and the visual positioning deviation is within the allowable range), the remaining unfinished atomic motion sequence is automatically resumed. If the conditions for automatic recovery are not met, the robot switches to remote operation mode, where the operator remotely takes control of the robot via a force-haptic feedback master controller to complete subsequent operations.

[0055] This invention achieves autonomous safety protection and flexible human-machine collaborative control for robots in abnormal states by using a six-dimensional force sensor to monitor contact force in real time, combining high-precision map planning for collision-free safe retreat path, and intelligent switching between automatic recovery and remote operation modes. This significantly improves the safety and reliability of live-line work at UHV converter stations.

[0056] Specifically, in step S4, the status data is uploaded to generate a fault report, including: The real-time output status data and force data during the live operation are packaged and uploaded to the remote monitoring system via wireless communication. The remote monitoring system performs time-series analysis on the received status data and, in conjunction with the time markers that trigger the security self-correction mechanism, determines the time point and operation stage of the anomaly. Based on the time point of the anomaly, the corresponding environmental semantic context information is extracted from the high-precision map. The environmental semantic context information includes the current operating cabinet identifier, the target component type, and the spatial pose of the robotic arm end effector when the anomaly occurs. The time of occurrence of the anomaly, the anomaly type, the environmental semantic context information, and the execution result of the security self-correction mechanism are associated and stored to generate a fault report.

[0057] In this embodiment of the invention, firstly, the robot packages the real-time output status data and force data during the live operation and uploads them to a remote monitoring system via wireless communication. The status data includes robot pose data, robotic arm joint angle data, visual positioning deviation data, and monitoring data from various sensors; the force data includes contact force and torque data collected in real-time by a six-dimensional force sensor, negative pressure curve data from a negative pressure sensor, and torque curve data from a torque sensor. The packaged data includes a timestamp and an operation stage identifier for easy subsequent analysis. Secondly, the remote monitoring system performs time-series analysis on the received status data, combining the time stamp that triggers the safety self-correction mechanism to determine the time point and operation stage where the anomaly occurred. For example, the remote monitoring system analyzes the force data curve to identify the time point when the force value suddenly increases or exceeds the safety threshold, and combines this with the type of atomic action performed before that time point to determine which operation stage the anomaly occurred in, such as the "board removal stage" or the "plate engagement / disengagement contact stage."

[0058] Then, based on the time point of the anomaly, the corresponding environmental semantic context information is extracted from the high-precision map constructed in step S1. The environmental semantic context information includes the identifier of the currently operating cabinet, the type of the target component, the nominal spatial position of the target component, and the spatial pose of the robotic arm's end effector at the time of the anomaly. Since the spatial coordinates and semantic information of each cabinet and panel component are pre-labeled in the high-precision map, the corresponding environmental semantic context information can be quickly retrieved and extracted based on the robot's pose information at the time of the anomaly. Finally, the time point of the anomaly, the anomaly type (such as "collision anomaly," "electromagnetic anomaly," "visual deviation anomaly," "operation over-limit anomaly," or "anti-falling anomaly"), the environmental semantic context information, and the execution result of the safety self-correction mechanism (such as "safety rollback executed," "switched to remote operation mode," or "automatic recovery operation") are associated and stored to generate a fault report. The fault report includes a formatted document with location annotations and image evidence, where the image evidence comes from images of the cabinet panels collected by the robot during the operation, and the operation target and contact position at the time of the anomaly are marked in the images.

[0059] This invention generates a fault report containing location annotations and image evidence by associating and storing operational status data, force data, and environmental semantic context information in a high-precision map. This solves the problems of incomplete fault information recording and lack of environmental context association in the prior art, realizes closed-loop operation and maintenance data accumulation and continuous model optimization, and provides complete data support for subsequent fault analysis, equipment maintenance and iterative training of large models.

[0060] Specifically, when multiple robots are performing inspection and live-line work tasks in the converter station at the same time, a centralized collaborative scheduler is used to uniformly allocate tasks and resolve conflicts among the robots.

[0061] In this embodiment of the invention, when multiple robots simultaneously perform inspection and live-line work tasks within a converter station, a centralized collaborative scheduler uniformly allocates tasks and resolves conflicts among the robots. Specifically, the centralized collaborative scheduler is deployed in a remote monitoring system, receiving pose data, task progress, and planned path information uploaded by each robot in real time. First, the collaborative scheduler constructs a spatiotemporal occupancy map based on the current pose and task objective of each robot, discretizing the converter station space into a three-dimensional voxel grid, with each voxel grid recording the occupied time window. When a robot's planned path involves other robots occupying the same voxel grid with overlapping time windows, a spatiotemporal conflict is identified. Second, when a spatiotemporal conflict is detected, the collaborative scheduler resolves the conflict based on the priority of each task. The task priority is dynamically determined according to the task type; for example, live-line operation tasks have a higher priority than inspection tasks, and emergency fault handling tasks have a higher priority than routine maintenance tasks. Conflict resolution strategies include: having low-priority robots wait outside the conflict area until high-priority robots pass through; or replanning alternative paths for high-priority robots to bypass the conflict area; or temporarily adjusting the task execution order of low-priority robots to postpone the operation in the conflict area.

[0062] Furthermore, the collaborative scheduler also enables tool and resource sharing among multiple robots. For example, when a robot performs a board replacement task, if its end-effector tool library lacks a specific type of pick-and-place tool, the collaborative scheduler can query the tool inventory of other nearby robots and dispatch a robot carrying the tool to assist in the task, or transfer the tool through the tool docking interface between robots. Finally, the collaborative scheduler sends the task allocation results and conflict resolution instructions to each robot, and each robot dynamically adjusts its navigation path and atomic action sequence according to the received instructions. Through the above multi-robot collaborative mechanism, this embodiment achieves safe and efficient collaborative operation of multiple robots within the limited space of an UHV converter station.

[0063] This invention achieves task allocation, path conflict resolution, and resource sharing among multiple robots through a centralized collaborative scheduler and spatiotemporal occupancy map. It solves the problem of collisions and resource competition that easily occur when multiple robots are working in a small cabinet space, and significantly improves the overall efficiency and safety of intelligent operation and maintenance of converter stations.

[0064] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this application.

Claims

1. A method for collaborative robot inspection and live-line work, characterized in that, include: Step S1: A high-precision map containing 3D geometric information and semantic annotation information is constructed and stored using a combination of offline pre-construction and online updating in a multimodal tightly coupled manner. Step S2: Based on the three-dimensional geometric information in the high-precision map, perform path planning to navigate to the target cabinet, identify the target element on the panel of the target cabinet and determine its image coordinates, and obtain the relative pose between the end effector of the robot's robotic arm and the target element based on the image coordinates through pose calculation. Step S3: Obtain the natural language command issued by the user, combine the semantic annotation information in the high-precision map and the relative pose, decompose the natural language command into an atomic action sequence, switch the end tool to perform the live operation according to the atomic action sequence, and dynamically adjust the remaining unexecuted atomic action sequence according to real-time perception and operation feedback during the live operation, while outputting force data and status data during the live operation in real time. Step S4: Perform force-guided teleoperation and safety self-correction based on the force data, upload the status data, and generate a fault report.

2. The method according to claim 1, characterized in that, In step S1, a high-precision map containing 3D geometric information and semantic annotation information is constructed, including: The robot is equipped with multiple types of environmental perception sensors and constructs a three-dimensional environmental map through a multimodal tight coupling method; Semantic annotation is performed on the three-dimensional environment map to obtain semantic annotation information, which includes the spatial location information of the cabinet and the information of the passable area.

3. The method according to claim 2, characterized in that, Constructing 3D environment maps using a multimodal tightly coupled approach includes: The robot is equipped with a lidar, a vision camera, and an inertial measurement unit to collect 3D point cloud data, image data, and inertial data, respectively. The image data is input into a semantic segmentation network to generate a pixel-level semantic label map; The three-dimensional point cloud data collected by the lidar is spatiotemporally aligned with the semantic tag map, and the semantic tags are mapped to each point in the three-dimensional point cloud through coordinate transformation to generate a colored point cloud with semantic tags. The inertial data and the semantically labeled color point cloud are input into the graph optimization model. By minimizing the reprojection error and the inertial pre-integration residual, a three-dimensional environment map is constructed.

4. The method according to claim 1, characterized in that, In step S2, obtaining the relative pose between the robot's end effector and the target element includes: Step S21: Based on the three-dimensional geometric information in the high-precision map, perform path planning that combines global and local methods, and control the robot to autonomously move to the predetermined working position of the target cabinet. Step S22: Acquire the panel image of the target cabinet, identify the target element in the panel image, and determine the image coordinates of the target element in the image; Step S23: Calculate the relative pose between the end effector of the robotic arm and the target element based on the mapping relationship between image coordinates and spatial coordinates.

5. The method according to claim 4, characterized in that, Calculating the relative pose between the robotic arm's end effector and the target element includes: Establish the coordinate transformation relationship between the camera coordinate system for acquiring images, the robot coordinate system where the end effector of the robotic arm is located, and the pre-calibrated spatial coordinate system; Based on the image coordinates of the target element and the coordinate transformation relationship, the target element is mapped from the image space to the three-dimensional physical space to obtain the spatial coordinates of the target element; Based on the spatial coordinates of the target element and the current spatial coordinates of the robotic arm end effector, the relative pose between the robotic arm end effector and the target element is calculated.

6. The method according to claim 1, characterized in that, In step S3, the natural language instruction is decomposed into a sequence of atomic actions, including: Semantic parsing is performed on the acquired natural language instructions to extract the operation goals and intentions contained in the natural language instructions; Based on the operation objective and operation intent, combined with the semantic annotation information in the high-precision map and the relative pose, the natural language instruction is decomposed into a sequence of atomic actions to be executed sequentially.

7. The method according to claim 6, characterized in that, In step S3, the remaining unexecuted atomic action sequences are adjusted, including: The robot's status data during the execution of its current action is monitored in real time, and the status data is compared with a preset safety threshold to determine whether there is any abnormality in the current operation. When an anomaly is detected in the current operation, the sequence of atomic actions that have not yet been executed is adjusted according to the type of anomaly in the current operation.

8. The method according to claim 1, characterized in that, In step S4, force-guided teleoperation and safety self-correction are performed based on the force data, including: Step S41: Monitor the force data generated by the end effector of the robotic arm during the live operation in real time, and compare the force data with a preset safety force threshold. Step S42: When the force data exceeds the safety force threshold, a safety self-correction mechanism is triggered; Step S43, the safety self-correction mechanism includes immediately pausing the currently executing atomic action and locking the motion state of the robotic arm; Step S44: Calculate the safe retraction path of the robotic arm end effector based on the direction and magnitude of the contact force in the force data. Step S45: Control the end effector of the robotic arm to move along the safe retraction path until the force data drops below the safe force threshold, thereby releasing the abnormal contact state; Step S46: After the abnormal contact state is resolved, read the current state data of the robot; Step S47: Determine whether the conditions for automatic recovery operation are met based on the current state data of the robot. If the conditions for automatic recovery operation are met, the remaining unfinished atomic action sequence will be automatically resumed. If the conditions for automatic recovery operation are not met, switch to remote operation mode.

9. The method according to claim 8, characterized in that, In step S44, the safe retraction path of the robotic arm's end effector is calculated, including: The current pose of the robotic arm end effector, the spatial pose of the target element, and the direction and magnitude of the contact force in the force data are obtained at the moment when the safety self-correction mechanism is triggered. The initial retraction direction is determined based on the opposite direction of the contact force, and the initial retraction step size is calculated in conjunction with the magnitude of the contact force. Starting from the current pose of the end effector of the robotic arm, candidate retraction path points are generated along the initial retraction direction according to the initial retraction step size; Map the candidate backtracking path points onto the high-precision map, and query whether the candidate backtracking path points collide or interfere with the cabinet panels or target components in the high-precision map. If a collision or interference occurs, adjust the backtracking direction and step size, regenerate candidate backtracking path points, and continue until a collision-free backtracking path point is obtained. The path between the collision-free backtracking path point and the current pose is determined as the safe backtracking path, and the sequence of changes in the length and direction of the safe backtracking path is recorded.

10. The method according to claim 1, characterized in that, In step S4, the status data is uploaded to generate a fault report, including: The real-time output status data and force data during the live operation are packaged and uploaded to the remote monitoring system via wireless communication. The remote monitoring system performs time-series analysis on the received status data and, in conjunction with the time markers that trigger the security self-correction mechanism, determines the time point and operation stage of the anomaly. Based on the time point of the anomaly, the corresponding environmental semantic context information is extracted from the high-precision map. The environmental semantic context information includes the current operating cabinet identifier, the target component type, and the spatial pose of the robotic arm end effector when the anomaly occurs. The time of occurrence of the anomaly, the anomaly type, the environmental semantic context information, and the execution result of the security self-correction mechanism are associated and stored to generate a fault report.

Images