An operating system and method for power line inspection of humanoid robots based on multimodal large model and visual pose compensation
By using a humanoid robot system with multimodal large model and visual pose compensation, the problems of inaccurate operation and high safety risks of existing power inspection robots in high-voltage live environments have been solved. It has achieved autonomous closed-loop operation and improved the standardization and efficiency of power inspection and operation.
Patent Information
- Application Number
- CN202611603700.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing power inspection robots struggle to achieve natural language command understanding, domain knowledge task planning, autonomous navigation, station position compensation, three-dimensional fine positioning, panel semantic recognition, and contact operation in high-voltage live environments, resulting in inaccurate operation, high safety risks, and low efficiency.
A humanoid robot system based on multimodal large model and visual pose compensation is adopted, integrating LiDAR, depth camera, heterogeneous dual-arm end effector and large model architecture to achieve voice interaction, task decomposition, autonomous navigation, precise positioning and contact operation. Multimodal perception and dual-model planning improve the task success rate.
It enables autonomous closed-loop operation in high-voltage energized environments, reducing the labor intensity and safety risks for maintenance personnel, improving the standardization and efficiency of inspections and operations, and achieving a success rate of over 75%.
Smart Images

Figure CN123103700A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent operation and maintenance of power systems and special robots, specifically to an operating system and method for power inspection of primary and secondary equipment such as substations, distribution rooms, high-voltage switchgear, protection and control cabinets, control cabinets, and terminal boxes, based on a multimodal large model and visual pose compensation humanoid robot. This invention can be applied to sub-fields such as intelligent operation and maintenance of power systems, intelligent operation and maintenance equipment for substations, power inspection operation robots, embodied intelligence, and intelligent grid operation and maintenance automation. It can also be extended to industrial scenarios such as petrochemicals, rail transportation, data centers, and metallurgy where high-voltage live equipment exists and requires on-site manual operation panels. Background Technology
[0002] With the continuous advancement of smart grid construction, the number of primary and secondary equipment such as high-voltage switchgear, protection and control cabinets, control cabinets, and terminal boxes in power facilities such as substations and distribution rooms is constantly increasing. Routine inspections and on-site operations, including equipment status checks, switch opening and closing confirmations, button presses, and knob switching, have long relied on manual execution by maintenance personnel. These operations typically occur in high-voltage, energized environments or environments with potential electrical risks, requiring personnel to be close to or near the control panel to observe, confirm, operate, and record. Traditional manual methods suffer from high labor intensity, high safety risks, inconsistent operations affected by personnel condition, and insufficient standardization of inspection records. To improve maintenance efficiency and safety, the power industry is gradually introducing technologies and equipment such as inspection robots, remote video inspections, artificial intelligence recognition, and large-scale intelligent models.
[0003] Currently, there are three main types of solutions available on the market for power line inspection and operation. The first type consists of wheeled, tracked, or rail-mounted power inspection robots. These robots are typically equipped with a mobile chassis, wheeled or tracked walking mechanisms, LiDAR, visible light cameras, infrared cameras, gas sensors, temperature and humidity sensors, etc., and are used to perform tasks such as patrolling preset waypoints, reading equipment meters, detecting temperature rises, detecting abnormal noises, and environmental perception in substations or distribution rooms. This type of solution has a certain degree of mobility and visual inspection capabilities, but most models only have inspection capabilities and lack robotic arms and end effectors that can directly act on panel cabinets, let alone the ability to perform contact operations on buttons, knobs, switches, etc.
[0004] The second category is the "fixed camera + backend algorithm" remote video inspection and monitoring system. This type of solution deploys fixed cameras, smart cameras, or video sensing terminals in substations or distribution rooms. The backend platform uses video streams to identify equipment status, meter readings, analyze personnel behavior, and issue alarms for abnormal events. This solution can improve the level of remote visualization, but its operational capabilities mainly rely on video sensing and backend analysis, and it cannot replace human actions that require physical contact, such as pressing buttons, rotating knobs, or changing switch positions.
[0005] The third category is a general embodied operation scheme that directly generates actions end-to-end using a single large model. This type of scheme entrusts the processes of natural language instruction understanding, task decomposition, action planning, low-level code generation, API calls, and robot execution to the same large model or a single intelligent agent for sequential execution. While this type of scheme is attractive in general demonstration scenarios, directly migrating it to power panel inspection and operation scenarios exposes significant engineering reliability issues. Power operation tasks typically have characteristics such as long sequences, multiple objectives, strong constraints, strong safety requirements, and strong domain knowledge dependencies. As the task sequence lengthens, the single large model sequential scheme is prone to problems such as planning logic errors, API call anomalies, parameter out-of-bounds errors, reversed action order, and objective confusion, leading to a decrease in the success rate of complex operation tasks.
[0006] The existing solutions are not merely affected by external factors such as insufficient operational proficiency or personnel training, but rather by several inherent flaws at the structural, perceptual, planning, and execution levels. Specifically, the existing technology suffers from at least three concrete, verifiable, and non-operational error-related inherent flaws.
[0007] The first inherent flaw is insufficient semantic understanding and domain knowledge reasoning capabilities, preventing the system from performing a complete conversion from natural language instructions to executable task planning. Existing wheeled or tracked inspection robots typically can only execute pre-programmed instructions, such as fixed waypoints, fixed shooting points, fixed action scripts, and fixed data collection procedures. If maintenance personnel want the robot to perform tasks such as "switching the isolating switch of a certain cabinet from its local position to a remote position," "pressing the reset button on the protection and control panel," or "confirming that the position indicator of a certain circuit breaker is in the closed position," they need to manually translate the natural language operation intentions into a machine-recognizable sequence of instructions. This translation process increases the interaction burden and the possibility of human error. Power maintenance involves a large number of professional procedures, anti-misoperation interlocking logic, safety constraints, equipment manual knowledge, and operation ticket procedures. Existing systems struggle to perform context-adaptive planning based on specific scenarios, current equipment states, and safety constraints, and can only execute according to fixed procedures, failing to determine "whether an action needs to be performed" based on the current equipment state. Therefore, existing solutions have inherent deficiencies in semantic understanding, task decomposition, procedure reasoning, and state-adaptive planning.
[0008] The second inherent deficiency is the lack of contact-based operation capabilities, preventing inspection robots from truly replacing human operators in performing tasks specified in operation tickets. Wheeled, tracked, or rail-mounted power inspection robots are typically equipped with a mobile body and inspection sensors, but generally lack robotic arms and end effectors that can directly interact with the control panel. Even when equipped with a simple robotic arm, it can usually only perform limited actions such as grasping, touching, and taking pictures, making it difficult to reliably perform contact-based operations such as button pressing, knob switching, and switch opening and closing. Buttons, knobs, and switches on power distribution cabinets typically have centimeter-level effective working areas, with small gaps between adjacent components. Some components are live or adjacent to live parts, requiring force control, posture adjustment, precise positioning, and stable end-effector contact capabilities. Existing inspection robots lack heterogeneous dual-arm end effector structures and close-range fine-sensing structures adapted to power distribution cabinet panels. Therefore, they cannot directly replicate the human operation process of "walking to the correct location, checking the label, pressing the button, and turning the knob" without modifying the existing electrical cabinet and control panel, and thus cannot cover the contact-based operation tasks specified in the operation ticket. The third inherent deficiency is the insufficient dimensions of localization, state perception, and semantic recognition, making it difficult for robots to perform precise tasks in front of complex display panels. When relying solely on LiDAR SLAM (Simultaneous Localization and Mapping) for navigation, robots exhibit positioning deviations of approximately ±20cm and ±35° in complex industrial environments. Since the effective working area of buttons and knobs on display panels is typically only at the centimeter or even sub-centimeter level, navigation deviations can directly cause the robotic arm's end effector to extend beyond the working window or even touch adjacent electrified components. Conventional 2D vision or simple QR code localization can only provide planar pixel positions or identifiers, failing to directly output the target's 3D coordinates in the robot's base coordinate system, and making it difficult to directly drive the robotic arm to perform millimeter-level motion planning. Furthermore, 2D vision cannot reliably determine the current angle state of knob-like devices, causing the system to be unable to determine whether the device is already in the target position. This can lead to invalid operations such as "the device is already in the target position but still performing a rotational action," wasting time and potentially causing erroneous device status changes. In addition, the buttons and knobs on the cabinet panel look very similar, and it is difficult to distinguish their functions by shape alone. For example, "circuit breaker trip", "circuit breaker close", "local / remote", etc. It is necessary to combine the silk screen text labels on the panel for semantic recognition. Existing solutions generally do not have the semantic recognition capability of integrated detection, recognition and reading.
[0009] In summary, existing technologies struggle to simultaneously achieve natural language instruction understanding, domain knowledge task planning, autonomous navigation, position compensation, 3D fine positioning, panel semantic recognition, equipment status prediction, dual-arm contact operation, and closed-loop feedback on a single robot. Therefore, there is a need in this field for a power inspection and operation robot system and method that can replace manual labor in high-voltage environments such as substations and distribution rooms to perform a complete closed-loop operation: "understanding instructions—walking there—observing the equipment—finding the location—accurately operating." Summary of the Invention
[0010] The objective of this invention is to provide a humanoid robot power inspection operating system and method based on multimodal large model and visual pose compensation. Through the system and / or method, the invention addresses the technical problem of how to replace manual labor in high-voltage live environments such as substations and distribution rooms to perform a complete closed-loop operation of "understanding instructions - walking to the equipment - clearly seeing the equipment - finding the location - accurately operating" with a humanoid robot. This reduces the labor intensity and personal risk of maintenance personnel while ensuring safety, and improves the standardization, consistency and efficiency of inspection and operation.
[0011] In a first aspect of the invention, the aforementioned task is solved by a humanoid robot power inspection operating system based on a multimodal large model and visual pose compensation, the system comprising: The humanoid robot body carrier includes a head and neck sensor mounting bracket and a modular flange interface at the end of the forearm. The head pose perception unit includes a lidar and a depth camera mounted on the head and neck sensor mounting bracket. The lidar is configured to build an environment map and output odometry information, and the depth camera is configured to detect ArUco markers and calculate the robot's pose relative to the panel. The dual-arm local fine perception unit includes a first near-field depth camera and a second near-field depth camera, which are configured to output the three-dimensional coordinates of the target in the robot base coordinate system within the working distance. A heterogeneous dual-arm end effector unit, mounted at the modular flange interface of the forearm end, includes a two-finger gripper and a multi-finger dexterous hand. The two-finger gripper is configured to perform pressing actions, and the multi-finger dexterous hand is configured to perform twisting actions. The reasoning and voice interaction unit is equipped with a language interaction module, a visual language model, a large task decomposition model, a large coding model, and a motion planning component.
[0012] In one embodiment of the present invention, the humanoid robot body carrier further includes leg and waist joint groups, six-degree-of-freedom joint groups for both arms, head pitch joints, and onboard computing unit mounting positions.
[0013] In one embodiment of the present invention, the lidar is a three-dimensional lidar; and / or The working distance between the first close-range depth camera and the second close-range depth camera is 0.3 to 0.6 m.
[0014] In one embodiment of the present invention, the output of the task decomposition model is JSON format data, and the input of the encoding model is JSON format data.
[0015] In one embodiment of the present invention, the humanoid robot power inspection operating system based on multimodal large model and visual pose compensation further includes: The work object and auxiliary positioning identification unit includes power panel cabinets and ArUco signs.
[0016] In a second aspect of the invention, the aforementioned task is further addressed by a humanoid robot power line inspection operation method based on a multimodal large model and visual pose compensation. This method utilizes the aforementioned humanoid robot power line inspection operation method based on a multimodal large model and visual pose compensation, and includes the following steps: It receives natural language voice or text commands and generates command text. Receive the instruction text and output a standard subtask sequence; The robot is controlled to move to the preset waypoint in front of the power panel cabinet, detect the ArUco marker, use the PnP algorithm to calculate the robot's pose relative to the panel, and perform fine-tuning of the standing posture. The panel is imaged by a near-field depth camera within the working distance. The target is detected and the text label is read by a visual language model. The three-dimensional coordinates of the target in the robot base coordinate system are obtained by combining the depth map, camera intrinsic parameters and hand-eye calibration. Principal component analysis is used to estimate the current angle of the knob target. Organize the target label, category, text identifier, 3D coordinates, and knob angle into a JSON structure; and Executable code is generated based on the JSON structure and the robot API library, and the motion planning component plans the trajectory and sends it to the robot joints for execution.
[0017] In one embodiment of the present invention, receiving the instruction text and outputting a standard subtask sequence includes: The task decomposition model receives the instruction text; A large-scale task decomposition model, combined with knowledge from the power operation domain, generates a standard sub-task sequence, which includes navigation sub-tasks, positioning sub-tasks, and operation sub-tasks; and The navigation subtask, the positioning subtask, and the operation subtask are labeled with target objects, target states, and constraints.
[0018] In one embodiment of the present invention, the PnP algorithm uses known ArUco identifier geometry, camera intrinsics and head sensor extrinsics to calculate pose, and the pose fine-tuning ends after the pose deviation enters a set range.
[0019] In one embodiment of the present invention, the humanoid robot power inspection operation method based on multimodal large model and visual pose compensation further includes: reading back the joint state and camera image, verifying the target state, and broadcasting the task status through a voice synthesis module.
[0020] In a third aspect, the present invention also provides a computer-readable storage medium having stored thereon computer-readable instructions, which, when executed by a processor, perform the steps of the described method for power line inspection operation of a humanoid robot based on a multimodal large model and visual pose compensation.
[0021] The technical solution provided by this invention has the following advantages: First, a closed-loop system is constructed, encompassing voice, vision, positioning, and control. Operators can use natural language voice commands or directly input operation manuals or operation tickets. The system automatically performs voice recognition, task decomposition, autonomous navigation, visual positioning compensation, target recognition and 3D positioning, and dual-arm operation code generation and execution. The entire process requires no manual programming, enabling autonomous operation from receiving instructions to execution.
[0022] Secondly, the hardware structure boasts a high degree of integration, requiring no modification to on-site equipment. Structurally, the robot's head integrates a LiDAR and a D435 depth camera, the dual arms integrate D405 depth cameras, and the forearm end effectors integrate grippers and dexterous hands. All sensors are rigidly connected via standard brackets and modular flanges, ensuring all sensing and operational units move as an integrated unit with the robot body. Functionally, it directly replicates the human-like workflow of "walking to the designated position—checking the label—pressing the button—turning the knob," requiring no modification or addition of mechanisms to existing electrical cabinets, control panels, buttons, or knobs, resulting in zero on-site modification costs.
[0023] Third, it enables bimodal interaction of voice and text. Structurally, the microphone array and onboard speakers form a two-way voice channel, while retaining a text input interface. Functionally, it supports natural language voice commands and also supports direct import of operation manuals / operation tickets, which lowers the threshold for human-computer interaction and meets the requirements of scenarios where standardized operations must be strictly performed according to written procedures.
[0024] Fourth, the dual-model task planning architecture improves the reliability of long-sequence tasks. This invention decouples high-level task decomposition from low-level executable code generation, with the task decomposition model and the coding model handling these tasks respectively. This standardizes natural language instructions into a three-stage standard sub-task sequence of "navigation—localization—operation," and then generates executable code based on the robot API library. This structure reduces the planning error rate and abnormal API call rate in long-sequence complex tasks. In simulated power operation scenarios, the average success rate of complex operation tasks can reach 75%, and the overall average success rate can reach 82.5%, with a performance degradation of approximately 25%, significantly better than the approximately 50% performance degradation of the single-model baseline solution.
[0025] Fifth, a two-tiered visual positioning strategy combining global head localization and fine-grained arm localization enhances the alignment capability of the work window. The head camera detects pre-set ArUco markers on the panel, uses the PnP algorithm to calculate the robot's precise pose relative to the panel, and performs fine-tuning of the stance. This compensates for navigation residuals from approximately ±20cm and ±35° to ±4.2cm in distance and ±5.6° in attitude angle. The arm's near-field depth camera achieves 3D positioning accuracy of ±3.8mm in the X direction, ±2.1mm in the Y direction, and ±3.5mm in the Z direction within a working distance of 0.3–0.6m, ensuring that the robotic arm's end effector accurately lands in the work window.
[0026] Sixth, semantic-level perception and contextual decision-making of device status reduce unnecessary operations. Target detection and classification are performed using a visual language model, combined with OCR to read panel text labels, outputting a structured JSON result including target category, text identifier, 3D coordinates, and the current knob angle. For knob-type devices, a principal component analysis-based angle estimation algorithm is used to determine if the device is already in the target state; if so, the rotation action is automatically skipped. Experimental data shows that in 40% of the tests for knob tasks, the rotation step was automatically skipped because the knob was already at the target angle, saving an average of 6.5 seconds, reducing both unnecessary operation time and the risk of device malfunction.
[0027] Seventh, the heterogeneous dual-arm operating hardware structure is adapted to the operation requirements of power cabinets. The left arm's two-finger gripper corresponds to coarse actions such as clamping, pulling, and pressing, while the right arm's multi-finger dexterous hand corresponds to fine actions such as twisting and pinching. The manual operation process can be directly replicated without modifying the existing electrical cabinet and control panel, resulting in low on-site modification costs and high continuity and reliability of operations.
[0028] Eighth, reduced personnel risks and improved operational standardization. Robots handle close-range operation of high-voltage equipment, allowing maintenance personnel to issue commands and monitor from a safe distance, reducing the risk of electric shock, accidental contact, and misoperation. The system executes tasks according to a structured sub-task sequence and broadcasts the progress, improving operational traceability and standardization. Attached Figure Description
[0029] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.
[0030] Figure 1 This diagram illustrates an embodiment of the power inspection operating system for a humanoid robot based on a multimodal large model and visual pose compensation according to the present invention. Figure 2 This diagram illustrates the end-to-end control architecture of a humanoid robot power inspection operating system based on a multimodal large model and visual pose compensation, according to an embodiment of the present invention. Figure 3 A schematic diagram of the structure of a language interaction module according to an embodiment of the present invention is shown; Figure 4 This diagram illustrates a large-model dual-model task planning architecture according to an embodiment of the present invention. Figure 5 A schematic diagram of the experimental platform and power cabinet structure according to an embodiment of the present invention is shown; Figure 6 A flowchart illustrating an embodiment of the present invention of a humanoid robot power line inspection operation method based on multimodal large model and visual pose compensation is shown; and Figure 7 The diagram illustrates the success rate comparison of different methods on different tasks. Detailed Implementation
[0031] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of the embodiments of the invention. However, the invention is not limited to these specific details.
[0032] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.
[0033] In this specification, terms such as "first" and "second" are used to distinguish different components and do not indicate quantity limitations or order of structural importance. Unless otherwise stated, dimensions are expressed in millimeters, centimeters, and meters; angles in degrees; time in seconds; and success rates as percentages.
[0034] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.
[0035] In this invention, the various networks, modules, or units of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written in the form of program code into a read-only memory such as an EPROM or EEPROM of the device, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by a corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by a corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by a corresponding output device (such as a display, speaker, etc.), and so on.
[0036] This invention provides a humanoid robot power inspection operating system based on multimodal large model and visual pose compensation, which is used to perform autonomous inspection and on-site operation of power cabinets in environments such as substations and power distribution rooms. Figure 1 A schematic diagram of an operating system for a humanoid robot power inspection based on a multimodal large model and visual pose compensation, according to an embodiment of the present invention, is shown. Figure 1As shown, the humanoid robot power inspection operating system based on multimodal large model and visual pose compensation provided by the present invention includes: humanoid robot body carrier 101, head pose perception unit 102, dual-arm local fine perception unit 103, heterogeneous dual-arm end effector unit 104, and reasoning calculation and voice interaction unit 105.
[0037] The humanoid robot body carrier 101 serves as the mobile and operational body, such as the Unitree G1 humanoid robot. This body has a standing height of 1320mm, a total weight of approximately 35kg, and 29 degrees of freedom. Its structure includes leg and waist joint assemblies, six-degree-of-freedom joint assemblies for both arms, a head pitch joint, a head and neck sensor mounting bracket, modular flange interfaces at the forearm ends, and onboard computing unit mounting positions. The leg and waist joint assemblies support the robot's standing, walking, turning, and posture adjustment, enabling the robot to reach preset waypoints in front of the power supply cabinet. The six-degree-of-freedom joint assemblies for both arms adjust the position and posture of the left and right arm ends, allowing the gripper or dexterous hand to align with buttons, knobs, and other targets on the panel. The head pitch joint adjusts the pointing of the head sensor, ensuring the depth camera obtains a suitable viewing angle as the robot approaches the cabinet. The head and neck sensor mounting bracket rigidly mounts the LiDAR and depth camera, allowing them to move synchronously with the head pitch joint. The modular flange interface at the forearm end is used to install a two-finger gripper or a multi-finger dexterous hand, forming a replaceable and maintainable end-effector interface. The onboard computing unit is located in the back or chest compartment of the main body and is used to install the main controller, which communicates with the joints, sensors, and end effectors via an internal bus.
[0038] The head pose perception unit 102 is used for global perception and navigation localization, including a LiDAR and a depth camera. The LiDAR, for example, can be a 3D LiDAR. The LiDAR is used to construct an indoor environmental map of the power facility, output odometry information, and cooperate with the SLAM algorithm to achieve autonomous navigation. The depth camera, for example, can be an Intel RealSense D435 depth camera, used to acquire panel images when the robot approaches the power distribution cabinet, detect ArUco markings on the panel, and calculate the robot's pose relative to the panel using the PnP algorithm. The LiDAR and depth camera are rigidly connected to the head and neck sensor mounting bracket. Their extrinsic parameters relative to the robot's base coordinate system are calibrated and fixed once after assembly, thus ensuring the consistency of pose calculation.
[0039] The dual-arm local fine perception unit 103 is used for end-effector vision and includes a first near-field depth camera and a second near-field depth camera. The first near-field depth camera is mounted on the left forearm of the humanoid robot body, and the second near-field depth camera is mounted on the right forearm of the humanoid robot body. Both near-field depth cameras can be Intel RealSense D405 near-field depth cameras, with a working distance of 0.3 to 0.6 m. The D405 camera has high depth resolution at close range, suitable for observing small targets such as buttons and knobs. The first and second near-field depth cameras are rigidly connected to the left and right forearms via modular mounting interfaces on the forearms, changing their viewing angles with arm movements and forming a fixed hand-eye relationship with the corresponding arm ends. After hand-eye calibration, pixel coordinates can be converted into three-dimensional coordinates in the robot's base coordinate system. The two cameras observe the same work panel from different perspectives, forming redundant observation and occlusion complementarity, avoiding the target being obscured by the robotic arm itself or adjacent panel structures from a single perspective.
[0040] The heterogeneous dual-arm end effector 104 serves as a work tool, comprising a two-finger gripper and a multi-finger dexterous hand. The two-finger gripper is mounted to the end of the left forearm via a standard flange and is used for gripping, pulling, and pressing actions. The multi-finger dexterous hand is mounted to the end of the right forearm via a standard flange and is used for turning knobs and fine grasping actions. The left and right arm end effectors are structurally complementary, enabling the same robot to cover both coarse and fine movements in the operation of power distribution cabinets. This heterogeneous dual-arm structure avoids frequent tool changes between different tasks using a single end effector, improving work continuity.
[0041] The inference computing and voice interaction unit 105 serves as the brain and interaction channel, including the main controller. The main controller can utilize an NVIDIA Jetson Orin NX 16GB processor, running Ubuntu 20.04 and ROS Noetic software. Installed in the onboard computing unit mounting location, the main controller connects to the LiDAR, depth camera, first near-field depth camera, second near-field depth camera, and microphone array via Ethernet and USB 3.0 interfaces. It communicates with the 29 joints and end effectors throughout the body via an internal joint bus (CAN / EtherCAT, etc.) and drives speakers through an audio interface. The main controller houses a voice interaction module (including a recognition model and a speech synthesis model), a visual language model, a task decomposition model, a coding model, and a motion planning component. The speech recognition model converts speech commands into text, the speech synthesis model converts system status into speech announcements, the visual language model performs object detection, classification, and character recognition, the task decomposition model decomposes natural language commands into standard sub-task sequences, and the coding model generates executable code for the robot. The motion planning component, specifically the MoveIt motion planning component, is used for dual-arm trajectory planning. The controller also includes navigation and positioning control nodes. Structurally, the voice interaction module is connected to the computing unit via a microphone array (audio input) and onboard speakers (audio output), forming a two-way voice channel while retaining a direct text command input channel. In one embodiment of the invention, the output of the task decomposition model is JSON format data, and the input of the encoding model is JSON format data.
[0042] In one embodiment of the present invention, the humanoid robot power inspection operating system based on multimodal large model and visual pose compensation further includes: a work object and auxiliary positioning identification unit, which includes a power cabinet and an ArUco identifier. The work object and auxiliary positioning identification unit serves as the operated equipment and positioning reference. The power cabinet has a panel with buttons labeled with text and a rotatable knob. The buttons are used for functions such as protection reset, switch opening and closing, and equipment control, while the knob is used for functions such as local / remote switching, open / close selection, and operating mode switching. The ArUco identifier, for example, can be affixed to the lower right corner of the panel as a unified spatial reference between the robot and the panel, so that the positioning accuracy is not directly affected by the robot's accumulated navigation error. The ArUco identifier has known geometric dimensions. After the robot's head depth camera detects the identifier, the robot's pose relative to the panel can be calculated using the PnP algorithm, thereby compensating for navigation residuals.
[0043] Figure 2 This diagram illustrates the end-to-end control architecture of a humanoid robot power inspection operating system based on a multimodal large model and visual pose compensation, according to an embodiment of the present invention. Figure 2As shown, in the input layer 201, maintenance personnel input commands via voice or text, and the main controller converts the commands into command text. In the task planning layer 202, the task decomposition model decomposes the command text into executable sub-tasks, i.e., a standard sub-task sequence, which includes three stages: navigation, localization, and operation. In the navigation and posture adjustment layer 203, the navigation node controls the humanoid robot body to walk in front of the cabinet, the head depth camera detects the ArUco logo and calculates the panel pose, and the robot performs fine-tuning of its standing posture. In the precise perception layer 204, the sensors perceive the three-dimensional position of the target button / knob, obtain the three-dimensional coordinates, and estimate the angle. In the execution layer 205, the encoding model generates the motion code for the left arm gripper and the right arm dexterous hand, and the motion planning component plans the trajectory and sends it to the joints for execution.
[0044] Figure 3 A schematic diagram of the structure of a language interaction module according to an embodiment of the present invention is shown. Figure 3 This demonstrates the connections between the microphone array, speech recognition model 301, main controller, speech synthesis model 302, onboard speaker, and direct text command input channel. Natural language speech commands issued by maintenance personnel are collected by the microphone array and converted into text commands by the speech recognition model. In standardized operating scenarios, operation manuals or operation tickets can be directly input into the main controller via a text interface. The main controller calls the task decomposition model based on the command type and converts status information such as execution start, execution progress, execution confirmation, and anomaly alarms into speech through the speech synthesis model, which is then broadcast by the onboard speaker. Figure 3 The system features a dual-modal interaction channel combining voice and text, enabling it to support both on-site natural language interaction and written procedure-driven operations, meeting the traceability requirements of standardized power industry operations. This module lowers the barrier to human-machine interaction, allowing maintenance personnel to execute operational intentions without needing to understand the robot's underlying commands.
[0045] Figure 4 A schematic diagram of a large-model dual-model task planning architecture according to an embodiment of the present invention is shown. Figure 4 This section demonstrates the input-output interface relationship between the task decomposition model and the coding model. In the instruction and task decomposition section (401), the task decomposition model receives instruction text, combines it with knowledge of the power operation domain, and outputs a standard three-stage sub-task sequence of navigation, localization, and operation through a task decomposer (e.g., DeepSeek-R1). Each sub-task is labeled with its target object, target state, and constraints. In the API and execution section (402), the coding model receives the structured sub-task sequence and robot API library information. Through a code generator, it generates executable Python code, which is then used by the motion planning component to plan the trajectory before being sent to the humanoid robot's body joints and end effectors. Figure 4This architecture embodies the technical characteristics of structurally independent "semantic understanding and task decomposition" and "low-level code generation and API calls," connected through standardized sub-task sequences. This avoids potential logical inconsistencies and abnormal API calls that may occur when a single large model processes long sequences of tasks sequentially. The key to this architecture lies in the connection between the two models via a standardized JSON interface. The task decomposition model does not directly generate low-level code, and the coding model does not directly process natural language instructions. This separation of responsibilities allows each model to handle only relatively well-defined tasks. This structure reduces logical inconsistencies in long sequences of tasks and improves the stability of complex tasks.
[0046] Figure 5 A schematic diagram of the experimental platform and power distribution cabinet structure according to an embodiment of the present invention is shown. Figure 5 This demonstrates the relative positional relationship between humanoid robot 501 and power distribution cabinet 502. The humanoid robot stands in front of the power distribution cabinet. A depth camera on its head is used to detect the ArUco logo from a distance and to calculate the panel pose. Close-range depth cameras on its left and right forearms are used to observe the details of the buttons and knobs up close. The ArUco logo is placed in the lower right corner of the panel, serving as a unified spatial reference between the robot and the panel. Figure 5 This embodiment illustrates the spatial layout between the robot body, sensing unit, end effector unit, and the operated power cabinet, as well as the engineering configuration of "global head observation, local arm observation, and end-effector contact operation".
[0047] Figure 6 This diagram illustrates a flowchart of a power line inspection method for a humanoid robot based on a multimodal large model and visual pose compensation, according to an embodiment of the present invention. Figure 6 As shown, the method includes the following steps: Step 601: Receive natural language voice or text commands and generate command text. Upon receiving a natural language voice command, the speech recognition model converts the speech into text. Alternatively, in standardized operation scenarios, the system directly receives the operation manual or operation ticket text. The final command text is then generated. The command text may include information such as device number, operation object, target status, action type, and constraints.
[0048] Step 602: Receive the instruction text and output a standard subtask sequence. The task decomposition model receives the instruction text and, combining knowledge from the power operation domain, decomposes the instruction text into a standard subtask sequence. Each subtask adopts a three-stage structure of navigation, localization, and operation, and is labeled with the target object, target state, and constraints. The task decomposition results are output in JSON format.
[0049] Step 603: Control the robot to move to the preset waypoint in front of the power control cabinet, detect the ArUco marker, use the PnP algorithm to calculate the robot's pose relative to the panel, and perform posture fine-tuning. The navigation node, based on the SLAM results from the head-mounted 3D LiDAR, controls the humanoid robot to move to the preset waypoint in front of the power control cabinet. Upon arrival, the head-mounted depth camera captures images of the panel, detects the ArUco marker, and uses the PnP algorithm to calculate the robot's precise pose relative to the panel. The system performs posture fine-tuning in the forward / backward, left / right, and yaw directions based on the calculated pose, ensuring the dual-arm working space faces the target operating area. Posture fine-tuning can be iteratively executed until the deviation meets the requirements. In one embodiment of the invention, the PnP algorithm uses known ArUco marker geometry, camera intrinsics, and head sensor extrinsics for pose calculation, and the posture fine-tuning ends after the posture deviation enters a set range.
[0050] Step 604: The near-field depth camera images the panel within the working distance. A visual language model detects targets and reads text labels. Combining the depth map, camera intrinsics, and hand-eye calibration, the 3D coordinates of the target in the robot's base coordinate system are obtained. Principal component analysis is used to estimate the current angle for knob targets. After posture adjustment, the first and second near-field depth cameras image the panel at close range. The visual language model performs target detection and classification (buttons / knobs and other panel components). OCR reads the silkscreened text labels below the target, achieving semantic-level recognition. The 2D detection box is then registered with the depth map, and the 3D coordinates of the target in the robot's base coordinate system are obtained through camera intrinsics and hand-eye calibration. For knob-type targets, the PCA angle estimation algorithm is used to calculate the current angle.
[0051] Step 605: Organize the target label, category, text identifier, 3D coordinates, and knob angle into a JSON structure and pass it to the large coding model. This structure enables seamless integration between the perception module and the planning module.
[0052] Step 606: Generate executable code based on the JSON structure and the robot API library. The motion planning component plans the trajectory and sends it to the robot joints for execution. The large-scale coding model, combined with the robot API library (navigation API, arm kinematics API, gripper / dexterous hand control API, etc.), generates executable Python code according to sub-task semantics. For a two-finger gripper, generate gripping, pressing, and pulling motion code; for a multi-finger dexterous hand, generate twisting and fine pinching motion code. After the motion planning component executes the trajectory planning, it sends joint commands to the robot joints for execution.
[0053] In one embodiment of the present invention, the humanoid robot power inspection operation method based on multimodal large model and visual pose compensation further includes, after executing step 606: reading back the joint states and camera images, verifying the target state, and broadcasting the task status through a speech synthesis module. After execution, the joint states and camera images are read back to verify the results. For buttons, it can be verified whether the pressing action is executed and whether the button state changes; for knobs, it can be verified whether the knob angle reaches the target angle. The speech synthesis module broadcasts the task start, execution progress, execution confirmation, or abnormal alarm through a speaker. If the verification fails, the system reverts to the corresponding stage to replan or retry.
[0054] The following example illustrates the optimal operating state of the humanoid robot power inspection operating system and method based on multimodal large model and visual pose compensation provided by this invention. The optimal operating state refers to the working condition in which all components of the system cooperate well, the sensor field of view is complete, the target is located at the center of the optimal working space of the two arms, and the interference of the on-site environment is minimal.
[0055] Optimal Usage State 1: Single-step operation, panel button pressing, easy-level task. Applicable scenarios include pressing single buttons, reset buttons, and open / close buttons on substation protection and control panel cabinets, or operating single buttons on switchgear panels in distribution rooms. In this state, the robot stands facing the panel, with the working surface centered in the optimal working space for both arms, a working distance of 0.3–0.6m, the ArUco logo "165" in the lower right corner of the panel fully visible, and uniform ambient lighting with no strong reflections or backlighting. The complete operation sequence in this state is: command input, task decomposition, autonomous navigation, posture fine-tuning, precise perception, code generation, execution, and verification. The success rate of single-step operation tasks in this state can reach 93%.
[0056] Optimal Usage State Two: Knob-type state switching operations, an intermediate-level task, including predicting whether an action is needed. Applicable scenarios include state switching of knob-type components such as "local / remote" selector switches and "open / close" selector switches on control cabinets. The key feature of this state is that the system first performs PCA principal component analysis on the knob area point set using the arm camera to estimate the current knob angle. If it is determined that the knob is already at the target angle, the rotation action is skipped and the current state is announced verbally; the turning code is only generated and executed when an action is required. This state avoids unnecessary disturbances to the equipment. Experimental data shows that 40% of knob tests showed the knob was already at the target angle, saving an average of 6.5 seconds.
[0057] Optimal Usage State 3: Multi-step compound operations, performing multiple target operations in a single navigation, representing a challenging task. This is suitable for inspection tasks requiring sequential execution of compound operations such as "pressing three different buttons" or "two button presses + two knob rotations." The robot only needs to perform one navigation and one posture fine-tuning, then continuously performs multiple target recognition and operations while maintaining its position. The dual-model planning architecture and structured JSON interface show significant advantages in this state, achieving a success rate of up to 70% for compound operation 2, with a performance degradation of approximately 25%.
[0058] Optimal Usage Mode 4: Pure Text Command Mode. Suitable for standardized switching operations or routine inspections, where operating procedures are based on written protocols. The system does not use voice input; instead, the operation manual or operation ticket text is directly entered into the system, and the task decomposition model executes the procedures line by line. The system provides full voice feedback on the execution progress and reads back the equipment status, suitable for operational scenarios requiring strict adherence to written procedures and high traceability.
[0059] Under the above-mentioned optimal usage conditions, the test results for four typical tasks (each task was repeated 30 times independently, with the initial position and state of the equipment set randomly) are shown in Table 1.
[0060] Table 1 Typical tasks and their number of objectives, number of steps, difficulty level, and measured success rate Figure 7 A schematic diagram comparing the success rates of different methods on different tasks is shown. Table 2 shows the comparison results of the method provided by this invention with two baseline methods on different evaluation metrics.
[0061] Table 2 Comparison of key indicators for different methods From Table 1, Table 2 and Figure 7It can be seen that the method provided by this invention has significant advantages in single-step operation tasks. In button pressing tasks, the success rate of baseline method 1 is 30%, while the success rate of the method provided by this invention is 93%. In complex operation tasks, the average success rate of baseline method 1 is 27%, the average success rate of baseline method 2 is 53%, and the average success rate of the method provided by this invention is 75%. Regarding the performance degradation, as the task sequence length increases, the degradation of baseline method 1 is 50%, while the degradation of the method provided by this invention is 25%. Due to its dual-model architecture and structured interface, the method provided by this invention has greater engineering reliability in long-sequence complex tasks. Regarding positioning deviation, without ArUco compensation, the deviation is ±20cm and ±35°, while with ArUco pose compensation, the deviation is ±4.2cm in distance and ±5.6° in attitude angle. Regarding 3D positioning accuracy, the D405 arm achieves ±3.8mm in the X direction, ±2.1mm in the Y direction, and ±3.5mm in the Z direction within a working distance of 0.3 to 0.6m. The overall average success rate of the method provided by this invention for the four types of tasks is 82.5%.
[0062] In the failure cases of baseline method 2, about 70% can be attributed to "inability to compensate for navigation residuals and insufficient end-effector accuracy" and failure at the edge of the robot arm's workspace. This phenomenon directly verifies the necessity of the ArUco assisted posture fine-tuning step and also shows that the optimal use state of the present invention is "the robot and the panel face each other, the ArUco logo is fully visible, and the work point is located in the middle of the dual-arm workspace".
[0063] The following further explains the differences between the robot system provided by this invention and existing solutions. Existing wheeled inspection robots primarily address mobile inspection problems. This invention integrates dual-arm end-effector execution and close-range fine perception into a humanoid robot body, enabling the robot to perform contact-based operations. Existing fixed-camera solutions primarily address remote visualization problems. This invention, through autonomous robot movement and dual-arm operation, extends operational capabilities from "seeing" to "doing." Existing single large-model solutions primarily address end-to-end generation problems. This invention reduces planning errors in long-sequence tasks through dual-model decoupling and a structured JSON interface. Existing two-dimensional visual positioning solutions primarily address target detection problems. This invention achieves three-dimensional positioning in the robot's base coordinate system through ArUco labeling, PnP pose compensation, a near-field depth camera on the arm, and hand-eye calibration. Existing shape recognition solutions primarily address target detection problems. This invention solves the problem of indistinguishable targets with similar appearances through OCR text labels and semantic recognition. Existing motion execution solutions typically do not consider the current state of the device. This invention uses PCA angle estimation to determine whether a knob is in the target state, achieving context-sensitive decision-making.
[0064] This invention also provides a computer-readable storage medium storing computer-readable instructions. When executed by a processor, these computer-readable instructions perform the following processing steps: receiving natural language speech or text instructions and forming instruction text; receiving the instruction text and outputting a standard subtask sequence; controlling the robot to move to a preset waypoint in front of the power panel cabinet, detecting the ArUco marker, using the PnP algorithm to calculate the robot's pose relative to the panel, and performing fine-tuning of the standing posture; imaging the panel within the working distance using a near-field depth camera, detecting the target and reading the text label through a visual language model, obtaining the target's three-dimensional coordinates in the robot's base coordinate system by combining the depth map, camera intrinsics, and hand-eye calibration, and estimating the current angle of the knob target using principal component analysis; organizing the target label, category, text identifier, three-dimensional coordinates, and knob angle into a JSON structure; generating executable code based on the JSON structure and the robot API library, planning the trajectory by the motion planning component, and sending it to the robot joints for execution.
[0065] The computer-readable storage medium includes various media capable of storing machine-readable instructions, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0066] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined according to the technical solutions of the invention and their equivalents.
Claims
1. A power inspection operating system for humanoid robots based on multimodal large models and visual pose compensation, characterized in that, include: The humanoid robot body carrier includes a head and neck sensor mounting bracket and a modular flange interface at the end of the forearm. The head pose perception unit includes a lidar and a depth camera mounted on the head and neck sensor mounting bracket. The lidar is configured to build an environment map and output odometry information, and the depth camera is configured to detect ArUco markers and calculate the robot's pose relative to the panel. The dual-arm local fine perception unit includes a first near-field depth camera and a second near-field depth camera, which are configured to output the three-dimensional coordinates of the target in the robot base coordinate system within the working distance. A heterogeneous dual-arm end effector unit, mounted at the modular flange interface of the forearm end, includes a two-finger gripper and a multi-finger dexterous hand. The two-finger gripper is configured to perform pressing actions, and the multi-finger dexterous hand is configured to perform twisting actions. The reasoning and voice interaction unit is equipped with a language interaction module, a visual language model, a large task decomposition model, a large coding model, and a motion planning component.
2. The humanoid robot power inspection operating system based on multimodal large model and visual pose compensation according to claim 1, characterized in that, The humanoid robot body carrier also includes leg and waist joint groups, six-degree-of-freedom joint groups for both arms, head pitch joints, and onboard computing unit mounting positions.
3. The humanoid robot power inspection operating system based on multimodal large model and visual pose compensation according to claim 1, characterized in that: The lidar is a three-dimensional lidar; and / or The working distance between the first close-range depth camera and the second close-range depth camera is 0.3 to 0.6 m.
4. The humanoid robot power inspection operating system based on multimodal large model and visual pose compensation according to claim 1, characterized in that, The output of the task decomposition model is JSON format data, and the input of the encoding model is JSON format data.
5. The humanoid robot power inspection operating system based on multimodal large model and visual pose compensation according to claim 1, characterized in that, Also includes: The work object and auxiliary positioning identification unit includes power panel cabinets and ArUco signs.
6. A method for power line inspection operation of a humanoid robot based on a multimodal large model and visual pose compensation, characterized in that, The power inspection operating system for humanoid robots based on multimodal large models and visual pose compensation, as described in any one of claims 1 to 5, includes the following steps: It receives natural language voice or text commands and generates command text. Receive the instruction text and output a standard subtask sequence; The robot is controlled to move to the preset waypoint in front of the power panel cabinet, detect the ArUco marker, use the PnP algorithm to calculate the robot's pose relative to the panel, and perform fine-tuning of the standing posture. The panel is imaged by a near-field depth camera within the working distance. The target is detected and the text label is read by a visual language model. The three-dimensional coordinates of the target in the robot base coordinate system are obtained by combining the depth map, camera intrinsic parameters and hand-eye calibration. Principal component analysis is used to estimate the current angle of the knob target. Organize the target label, category, text identifier, 3D coordinates, and knob angle into a JSON structure; as well as Executable code is generated based on the JSON structure and the robot API library, and the motion planning component plans the trajectory and sends it to the robot joints for execution.
7. The method for power line inspection operation of a humanoid robot based on multimodal large model and visual pose compensation according to claim 6, characterized in that, Upon receiving the instruction text, the output standard subtask sequence includes: The task decomposition model receives the instruction text; A large-scale task decomposition model, combined with knowledge from the power operation domain, generates a standard sub-task sequence, which includes navigation sub-tasks, positioning sub-tasks, and operation sub-tasks; and The navigation subtask, the positioning subtask, and the operation subtask are labeled with target objects, target states, and constraints.
8. The method for power line inspection operation of a humanoid robot based on multimodal large model and visual pose compensation according to claim 6, characterized in that, The PnP algorithm uses known ArUco identifier geometry, camera intrinsics, and head sensor extrinsics to calculate pose. The pose fine-tuning ends when the pose deviation enters a set range.
9. The method for power line inspection operation of a humanoid robot based on multimodal large model and visual pose compensation according to claim 6, characterized in that, Also includes: The system reads back the joint status and camera images, verifies the target status, and broadcasts the task status through the speech synthesis module.
10. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by the processor, they perform the steps of the power inspection operation method for a humanoid robot based on multimodal large model and visual pose compensation as described in any one of claims 6 to 9.
