A somatic robot patrol method and system based on human-computer interface visual detection

Abstract

The application discloses a kind of based on human-computer interface visual detection somatotopic robot patrol method and system, belong to industrial vision robot technical field, to solve the problem that existing inspection means is difficult to realize entity robot to industrial equipment human-computer interface automatic touch control operation.The method of the application comprises: robot moves along preset route, identifies alarm indicator light, first handles exception if there is exception;Orthovision camera collects image, matches equipment code, locates screen and adjusts pose, corrects distortion, suppresses reflection and obtains screen image;Input visual language action model, output analysis result and mechanical arm instruction;If for in-process state, then execute queuing inspection, if for completed state, then execute touch review;Continue to inspect next equipment.The application constructs "visual perception-semantic analysis-physical touch-state verification" complete patrol operation loop, improves the inspection efficiency of industrial equipment human-computer interface, especially suitable for harsh industrial plant scene.

Description

Technical Field

[0001] This invention belongs to the field of industrial vision robot technology, specifically relating to a method and system for embodied robot patrol based on human-machine interface visual detection. Background Technology

[0002] With the rapid development of intelligent manufacturing and industrial automation technologies, the stable operation of a large number of automated equipment in industrial production scenarios has become a core guarantee for production efficiency and product quality. Industrial production equipment is generally equipped with status alarm indicator lights and human-machine interface (HMI) touchscreens. During daily operation and maintenance, inspection personnel need to check the status of equipment alarm indicator lights, read screen operating parameters, and perform corresponding screen touch operations according to the work progress to ensure the orderly progress of the production process.

[0003] Traditional industrial equipment inspections primarily rely on manual labor. Inspectors check the status of each piece of equipment along a predetermined route and perform operations. However, manual inspections suffer from low efficiency. Furthermore, some industrial production plants operate under harsh conditions such as high temperatures, high noise levels, strong radiation, and toxic substances. Prolonged on-site inspections pose a threat to personnel health and are difficult to conduct continuously, failing to meet the maintenance needs of highly automated production lines.

[0004] With the iteration of robotics and computer vision technologies, industrial inspection robots are gradually being applied to equipment maintenance scenarios. Most existing inspection robots are equipped with vision acquisition modules, using image recognition technology to detect the status of equipment alarm indicator lights, read instrument values, and identify equipment appearance abnormalities. However, current inspection robots are largely still in the "observation-based inspection" stage. They cannot replace manual operation of physical touchscreens on industrial HMIs, nor can they autonomously execute equipment maintenance processes. Inspection and operation are disconnected, still requiring subsequent manual intervention, making it difficult to form a complete unmanned maintenance closed loop.

[0005] In the field of UI automation, software robotic process automation (RPA) technology can automate operations such as clicking and input on the software system interface. However, this type of technology is a software-level simulation interaction, and its operation depends on the open software interface of the target device. However, there are many old devices in industrial sites, most of which do not have standardized open interfaces, making software RPA technology unsuitable.

[0006] In summary, existing technologies are insufficient to enable automated touch operation of human-machine interfaces on industrial equipment by physical robots, and cannot construct a complete unmanned inspection process from perception and analysis to operation and verification. This makes it difficult to meet the development needs of unmanned operation and maintenance of industrial equipment under harsh working conditions. Summary of the Invention

[0007] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0008] On the one hand, the present invention provides a method for embodied robot patrol based on human-machine interface visual detection, including the following steps:

[0009] S1. The embodied mobile robot moves along the preset inspection route, and identifies the status of alarm indicator lights within its field of vision in real time during the movement; if an abnormal status of an alarm indicator light is detected, an abnormal handling procedure is executed; if all alarm indicator lights within the field of vision are in normal status, the regular screen inspection procedure is executed on the current device.

[0010] S2. Acquire the front image of the current device through the front-viewing camera, identify the identification code on the device to complete the device number matching, locate the human-machine interface screen area of ​​the device, adjust the pose of the robot body and the front-viewing camera, and obtain the screen interface image after completing perspective distortion correction and screen reflection suppression.

[0011] S3. Input the screen interface image of the current device into the visual language action model, and the model outputs the structured parsing results of the screen interface content and the touch robotic arm action instructions;

[0012] S4. If the structured parsing result is in progress, then execute the queuing inspection process to generate a priority queue to be inspected. If the structured parsing result of the screen content is in completed state, then run the touch robotic arm action command to execute the touch review process.

[0013] S5. After completing the inspection of the current equipment, proceed to inspect the next piece of equipment in sequence according to the established inspection route and the priority queue of equipment to be inspected.

[0014] The exception handling process described in step S1 specifically includes:

[0015] S101. Control the robot to move and position it to the alarm indicator light location for abnormal conditions;

[0016] S102. Detect the identification code on the device at the location of the alarm indicator light and identify the device number;

[0017] S103, Position the emergency stop button area on the positioning device and control the touch robotic arm to trigger the emergency stop operation;

[0018] S104. Report equipment malfunction information and handling records.

[0019] Further, the mobile positioning method described in step S101 is as follows: acquire a monocular RGB image, input it into a monocular depth estimation network to obtain a relative depth map; combine the camera intrinsic calibration parameters to convert the two-dimensional pixel coordinates of the target into three-dimensional spatial coordinates in the robot base coordinate system, calculate the distance and azimuth angle, and generate navigation control commands to drive the robot to move to the target position.

[0020] Specifically, the perspective distortion correction and screen reflection suppression in step S2 are as follows: Correction markers are set in advance around the human-machine interface touch screen. The pixel coordinates of each correction marker are visually identified, the perspective transformation matrix is ​​calculated, and the screen image is distorted. When the camera is close to the screen to capture an image, the retractable light shield at the front of the lens is attached to the periphery of the screen to isolate ambient light and suppress screen reflection.

[0021] In step S3, the input to the visual language action model is a screen interface image, and also includes preset inspection task text instructions. The model internally includes a visual encoder, a text encoder, a general large language model, an action decoder, and a parsing output head. The visual encoder is used to extract the visual spatial features of the screen image. The text encoder is used to encode the text semantic features of the preset inspection task text instructions. The general large language model is used to fuse visual spatial features and text semantic features to generate hidden state features. The action decoder is used to convert the hidden state features into discrete touch-sensitive robotic arm action instructions. The parsing output head is used to convert the hidden state features into structured parsing results of the screen content.

[0022] Furthermore, the control parameters of the touch-sensitive robotic arm motion command include: the target position of the end effector of the touch-sensitive robotic arm on the X, Y, and Z axes in the robot base coordinate system, the pressure value, and the touch dwell time.

[0023] Furthermore, the robot base coordinate system is specifically defined as follows: the camera coordinate system and the robot base coordinate system are pre-calibrated using a calibration board to establish a mapping relationship between screen pixel coordinates and three-dimensional target points in the robot base coordinate system; based on the screen touch point pixel coordinates obtained by visual recognition, they are converted into three-dimensional motion target points at the end of the robotic arm through the mapping relationship.

[0024] The queuing inspection process described in step S4 is as follows:

[0025] S401. Extract the current device operation stage and estimated remaining time from the structured parsing results of the screen content;

[0026] S402. Write the equipment number, operation stage and estimated remaining time into the task list to be inspected.

[0027] S403. Sort the tasks according to the order of the work stages and the estimated remaining time to generate a priority queue for inspection.

[0028] The touch verification process in step S4 is as follows:

[0029] S411. Perform screen touch operation according to the touch robotic arm's motion command;

[0030] S412. Return the touch-screen robotic arm to its original position and capture the screen interface image after the operation again through the front-facing camera.

[0031] S413. Input the screen image after the operation back into the visual language action model to obtain the structured parsing result of the screen interface content after the operation.

[0032] S414. If the analysis result determination operation is effective, then the inspection is confirmed to be complete; if the analysis result determination operation is not effective, then a touch abnormality alarm is reported.

[0033] On the other hand, the present invention provides a human-machine interface visual detection-based embodied robot patrol system, including a mobile embodied robot body, a multi-view visual acquisition module, a touch-controlled robotic arm execution mechanism, a logic processing module, and a visual language action model.

[0034] The multi-view vision acquisition module is mounted on the mobile embodied robot body and includes a front camera, a rear camera and a front-view camera. It is used to acquire images of equipment alarm indicator lights on the inspection path and front images of the human-machine interface touch screen.

[0035] The touch-sensitive robotic arm actuator is used to perform human-machine interface touch operations and emergency stop button trigger operations according to action commands.

[0036] The visual language action model is used to parse screen interface images and output the structured parsing results of the interface content and the touch action instructions of the robotic arm.

[0037] The logic processing module is used to perform inspection route planning, equipment number matching, anomaly priority scheduling, inspection task database management, and communication and interaction with the host system.

[0038] The beneficial effects of this invention are:

[0039] This invention employs a unibody mobile robot to replace manual labor in performing the entire inspection and operation process. It can operate continuously and stably in harsh industrial production environments such as high temperature, high noise, and strong radiation, reducing the safety risks for on-site personnel and effectively improving inspection efficiency. This invention overcomes the technical limitations of existing inspection robots that "only observe and do not move," autonomously completing equipment maintenance operations and realizing a shift from "passive perception and alarm" to "proactive execution and handling," achieving an unmanned inspection system. This invention uses visual perception and physical interaction with a robotic arm to operate industrial HMI screens, requiring no open software interfaces on the target equipment. It is widely adaptable to various older industrial equipment, requiring no hardware or software modifications to existing equipment, and exhibits strong equipment compatibility. In summary, this invention achieves a complete closed-loop inspection process of "visual perception - semantic parsing - physical touch control - status verification," improving the inspection efficiency of industrial equipment HMIs, and is compatible with industrial production equipment without open software interfaces. It has particularly strong application value in harsh industrial production environments. Attached Figure Description

[0040] Figure 1 This is a flowchart of the steps of the embodied robot patrol method based on human-machine interface visual detection in Embodiment 1 of the present invention;

[0041] Figure 2 This is a flowchart of the abnormal handling process in step S1 of Embodiment 1 of the present invention;

[0042] Figure 3 This is a schematic diagram of the internal structure of the visual language action model in Embodiment 1 of the present invention;

[0043] Figure 4 This is a flowchart of step S4, queuing inspection, in Embodiment 1 of the present invention.

[0044] Figure 5 This is a flowchart of step S4, touch verification, in Embodiment 1 of the present invention;

[0045] Figure 6 This is a block diagram of the embodied robot patrol system based on human-machine interface visual detection in Embodiment 2 of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0048] Example 1

[0049] This embodiment provides a method for embodied robot patrol based on human-machine interface visual detection, such as... Figure 1 As shown, it includes the following steps:

[0050] S1. The embodied mobile robot moves along the preset inspection route, and identifies the status of alarm indicator lights within its field of vision in real time during the movement; if an abnormal status of an alarm indicator light is detected, an abnormal handling procedure is executed; if all alarm indicator lights within the field of vision are in normal status, the regular screen inspection procedure is executed on the current device.

[0051] S2. Acquire the front image of the current device through the front-viewing camera, identify the identification code on the device to complete the device number matching, locate the human-machine interface screen area of ​​the device, adjust the pose of the robot body and the front-viewing camera, and obtain the screen interface image after completing perspective distortion correction and screen reflection suppression.

[0052] S3. Input the screen interface image of the current device into the visual language action model, and the model outputs the structured parsing results of the screen interface content and the touch robotic arm action instructions;

[0053] S4. If the structured parsing result is in progress, then execute the queuing inspection process to generate a priority queue to be inspected. If the structured parsing result of the screen content is in completed state, then run the touch robotic arm action command to execute the touch review process.

[0054] S5. After completing the inspection of the current equipment, proceed to inspect the next piece of equipment in sequence according to the established inspection route and the priority queue of equipment to be inspected.

[0055] In this embodiment, as Figure 2 As shown, the exception handling process in step S1 specifically includes:

[0056] S101. Control the robot to move and position it to the alarm indicator light location for abnormal conditions;

[0057] S102. Detect the identification code on the device at the location of the alarm indicator light and identify the device number;

[0058] S103, Position the emergency stop button area on the positioning device and control the touch robotic arm to trigger the emergency stop operation;

[0059] S104. Report equipment malfunction information and handling records.

[0060] It should be noted that the anomaly handling procedure has a higher priority than the regular screen patrol procedure. The current patrol task should be immediately interrupted when an alarm light anomaly is detected. The regular screen patrol procedure consists of steps S2, S3, S4, and S5.

[0061] In this embodiment, the mobile positioning method in step S101 is as follows: acquire a monocular RGB image, input it into a monocular depth estimation network to obtain a relative depth map; combine the camera intrinsic calibration parameters to convert the two-dimensional pixel coordinates of the target into three-dimensional spatial coordinates in the robot base coordinate system, calculate the distance and azimuth angle, and generate navigation control commands to drive the robot to move to the target position.

[0062] Specifically, the monocular depth estimation network uses R-MSFM, PyDNet, or FastDepth, taking an RGB image as input and outputting a relative depth map at the same resolution. Camera intrinsic parameters are pre-calibrated using Zhang's calibration method, including focal length, principal point coordinates, and distortion coefficients. During coordinate transformation, the coordinates of the warning light pixel center point are first output, combined with the depth value and intrinsic parameters to calculate the 3D coordinates, and then converted into a global navigation target point using the chassis odometer.

[0063] In this embodiment, the method for locating the emergency stop button area on the device in step S103 is as follows: The two-dimensional pixel area of ​​the emergency stop button is initially defined in the RGB image using target detection; the depth value of this area is extracted by combining the relative depth map output by the monocular depth estimation network. Using pre-calibrated camera intrinsic calibration parameters, the two-dimensional pixel coordinates are transformed to the robot's base coordinate system, and the three-dimensional spatial coordinates and orientation of the emergency stop button are calculated, thereby accurately locating the emergency stop button area.

[0064] In this embodiment, the method for locating the human-machine interface screen area of ​​the device in step S2 is as follows: A two-dimensional pixel area of ​​the human-machine interface screen is initially defined in the RGB image using target detection, and the coordinates of the four corners of the screen are extracted. The depth value of this area is extracted by combining the relative depth map output by the monocular depth estimation network. Using pre-calibrated camera intrinsic calibration parameters, the two-dimensional pixel coordinates are transformed to the robot's base coordinate system, and the three-dimensional spatial coordinates and pose of the screen plane are calculated, thereby accurately locating the screen area.

[0065] In this embodiment, the perspective distortion correction and screen reflection suppression in step S2 are specifically as follows: correction markers are set in advance around the human-machine interface touch screen, the pixel coordinates of each correction marker are visually identified, the perspective transformation matrix is ​​calculated, and the screen image is distorted; when the camera is close to the screen to capture the image, the retractable light shield at the front of the lens is attached to the periphery of the screen to isolate ambient light and suppress screen reflection.

[0066] Specifically, four circular black markers, each 15mm in diameter, are affixed to the outer edges of the four corners of the touchscreen. A corner detection algorithm identifies the center coordinates of each marker's pixel, and a four-point method is used to calculate the perspective transformation matrix to correct the screen area's perspective. The retractable light shield is made of black silicone light-blocking fabric and is driven by a miniature electric actuator, stopping when it conforms to the screen frame.

[0067] In this embodiment, the input to the visual language action model in step S3 is a screen interface image, and also includes preset inspection task text instructions; such as Figure 3 As shown, the model internally includes a visual encoder, a text encoder, a general large language model, an action decoder, and a parsing output head. The visual encoder is used to extract the visual spatial features of the screen image; the text encoder is used to encode the text semantic features of the preset inspection task text instructions; the general large language model is used to fuse visual spatial features and text semantic features to generate hidden state features; the action decoder is used to convert the hidden state features into discretized touch-controlled robotic arm action instructions; and the parsing output head is used to convert the hidden state features into structured parsing results of the screen content.

[0068] Specifically, the visual encoder employs a SigLIP or CLIP ViT structure. The input is a screen interface image, which, after block embedding and multi-layer self-attention computation, extracts the visual spatial features of elements such as interface buttons and text, outputting a visual feature vector. The text encoder uses the built-in text encoder parameters of the large language model to perform word embedding and semantic encoding on preset inspection task text instructions such as "recognize the content of the current screen image," outputting a text semantic feature vector. The general large language model uses an LLaMA or Qwen large language model structure, generating hidden state features containing interface semantic understanding and operational intent through attention-based cross-modal fusion. The action decoder uses a discrete action token mapping network to convert the fused hidden state features into discretized touch-sensitive robotic arm action instructions, totaling 96 action tokens. Each token corresponds to the three-dimensional position offset of the robotic arm's end effector. After coordinate transformation, the absolute target position in the robot's base coordinate system is generated, corresponding to the pressure level and touch dwell time level. The parsing output header uses a fully connected MLP to map the hidden state features into structured parsing results, outputting key-value pair format data, including device operating status, job stage, and estimated remaining time, for example {"status": "running", "stage": "heating", "remaining_time": "120s"}.

[0069] The visual language action model uses industrial HMI interface data. The training set covers 2,800 samples of various types of industrial touch screen interfaces and corresponding operation annotations. The visual language action model is fine-tuned with an initial learning rate of 1e-4, AdamW optimizer, batch size of 8, and 10 training rounds.

[0070] In this embodiment, the control parameters of the touch-sensitive robotic arm motion command include: the target position of the end effector of the touch-sensitive robotic arm on the X, Y, and Z axes in the robot base coordinate system, the pressure value, and the touch dwell time.

[0071] In this embodiment, the robot base coordinate system is specifically defined as follows: A calibration board is used beforehand to perform hand-eye calibration between the camera coordinate system and the robot base coordinate system, establishing a mapping relationship between screen pixel coordinates and three-dimensional target points in the robot base coordinate system. Based on the screen touch point pixel coordinates obtained through visual recognition, these coordinates are converted into three-dimensional motion target points at the end effector of the robotic arm through the mapping relationship. Specifically, a 9×12 grid calibration board is used, with each square having a side length of 20mm. During calibration, the calibration board is fixed in different poses within the robotic arm's workspace, and more than 15 sets of calibration board images and corresponding robotic arm end effector coordinates are acquired. The coordinate system transformation matrix is ​​then solved using a hand-eye calibration algorithm.

[0072] In this embodiment, the queuing inspection process described in step S4 is as follows: Figure 4 As shown, specifically:

[0073] S401. Extract the current device operation stage and estimated remaining time from the structured parsing results of the screen content;

[0074] S402. Write the equipment number, operation stage and estimated remaining time into the task list to be inspected.

[0075] S403. Sort the tasks according to the order of the work stages and the estimated remaining time to generate a priority queue for inspection.

[0076] Specifically, the task list to be inspected includes fields for equipment number, work stage, estimated remaining time, and current priority. The sorting method is as follows: sorted from high to low according to the preset work stage priority; within the same stage, sorted from shortest to longest estimated remaining time; tasks with less than 5 minutes remaining time are given priority for scheduling.

[0077] In this embodiment, the touch verification process described in step S4 is as follows: Figure 5 As shown, specifically:

[0078] S411. Perform screen touch operation according to the touch robotic arm's motion command;

[0079] S412. Return the touch-screen robotic arm to its original position and capture the screen interface image after the operation again through the front-facing camera.

[0080] S413. Input the screen image after the operation back into the visual language action model to obtain the structured parsing result of the screen interface content after the operation.

[0081] S414. If the analysis result determination operation is effective, then the inspection is confirmed to be complete; if the analysis result determination operation is not effective, then a touch abnormality alarm is reported.

[0082] Specifically, after a touch operation is executed, the robotic arm retracts to a safe position to avoid obstructing the screen. If the operation is determined to be ineffective, a retry mechanism of up to 2 times is automatically triggered. During the retry, the touch point coordinates are finely adjusted by ±1mm and the pressing pressure is increased by 1 level. If the retry still fails, a touch abnormality alarm is reported.

[0083] In this embodiment, step S5: After completing the inspection operation of the current device, the next device is inspected sequentially according to the predetermined inspection route and the priority queue to be inspected. Specifically, the inspection order adopts the strategy of "predetermined route as the main method and priority queue insertion as the secondary method". Under normal circumstances, the devices are inspected sequentially according to the preset route; when a high-priority task appears in the priority queue, it is inserted for execution after the inspection of the current device is completed.

[0084] Example 2

[0085] This embodiment provides an embodied robot patrol system based on human-machine interface visual detection, such as... Figure 6 As shown, it includes a mobile embodied robot body, a multi-view vision acquisition module, a touch-controlled robotic arm actuator, a logic processing module, and a visual language action model.

[0086] In this embodiment, the multi-view vision acquisition module is mounted on the mobile android body and includes a front camera, a rear camera and a front-view camera, used to acquire images of equipment alarm indicator lights on the inspection path and front images of the human-machine interface touch screen.

[0087] Specifically, the front and rear cameras are mounted on the upper front and rear sides of the robot body, respectively. These are wide dynamic range industrial cameras with a resolution of 1920×1080 and a field of view of 120°, used for wide-area identification of equipment alarm indicators. The front-view camera is mounted on a gimbal next to the touch-sensitive robotic arm. It is a fixed-focus camera with a resolution of 1920×1080 and a field of view of 60°. The gimbal supports horizontal ±180° and pitch ±90° rotation.

[0088] In this embodiment, the touch-sensitive robotic arm actuator is used to perform human-machine interface touch operations and emergency stop button triggering operations according to action commands. Specifically, the touch-sensitive robotic arm is a six-degree-of-freedom collaborative robotic arm with an arm span of 600mm and a repeatability of ±0.5mm. The end effector uses a cylindrical silicone touch head with a diameter of 8mm and a Shore hardness of 40.

[0089] In this embodiment, the visual language action model is used to parse screen interface images and output structured parsing results of the interface content and touch action commands for the robotic arm. Specifically, the visual language action model is deployed on a server and invokes the model inference service via remote access.

[0090] In this embodiment, the logic processing module is used to perform inspection route planning, equipment number matching, anomaly priority scheduling, inspection task database management, and communication interaction with the upper-level system. Specifically, the logic processing module has a built-in SQLite database to store inspection tasks, equipment information, and historical records. The logic processing module supports multiple industrial communication protocols such as Modbus, TCP / IP, and MQTT, and can interface with factory MES and SCADA systems to realize the uploading of inspection data and the distribution of tasks.

[0091] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0092] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within this application.

Claims

1. A method for embodied robot patrol based on human-machine interface visual detection, characterized in that, Includes the following steps: S1. The embodied mobile robot moves along the preset inspection route, and identifies the status of alarm indicator lights within its field of vision in real time during the movement; if an abnormal status of the alarm indicator light is detected, the abnormal handling procedure is executed. If all alarm indicator lights within the field of view are in normal condition, perform the standard screen inspection procedure on the current device. S2. Acquire the front image of the current device through the front-viewing camera, identify the identification code on the device to complete the device number matching, locate the human-machine interface screen area of ​​the device, adjust the pose of the robot body and the front-viewing camera, and obtain the screen interface image after completing perspective distortion correction and screen reflection suppression. S3. Input the screen interface image of the current device into the visual language action model, and the model outputs the structured parsing results of the screen interface content and the touch robotic arm action instructions; S4. If the structured parsing result is in progress, then execute the queuing inspection process to generate a priority queue to be inspected. If the structured parsing result of the screen content is in completed state, then run the touch robotic arm action command to execute the touch review process. S5. After completing the inspection of the current equipment, proceed to inspect the next piece of equipment in sequence according to the established inspection route and the priority queue of equipment to be inspected.

2. The embodied robot patrol method based on human-machine interface visual detection according to claim 1, characterized in that, The exception handling process described in step S1 specifically includes: S101. Control the robot to move and position it to the alarm indicator light location for abnormal conditions; S102. Detect the identification code on the device at the location of the alarm indicator light and identify the device number; S103, Position the emergency stop button area on the positioning device and control the touch robotic arm to trigger the emergency stop operation; S104. Report equipment malfunction information and handling records.

3. The embodied robot patrol method based on human-machine interface visual detection according to claim 2, characterized in that, The mobile positioning method described in step S101 is as follows: acquire a monocular RGB image, input it into a monocular depth estimation network to obtain a relative depth map; combine the camera intrinsic calibration parameters to convert the two-dimensional pixel coordinates of the target into three-dimensional spatial coordinates in the robot base coordinate system, calculate the distance and azimuth angle, and generate navigation control commands to drive the robot to move to the target position.

4. The embodied robot patrol method based on human-machine interface visual detection according to claim 1, characterized in that, The perspective distortion correction and screen reflection suppression described in step S2 are as follows: correction markers are set in advance around the human-machine interface touch screen, the pixel coordinates of each correction marker are visually identified, the perspective transformation matrix is ​​calculated, and the screen image is distorted; when the camera is close to the screen to capture the image, the retractable light shield at the front of the lens is attached to the periphery of the screen to isolate ambient light and suppress screen reflection.

5. The embodied robot patrol method based on human-machine interface visual detection according to claim 1, characterized in that, The input to the visual language action model in step S3 is a screen interface image, and also includes preset inspection task text instructions. The model internally includes a visual encoder, a text encoder, a general large language model, an action decoder, and a parsing output head. The visual encoder is used to extract the visual spatial features of the screen image. The text encoder is used to encode the text semantic features of the preset inspection task text instructions. The general large language model is used to fuse visual spatial features and text semantic features to generate hidden state features. The action decoder is used to convert the hidden state features into discrete touch-sensitive robotic arm action instructions. The parsing output head is used to convert the hidden state features into structured parsing results of the screen content.

6. The embodied robot patrol method based on human-machine interface visual detection according to claim 5, characterized in that, The control parameters for the touch-sensitive robotic arm's motion commands include: the target positions of the touch-sensitive robotic arm's end effector on the X, Y, and Z axes in the robot's base coordinate system, the pressure value, and the touch dwell time.

7. The embodied robot patrol method based on human-machine interface visual detection according to claim 6, characterized in that, The robot base coordinate system is specifically defined as follows: a calibration board is used in advance to complete the hand-eye calibration of the camera coordinate system and the robot base coordinate system, and a mapping relationship is established between the screen pixel coordinates and the three-dimensional target points in the robot base coordinate system; based on the screen touch point pixel coordinates obtained by visual recognition, they are converted into three-dimensional motion target points at the end of the robotic arm through the mapping relationship.

8. The embodied robot patrol method based on human-machine interface visual detection according to claim 1, characterized in that, The queuing inspection process described in step S4 is as follows: S401. Extract the current device operation stage and estimated remaining time from the structured parsing results of the screen content; S402. Write the equipment number, operation stage and estimated remaining time into the task list to be inspected. S403. Sort the tasks according to the order of the work stages and the estimated remaining time to generate a priority queue for inspection.

9. The embodied robot patrol method based on human-machine interface visual detection according to claim 1, characterized in that, The touch verification process described in step S4 is as follows: S411. Perform screen touch operation according to the touch robotic arm's motion command; S412. Return the touch-screen robotic arm to its original position and capture the screen interface image after the operation again through the front-facing camera. S413. Input the screen image after the operation back into the visual language action model to obtain the structured parsing result of the screen interface content after the operation. S414. If the analysis result determines that the operation is effective, then the inspection is confirmed to be complete. If the analysis result indicates that the operation has not taken effect, a touch abnormality alarm will be reported.

10. A human-machine interface visual inspection-based embodied robot patrol system, characterized in that, The system includes a mobile embodied robot body, a multi-view vision acquisition module, a touch-sensitive robotic arm actuator, a logic processing module, and a visual language motion model. The multi-view vision acquisition module, mounted on the mobile embodied robot body, includes a front-facing camera, a rear-facing camera, and a front-view camera, used to acquire images of equipment alarm indicator lights along the inspection path, as well as frontal images of the human-machine interface touchscreen. The touch-sensitive robotic arm actuator is used to execute touch operations on the human-machine interface and trigger emergency stop buttons according to action commands. The visual language motion model is used to parse the screen interface images, outputting structured parsing results of the interface content and touch action commands for the robotic arm. The logic processing module is used to perform inspection route planning, equipment number matching, anomaly priority scheduling, inspection task database management, and communication and interaction with the host system.

Images