Wheeled inspection robot and camera motion understanding method and system thereof

Abstract

The invention discloses a wheeled inspection robot and a camera motion understanding method and system thereof. Displacement, video stream data and pose data are acquired in real time through a data acquisition module; the camera motion classification module performs motion classification and visual detection on the video frames, outputs geometric primitive and semantic primitive tags and an equipment state detection result, and maps the geometric primitive and semantic primitive tags and the equipment state detection result to a reference frame; the SfM-VLM fusion inference module extracts geometric parameters of camera motion through an SfM algorithm, fuses visual features and then inputs the fused visual features into a VLM model to generate natural language semantic description; the structured annotation module generates double-track annotations of a tag layer and a subtitle layer based on geometric primitives, semantic primitive tags and natural language semantic description; the multi-modal decision module fuses geometric primitive and semantic primitive tags, equipment state detection results and data acquired by the data acquisition module to generate an inspection decision; the problems of motion decoupling, target tracking, data annotation and multi-modal decision making in a dynamic environment are solved, and the routing inspection reliability and the intelligent level are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent inspection robots, and specifically discloses a wheeled inspection robot and a camera motion understanding method and system thereof. Background Art

[0002] In the field of industrial automation and intelligent monitoring, wheeled inspection robots have been widely used for equipment status monitoring and fault diagnosis. However, existing technologies face the following technical bottlenecks in complex environments: 1. The difficulty of decoupling camera motion from scene dynamics: Traditional SLAM algorithms have difficulty distinguishing between the robot's own motion and the motion signals of dynamic objects in the scene. For example, during factory assembly line inspections, the rotation of the conveyor belt captured by the camera as the robot moves forward can easily be misinterpreted as camera translation, resulting in accumulated errors in 3D map construction and reduced object detection accuracy. Experimental data shows that the pose estimation error of traditional SLAM in dynamic scenes can reach over ±5cm. 2. Target tracking failure in complex motion modes: When a robot performs complex movements such as translation and rotation or tracks a moving target, traditional visual tracking algorithms lack the ability to understand motion semantics and struggle to keep the target in the center of their field of view. For example, when an inspection drone tracks a power transmission line, the camera simultaneously rotates horizontally and vertically, resulting in a target loss rate exceeding 40%. 3. Multimodal information fusion and decision-making defects: Robots have difficulty associating camera motion semantics with data from sensors such as infrared and lidar, and are unable to generate structured inspection reports. For example, when a camera detects a hotspot on a device, it cannot determine whether the hotspot is a real fault or a motion artifact based on the motion trajectory, resulting in a false alarm rate of up to 15%.

[0003] Therefore, it is necessary to invent a wheeled inspection robot and its camera motion understanding method and system to solve the above problems. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a wheeled inspection robot and a camera motion understanding method and system thereof, which obtains displacement, video stream data and posture data in real time through a data acquisition module; the camera motion classification module performs motion classification and visual detection on the video frames, outputs geometric primitives and semantic primitive labels and equipment status detection results, and maps them to a reference frame; the SfM-VLM fusion reasoning module extracts the geometric parameters of the camera motion through the SfM algorithm, fuses the visual features and inputs them into the VLM model to generate a natural language semantic description; the structured annotation module generates dual-track annotations of the label layer and the subtitle layer based on the geometric primitives, semantic primitive labels and natural language semantic descriptions; the multimodal decision module fuses the geometric primitives and semantic primitive labels, the equipment status detection results and the data collected by the data acquisition module to generate inspection decisions, effectively solving the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: a wheeled inspection robot and a camera motion understanding method and system thereof, specifically comprising: a wheeled inspection robot, a data acquisition module, a camera motion classification module, a SfM-VLM fusion reasoning module, a structured annotation module, a multimodal decision module, and a computer-readable storage medium, wherein the wheeled inspection robot is equipped with a high-precision encoder, the data acquisition module includes a vision module and a multi-sensor fusion unit, the vision module includes an RGB camera and an infrared camera, and the multi-sensor fusion unit includes an inertial measurement unit (IMU) and a lidar; Data acquisition module: real-time acquisition of displacement data, video stream data and posture auxiliary data; Camera motion classification module: Based on the ResNet+Transformer model, it preprocesses video frames, extracts visual features, and performs temporal modeling. It outputs geometric primitive and semantic primitive labels and device status detection results, and maps them to a reference frame. The SfM-VLM fusion reasoning module extracts the geometric parameters of camera motion using the Structure from Motion (SfM) algorithm and converts them into semantic intent described in natural language using the Visual Language Model (VLM). Structured annotation module: generates dual-track annotations for label layer and subtitle layer; Multimodal decision module: This module integrates geometric primitives and semantic primitive labels, equipment status detection results, and data collected by the data acquisition module to generate inspection control instructions and structured reports. Computer-readable storage medium: stores a program that implements the functions of the above-mentioned data acquisition module, camera motion classification module, SfM-VLM fusion reasoning module, structured annotation module and multimodal decision module.

[0006] Preferably, the displacement data includes the translation distance and heading angle output by the chassis encoder; the video stream data includes RGB video and infrared thermal imaging video; the posture auxiliary data includes the angular velocity, acceleration and lidar point cloud of the IMU, and the geometric parameters include the translation vector, rotation matrix and focal length change.

[0007] Preferably, the geometric primitives include translation, rotation, and scaling; the semantic primitives include tracking, arc motion, and scene revealing; and the reference frame includes a camera center, an object center, and a ground center.

[0008] Preferably, the specific execution steps of the camera motion classification module are: Preprocess the input video frames by image scaling and normalization; Use the ResNet model to extract visual features; Use the Transformer model to perform temporal modeling of continuous frame features; Leveraging a pre-trained camera motion classification model, the system outputs motion primitive labels and confidence scores based on visual features, while simultaneously performing device visual inspection. The motion primitive labels include geometric and semantic primitive categories. Map motion primitive labels to camera-centered, object-centered, or ground-centered reference frames.

[0009] Preferably, the specific execution steps of the SfM-VLM fusion reasoning module are: Based on the video frame sequence and IMU data, the camera pose trajectory is calculated using the improved COLMAP algorithm; Extract geometric parameters; Fusion of geometric parameters and visual features to construct input; The input is converted into natural language semantic description through the fine-tuned generative visual language model Qwen2.5VL model.

[0010] Preferably, the specific execution steps of the structured annotation module are: Generate a label layer to mark the geometric motion type, semantic motion type, stability status and equipment detection results; Caption layer generation, combining motion intention and scene information to generate natural language description through VLM; Integrate the label layer and the subtitle layer to output structured annotations with timestamps.

[0011] Preferably, the specific execution steps of the multimodal decision module are: Align camera motion semantics with multi-sensor data based on timestamps; Determine the authenticity of equipment anomalies based on motion stability status; If there is an abnormality, an alarm instruction and a fault report will be generated; otherwise, a normal inspection report will be generated; Output robot control instructions based on motion semantic intentions.

[0012] Preferably, the wheeled inspection robot and the camera motion understanding method thereof specifically include the following steps: S1, acquiring displacement data, video stream data and posture data in real time through a data acquisition module; S2, the camera motion classification module performs motion classification and visual detection on the video frames, outputs geometric primitives and semantic primitive labels and device status detection results, and maps them to the reference frame; S3, the SfM-VLM fusion reasoning module extracts the geometric parameters of camera motion through the SfM algorithm, fuses the visual features and inputs them into the VLM model to generate natural language semantic description; S4, the structured annotation module generates dual-track annotations of the label layer and the subtitle layer based on geometric primitives, semantic primitive labels and natural language semantic descriptions; S5. The multimodal decision module integrates geometric primitives and semantic primitive labels, equipment status detection results and data collected by the data acquisition module to generate inspection decisions.

[0013] Technical effects and advantages of the present invention: 1. Improved adaptability to dynamic environments: By decoupling motion primitives, the pose estimation error of SLAM in dynamic scenes is reduced by more than 40%. For example, during inspections of running conveyor belts, the 3D reconstruction accuracy is improved to ±2cm. 2. Enhanced target tracking reliability: A tracking strategy based on semantic primitives reduces target loss by 60% in complex motion scenarios. For example, valve detection accuracy increases from 72% to 91% when simultaneously translating and rotating. 3. Data annotation standardization: The structured annotation framework improves the annotation consistency of inspection datasets by 85% and improves the model's generalization across scenarios by 20-30%. 4. Intelligent multimodal decision-making: Motion-enhanced report generation reduces manual misjudgment by 50%, and the false alarm rate of equipment heating failure is reduced from 15% to less than 5%. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0015] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0016] Figure 2 It is a flow chart of the overall steps of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] The present invention provides a wheeled inspection robot and a camera motion understanding system thereof, the structure of which is as follows: Figure 1As shown, specifically includes: wheeled inspection robot, data acquisition module, camera motion classification module, SfM-VLM fusion reasoning module, structured annotation module, multi-modal decision module and computer readable storage medium, the wheeled inspection robot is equipped with high-precision encoder, the data acquisition module includes vision module and multi-sensor fusion unit, the vision module includes RGB camera and infrared camera, the multi-sensor fusion unit contains inertial measurement unit IMU and laser radar.

[0019] Further, in the above technical solution, the data acquisition module is used for acquiring displacement data, video stream data and pose auxiliary data in real time; The camera motion classification module is used for pre-processing, visual feature extraction and time series modeling of video frames based on a ResNet+Transformer model, and outputs geometric primitive and semantic primitive labels and device state detection results, and maps to a reference frame; The SfM-VLM fusion reasoning module is used for extracting geometric parameters of camera motion through a motion recovery structure SfM algorithm, and converting the geometric parameters into natural language description semantic intent using a visual language model VLM; The structured annotation module is used for generating dual-track annotations of label layer and subtitle layer; The multi-modal decision module is used for fusing geometric primitive and semantic primitive labels, device state detection results and data collected by the data acquisition module, to generate inspection control instructions and structured reports; The computer readable storage realizes the functions of the above-mentioned data acquisition module, camera motion classification module, SfM-VLM fusion reasoning module, structured annotation module and multi-modal decision module.

[0020] As shown, Figure 2 The wheeled inspection robot and the camera motion understanding method thereof specifically include the following steps: S1, acquiring displacement data, video stream data and pose data in real time through a data acquisition module; Further, in the above technical solution, the displacement data includes translation distance and heading angle output by a chassis encoder; the video stream data includes RGB video and infrared thermal imaging video; the pose auxiliary data includes angular velocity, acceleration of the IMU and laser radar point cloud, and the geometric parameters include translation vector, rotation matrix and focal length change.

[0021] S2, the camera motion classification module classifies and detects the motion of the video frame, and outputs geometric primitive and semantic primitive labels and device state detection results, and maps to a reference frame; Furthermore, in the above technical solution, the geometric primitives include translation, rotation, and scaling; the semantic primitives include tracking, arc motion, and scene revealing; and the reference frame includes the camera center, the object center, and the ground center.

[0022] Furthermore, in the above technical solution, the specific execution steps of the camera motion classification module are: Preprocess the input video frames by image scaling and normalization; Use the ResNet model to extract visual features; Use the Transformer model to perform temporal modeling of continuous frame features; Leveraging a pre-trained camera motion classification model, the system outputs motion primitive labels and confidence scores based on visual features, while simultaneously performing device visual inspection. The motion primitive labels include geometric and semantic primitive categories. Map motion primitive labels to camera center, object center, or ground center references.

[0023] It should be further explained that the camera motion classification model is trained based on a classification system containing more than 50 motion primitives, takes video stream as input, and outputs motion primitive labels.

[0024] It should be further explained that after the camera motion classification module outputs the motion primitive label, it needs to select a reference frame based on the scene dynamics and motion stability. The mapping logic is as follows: When the target object occupies a dominant position in the field of view, that is, the target detection box covers more than 30% of the image area, the motion semantics is based on the target, that is, the motion primitive label is mapped to the object center frame; When the robot moves smoothly, the IMU acceleration variance is less than 0.1m / s 2 ,The motion semantics is based on the ground coordinate system, i.e. the motion primitive labels are mapped to the ground center frame; When the above two rules are not met, that is, when the target is small or the robot motion is shaking violently, the camera's own coordinate system is used as the default reference, that is, the motion primitive label is mapped to the camera center frame.

[0025] S3, the SfM-VLM fusion reasoning module extracts the geometric parameters of camera motion through the SfM algorithm, fuses the visual features and inputs them into the VLM model to generate natural language semantic description; Furthermore, in the above technical solution, the specific execution steps of the SfM-VLM fusion reasoning module are as follows: Based on the video frame sequence and IMU data, the camera pose trajectory is calculated using the improved COLMAP algorithm; Extract geometric parameters; Fusion of geometric parameters and visual features to construct input; The input is converted into natural language semantic description through the fine-tuned generative visual language model Qwen2.5VL model.

[0026] It should be further explained that the improved COLMAP algorithm initializes the camera pose using IMU data and uses dynamic object masks to filter the point cloud to reduce the error in dynamic scenes. The Qwen2.5VL model fine-tuning method uses the COCO-Motion dataset, with geometric parameters + ResNet features as input and semantic descriptions as labels for supervised fine-tuning. S4, the structured annotation module generates dual-track annotations of the label layer and the subtitle layer based on geometric primitives, semantic primitive labels and natural language semantic descriptions; Furthermore, in the above technical solution, the specific execution steps of the structured annotation module are: Generate a label layer to mark the geometric motion type, semantic motion type, stability status and equipment detection results; Caption layer generation, combining motion intention and scene information to generate natural language description through VLM; Integrate the label layer and the subtitle layer to output structured annotations with timestamps.

[0027] S5. The multimodal decision module integrates geometric primitives and semantic primitive labels, equipment status detection results and data collected by the data acquisition module to generate inspection decisions.

[0028] Furthermore, in the above technical solution, the specific execution steps of the multimodal decision module are: Align camera motion semantics with multi-sensor data based on timestamps; Determine the authenticity of equipment anomalies based on motion stability status; If there is an abnormality, an alarm instruction and a fault report will be generated; otherwise, a normal inspection report will be generated; Output robot control instructions based on motion semantic intentions.

[0029] Taking the power inspection scenario as an example, the wheeled robot starts from a starting point and inspects high-voltage equipment along a preset path: 1. Data acquisition: The chassis encoder outputs real-time displacement data, including translation distance and heading angle. The RGB camera captures 1080p video streams, the infrared camera outputs 640×480 thermal imaging video, the IMU provides angular velocity and acceleration, and the lidar generates a point cloud. 2. Camera motion classification and device detection: RGB frames are scaled to 256×256 and pixel values ​​are normalized to [0, 1]. Visual features are extracted using ResNet-50, and the YOLOv7 model is run simultaneously to detect device status. The Transformer encoder processes 10 consecutive frames of features and outputs motion primitive labels. The motion primitive labels are mapped to a reference frame based on the position of the target detection box. 3. SfM-VLM fusion reasoning: The camera trajectory is calculated based on the SfM algorithm to obtain the translation vector and rotation matrix. The geometric parameters and extracted visual features are fused to construct the input. The input is converted into a natural language semantic description through the fine-tuned generative visual language model Qwen2.5VL model. For example, in a certain acquired data, the translation vector is 0.3 / s and the rotation matrix is ​​a 20° upward rotation. Through visual inspection, a 5mm crack is found on the top insulator of the transformer. The Qwen2.5VL model converts this into "The camera moves forward at a speed of 0.3m / s and rotates upward by 20°, intending to check the top insulator of the transformer." It also associates the visually detected insulator crack and generates an annotation such as "The crack is approximately 5mm long and is located on the third insulator from the left."

[0030] 4. Multimodal decision generation: If an infrared camera detects a crack area temperature of 65°C, which is 20°C higher than the ambient temperature, and the camera motion is stationary, excluding motion artifacts, the multimodal decision module determines that this anomaly is a real fault, triggers an alarm, and generates a report containing the motion trajectory, visual images, temperature heat map, and semantic description.

[0031] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A wheeled inspection robot and its camera motion understanding system, characterized by: include: A wheeled inspection robot, a data acquisition module, a camera motion classification module, a SfM-VLM fusion reasoning module, a structured annotation module, a multimodal decision module, and a computer-readable storage medium. The wheeled inspection robot is equipped with a high-precision encoder. The data acquisition module integrates a vision module and a multi-sensor fusion unit. The vision module includes an RGB camera and an infrared camera. The multi-sensor fusion unit includes an inertial measurement unit (IMU) and a lidar. Data acquisition module: real-time acquisition of displacement data, video stream data and posture auxiliary data; Camera motion classification module: Based on the ResNet+Transformer model, it preprocesses video frames, extracts visual features, and performs temporal modeling. It outputs geometric primitive and semantic primitive labels and device status detection results, and maps them to a reference frame. The SfM-VLM fusion reasoning module extracts the geometric parameters of camera motion using the Structure from Motion (SfM) algorithm and converts them into semantic intent described in natural language using the Visual Language Model (VLM). Structured annotation module: generates dual-track annotations for label layer and subtitle layer; Multimodal decision module: This module integrates geometric primitives and semantic primitive labels, equipment status detection results, and data collected by the data acquisition module to generate inspection control instructions and structured reports. Computer-readable storage medium: stores a program that implements the functions of the above-mentioned data acquisition module, camera motion classification module, SfM-VLM fusion reasoning module, structured annotation module and multimodal decision module.

2. The wheeled inspection robot and camera motion understanding system thereof according to claim 1, characterized in that: The displacement data includes the translation distance and heading angle output by the chassis encoder; the video stream data includes RGB video and infrared thermal imaging video; the posture auxiliary data includes the angular velocity, acceleration and lidar point cloud of the IMU, and the geometric parameters include the translation vector, rotation matrix and focal length change.

3. The wheeled inspection robot and camera motion understanding system thereof according to claim 1, characterized in that: The geometric primitives include translation, rotation, and scaling; the semantic primitives include tracking, arc motion, and scene revealing; and the reference frame includes camera center, object center, and ground center.

4. The wheeled inspection robot and camera motion understanding system thereof according to claim 1, characterized in that: The specific execution steps of the camera motion classification module are: Preprocess the input video frames by image scaling and normalization; Use the ResNet model to extract visual features; Use the Transformer model to perform temporal modeling of continuous frame features; Leveraging a pre-trained camera motion classification model, the system outputs motion primitive labels and confidence scores based on visual features, while simultaneously performing device visual inspection. The motion primitive labels include geometric and semantic primitive categories. Map motion primitive labels to camera-centered, object-centered, or ground-centered reference frames.

5. The wheeled inspection robot and camera motion understanding system thereof according to claim 1, characterized in that: The specific execution steps of the SfM-VLM fusion reasoning module are as follows: Based on the video frame sequence and IMU data, the camera pose trajectory is calculated using the improved COLMAP algorithm; Extract geometric parameters; Fusion of geometric parameters and visual features to construct input; The input is converted into natural language semantic description through the fine-tuned generative visual language model Qwen2.5VL model.

6. The wheeled inspection robot and camera motion understanding system thereof according to claim 1, characterized in that: The specific execution steps of the structured annotation module are: Generate a label layer to mark the geometric motion type, semantic motion type, stability status and equipment detection results; Caption layer generation, combining motion intention and scene information to generate natural language description through VLM; Integrate the label layer and the subtitle layer to output structured annotations with timestamps.

7. The wheeled inspection robot and camera motion understanding system thereof according to claim 1, characterized in that: The specific execution steps of the multimodal decision module are: Align camera motion semantics with multi-sensor data based on timestamps; Determine the authenticity of equipment anomalies based on motion stability status; If there is an abnormality, an alarm instruction and a fault report will be generated; otherwise, a normal inspection report will be generated; Output robot control instructions based on motion semantic intentions.

8. A wheeled inspection robot and a camera motion understanding method thereof, characterized by: The specific steps include: S1, obtain displacement data, video stream data and posture data in real time through the data acquisition module; S2, the camera motion classification module performs motion classification and visual detection on the video frames, outputs geometric primitives and semantic primitive labels and device status detection results, and maps them to the reference frame; S3, the SfM-VLM fusion reasoning module extracts the geometric parameters of camera motion through the SfM algorithm, fuses the visual features and inputs them into the VLM model to generate natural language semantic description; S4, the structured annotation module generates dual-track annotations of the label layer and the subtitle layer based on geometric primitives, semantic primitive labels and natural language semantic descriptions; S5. The multimodal decision module integrates geometric primitives and semantic primitive labels, equipment status detection results and data collected by the data acquisition module to generate inspection decisions.

Images