Method for automatically drawing inspection point position by inspection robot based on multiple sensors

Abstract

The invention discloses a method for automatically drawing inspection point positions by an inspection robot based on multiple sensors, and relates to the technical field of intelligent inspection robots, and the method comprises the following steps: S1, synchronously collecting RGB images and infrared images of equipment to be inspected through a visible light camera and an infrared thermal imager carried by the robot; and S2, carrying out fusion processing on the RGB image and the infrared image by using a deep learning visual identification model, identifying target equipment and generating a target detection frame. According to the method for automatically drawing the inspection point location by the inspection robot based on the multiple sensors, automatic detection and accurate coordinate mapping of the inspection point location are realized through multi-modal sensor fusion and deep learning recognition technologies, the dependence of a traditional method on manual presetting is effectively reduced, and the automation level and the positioning accuracy of point location drawing are improved. And meanwhile, dynamic path planning and closed-loop environment perception are combined, so that the adaptability of the system to a complex industrial scene and the routing inspection reliability are enhanced.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection robot technology, specifically a method for automatically mapping inspection points using a multi-sensor-based inspection robot. Background Technology

[0002] In intelligent inspection applications in power systems and industrial manufacturing, path planning and dynamic obstacle avoidance control of inspection robots are among the core technologies for achieving autonomous operation. Existing methods largely rely on manual pre-calibration of the inspection target's coordinates, resulting in fixed inspection paths and a lack of adaptability to environmental changes. Especially in scenarios with complex and variable equipment layouts, such as substations, traditional methods cannot achieve automatic identification and dynamic updating of inspection points, severely limiting the robot's autonomy and inspection efficiency. The urgent problem to be solved is how to achieve dynamic planning and real-time optimization of inspection paths, including accurately and automatically determining inspection coordinates in complex environments, and based on this, performing obstacle avoidance and trajectory smoothing control, thereby improving the overall system's intelligence level and operational reliability. Summary of the Invention

[0003] To achieve the above objectives, the present invention provides the following technical solution: a method for automatically mapping inspection points using a multi-sensor-based inspection robot, comprising the following steps: S1: Simultaneously acquire RGB and infrared images of the equipment to be inspected using a visible light camera and an infrared thermal imager mounted on the robot; S2: The RGB image and infrared image are fused using a deep learning visual recognition model to identify the target device and generate a target detection box; S3: Generate a spatial location heatmap and a semantic distribution map based on the target detection box, and filter candidate inspection points according to a preset confidence threshold; S4: Based on the spatial quadrant partitioning of the candidate inspection points in the camera's field of view, the corresponding coordinate mapping model is used to transform the image pixel coordinates to the world coordinate system with the robot as the origin. S5: Cross-validate the obtained preliminary world coordinates with the temperature anomaly area based on infrared image analysis to determine the final inspection point coordinates; S6: Integrate the coordinates of all final inspection points to form an inspection point set, input the path planning algorithm and combine it with real-time environmental obstacle information to generate the final inspection path.

[0004] Preferably, the spatial quadrant partitioning is based on the optical center of the robot camera as the origin, and a coordinate system is established with the optical axis direction, horizontal direction and vertical direction to divide the field of view into four quadrants; the coordinate mapping model establishes the mapping relationship from image pixel coordinates to world coordinates for different quadrants, and corrects image edge distortion.

[0005] Preferably, for candidate inspection points located at the edge of the field of view or in the blind spot, a blind spot compensation algorithm is used during the coordinate transformation process. The blind spot compensation algorithm includes adjusting the robot pose or the camera viewpoint.

[0006] Preferably, the spatial location cross-validation includes: determining whether the preliminary world coordinates are located within a temperature anomaly region; if so, determining it as the final inspection point; otherwise, verifying it in conjunction with semantic categories, and retaining points belonging to the category of easily overheating equipment. The extraction of the temperature anomaly region is a process based on image processing and analysis. The system first acquires an image containing radiation data from an infrared thermal imager and converts it into a temperature matrix. Subsequently, the system calls historical temperature data from normal equipment operation or preset safe surface temperature limits for the equipment as a reference benchmark. By comparing the real-time temperature matrix with the benchmark value, an adaptive thresholding algorithm, such as Otsu's method or a region growing algorithm, is used to identify continuous pixel regions with temperatures significantly higher than the surrounding environment or the benchmark value; these regions are then marked as "temperature anomaly regions." The temperature tolerance limit for this process can be configured according to the equipment type, ambient temperature, and load conditions.

[0007] Preferably, the path planning algorithm adopts a model predictive control framework. The global path planning uses the coordinates of the final inspection point as the path point and satisfies the robot's kinematic constraints. The local real-time obstacle avoidance adjusts the global path online based on the environmental information perceived by the sensors.

[0008] Preferably, during the execution of the inspection route, the coordinates of the final inspection points are dynamically corrected and supplemented by continuously collected sensor data and by using synchronous positioning and mapping technology.

[0009] Preferably, the visible light camera is an industrial-grade camera with automatic white balance and wide dynamic range functions, and the infrared thermal imager is an uncooled infrared thermal imager; when acquiring images, the RGB image and the infrared image are ensured to be spatiotemporally aligned.

[0010] Preferably, the deep learning visual recognition model is a dual-branch network structure that processes RGB images and infrared images respectively, and outputs the target detection result through a feature fusion layer.

[0011] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for automatically drawing inspection points using a multi-sensor-based inspection robot.

[0012] An inspection robot system includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the steps of a method for automatically drawing inspection points based on a multi-sensor inspection robot.

[0013] This invention provides a method for automatically mapping inspection points using a multi-sensor-based inspection robot. It offers the following advantages: This method for automatically mapping inspection points using a multi-sensor-based inspection robot achieves automatic detection and precise coordinate mapping of inspection points through multi-modal sensor fusion and deep learning recognition technology. This effectively reduces the reliance on manual pre-setting in traditional methods, improving the automation level and positioning accuracy of point mapping. Furthermore, the integration of dynamic path planning and closed-loop environmental perception enhances the system's adaptability to complex industrial scenarios and improves inspection reliability.

[0014] This method for automatically mapping inspection points using a multi-sensor-based inspection robot, through quadrant coordinate transformation, multi-source data cross-validation, and a continuous map update mechanism, enables the system to autonomously cope with changes in equipment layout and environmental interference, thereby improving the accuracy and practicality of inspection data. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the system module interaction of the method for automatically drawing inspection points using a multi-sensor inspection robot according to the present invention; Figure 2 This is a flowchart illustrating the method for automatically mapping inspection points using a multi-sensor inspection robot according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 and Figure 2 This invention provides a technical solution: a method for automatically mapping inspection points using a multi-sensor-based inspection robot, comprising the following steps: S1: Simultaneously acquire RGB and infrared images of the equipment to be inspected using a visible light camera and an infrared thermal imager mounted on the robot; S2: Use a deep learning visual recognition model to fuse RGB and infrared images, identify target devices, and generate target detection boxes; S3: Generate spatial location heatmaps and semantic distribution maps based on target detection boxes, and filter candidate inspection points according to preset confidence thresholds; S4: Based on the spatial quadrant partitioning of the candidate inspection points in the camera's field of view, the corresponding coordinate mapping model is used to transform the image pixel coordinates to the world coordinate system with the robot as the origin. S5: Cross-validate the obtained preliminary world coordinates with the temperature anomaly area based on infrared image analysis to determine the final inspection point coordinates; S6: Integrate the coordinates of all final inspection points to form an inspection point set, input the path planning algorithm and combine it with real-time environmental obstacle information to generate the final inspection path.

[0018] It should be further explained that, in the specific implementation process, the inspection robot first simultaneously collects RGB and infrared images of the equipment to be inspected. Then, a deep learning-based visual recognition model fuses the two types of images to identify the target equipment and generate target detection boxes. Based on the detection box results, the system automatically generates a heatmap reflecting the spatial probability distribution of the target and a semantic distribution map representing the recognition results of different categories, and selects candidate inspection points according to a preset confidence threshold.

[0019] For each candidate point, the system uses a corresponding coordinate mapping strategy to transform its pixel coordinates to a world coordinate system with the robot as the origin, based on its location in different quadrants of the camera's field of view. For points at the edge of the field of view or in blind spots, a blind spot compensation algorithm is activated to correct the position by adjusting the robot's pose or the camera's viewpoint. Then, the preliminary world coordinates are cross-validated with the temperature anomaly areas extracted based on infrared image analysis: if the coordinates are within the temperature anomaly area, they are directly confirmed as the final inspection point; if they are not within the anomaly area but their semantic category belongs to heat-prone equipment, the point is marked and verified; otherwise, it is discarded.

[0020] Finally, all verified inspection points are aggregated into an ordered set and input into the path planning algorithm. This algorithm combines real-time acquired environmental obstacle information to perform global path planning and local real-time obstacle avoidance adjustments, outputting an executable inspection path that satisfies the robot's motion constraints. Throughout the process, the system continuously updates and corrects the point coordinates and environmental map online using newly acquired sensor data and SLAM technology to adapt to environmental changes.

[0021] Spatial quadrant partitioning uses the optical center of the robot's camera as the origin, establishing a coordinate system along the optical axis, horizontal direction, and vertical direction to divide the field of view into four quadrants. A coordinate mapping model establishes a mapping relationship from image pixel coordinates to world coordinates for each quadrant, and corrects image edge distortion. It should be further explained that in the specific implementation, the spatial quadrant division uses the optical center of the camera mounted on the robot as the spatial origin, and constructs a three-dimensional coordinate system along the camera's optical axis, horizontal direction, and vertical direction. Based on this coordinate system, the entire field of view captured by the camera is systematically divided into four different spatial quadrants. For candidate inspection points located in different quadrants, the system does not use a single mapping method, but rather, based on the unique spatial geometry of its quadrant, it calls a pre-established coordinate mapping model that matches that quadrant to complete the conversion from two-dimensional image pixel coordinates to three-dimensional world coordinates.

[0022] This process fully considers the spatial orientation characteristics of the points. Furthermore, to address the geometric distortion that may be introduced when a wide-angle lens captures image edge areas, the system integrates a dedicated distortion correction algorithm during coordinate mapping to correct the coordinate data of these points in real time, ensuring that the final converted world coordinates have higher spatial accuracy and reliability.

[0023] The establishment of the quadrant coordinate mapping model relies on the precise calibration performed beforehand by the system. Within each quadrant, the operator controls the robot to capture an image of a precision calibration board with known world coordinates from multiple different poses, including both visible and infrared features. For each calibration point, its image pixel coordinates (u, v) and real-world coordinates (X, Y, Z) are recorded. Based on the large number of calibration point pairs collected in each quadrant, an optimal projection transformation matrix, or a set of polynomial parameters, is fitted using the least squares method. This matrix defines the mapping relationship from image coordinates to world coordinates within that quadrant. This process is performed independently for each of the four quadrants, ultimately yielding four sets of mapping parameters that together constitute the complete coordinate mapping model.

[0024] For candidate inspection points located at the edge of the field of view or in blind spots, a blind spot compensation algorithm is used during coordinate transformation. This algorithm includes adjusting the robot's pose or the camera's viewing angle. It should be further explained that in the specific implementation, for candidate inspection points identified and mapped as being located at the edge of the camera's field of view or in blind spots, the system will automatically invoke the blind spot compensation algorithm for processing. This algorithm first analyzes the orientation and distance of the target point relative to the camera's optical center and the current field of view boundary. If the point enters the blind spot due to partial occlusion by an object or being at the lens's limit of field of view, the algorithm generates a pose adjustment command, controlling the robot chassis to make appropriate lateral or longitudinal movements to change the relative position of the robot as a whole with the target device, thereby attempting to bring the target back into the effective field of view. If the target is in a blind spot caused by the limited rotation range of the camera's pan / tilt unit, the algorithm will prioritize calculating and controlling the pan / tilt unit to adjust its pitch or horizontal angle to optimize the camera's pointing.

[0025] The blind spot compensation algorithm is a feedback-based control process. When a target point is determined to be in the blind spot, the algorithm first estimates its azimuth and pitch angles relative to the robot's current pose based on its world coordinates. Next, the algorithm calculates the required horizontal rotation angle Δψ and pitch angle Δθ of the gimbal to bring the point into the center of the camera's field of view. The control system prioritizes sending commands to drive the gimbal to rotate. If rotating the gimbal to its mechanical limits still fails to bring the target into the field of view, the algorithm further calculates the required robot chassis movement vector (Δx, Δy), controls the robot to move a certain distance, and then attempts to identify the target again. Throughout the process, the algorithm continuously monitors whether the target point reappears in the image; once successful detection is achieved, compensation stops.

[0026] Throughout the adjustment process, the system continuously monitors changes in the image coordinates of the target point, forming a closed-loop feedback loop until the coordinates of the point are successfully mapped or ultimately determined to be unreachable. This active pose and viewpoint adjustment strategy based on spatial judgment aims to expand the effective detection range and reduce missed detections due to limited field of view.

[0027] Spatial location cross-validation includes: determining whether the preliminary world coordinates are located within a temperature anomaly area; if so, it is determined as the final inspection point; otherwise, it is reviewed in conjunction with semantic category, and points belonging to the category of easily overheating equipment are retained. It should be further explained that, in specific implementation, the spatial location cross-validation process is as follows: The system first compares the preliminary world coordinates obtained based on image mapping with the temperature anomaly areas identified by infrared image analysis. If the preliminary coordinates are determined to be within the spatial range of any temperature anomaly area, the system directly adopts it as the final valid inspection point. If the preliminary coordinates do not fall into any temperature anomaly area, the system initiates a review mechanism, retrieving the equipment semantic category information determined by the image recognition model for that point.

[0028] For points whose semantic categories belong to the pre-set list of heat-prone equipment such as circuit breakers, busbar connection points, and transformer bushings, the system will not discard them immediately, but will mark them as "points to be verified" and package and store their coordinates, category, and infrared temperature data, prompting for key attention and secondary confirmation in subsequent inspection cycles or by maintenance personnel.

[0029] For candidate locations that are neither within the temperature anomaly zone nor belong to the category of equipment prone to overheating, the system determines them as low-priority or misidentified locations and eliminates them. This cross-validation mechanism, by integrating two physical attributes—visual appearance and thermal characteristics—to make joint decisions, aims to improve the ability to perceive and locate abnormal conditions in critical equipment, especially heat-related faults.

[0030] The path planning algorithm employs a model predictive control framework. Global path planning uses the coordinates of the final inspection points as path points and satisfies robot kinematic constraints. Local real-time obstacle avoidance adjusts the global path online based on sensor-perceived environmental information. It should be further noted that in the specific implementation, the path planning algorithm is processed using the model predictive control framework. This framework first receives all validated final inspection point coordinates and uses them as critical path points that the global path must traverse. When generating the initial global path, the algorithm strictly considers the robot's own kinematic constraints, such as minimum turning radius and maximum acceleration limits, ensuring that the planned path is theoretically a smooth trajectory that the robot can execute.

[0031] After the robot begins moving along the global path, the system does not stop planning but continues to run a local real-time obstacle avoidance thread. This thread continuously receives real-time environmental perception data from near-field sensors such as LiDAR and ultrasonic sensors. Once a dynamic obstacle, such as a moving person or vehicle, or a static obstacle not pre-marked on the global map, such as a temporarily placed object, is detected ahead of the preset path, the local planner is immediately activated.

[0032] Guided by the global path, it quickly calculates and generates a safe and smooth local obstacle avoidance trajectory around obstacles. This local trajectory, while satisfying the same kinematic constraints, aims to safely avoid all unexpected obstacles while minimizing deviations from the original global path. After obstacle avoidance, the robot automatically returns to the original global path to continue its inspection task.

[0033] During the execution of the inspection path, the coordinates of the final inspection points are dynamically corrected and supplemented by continuously collected sensor data and simultaneous localization and mapping (SLAM) technology. It should be further noted that, in the specific implementation, while the robot is executing the predetermined inspection path, the system continuously collects environmental data through its onboard visible light camera, infrared thermal imager, and LiDAR sensors. This real-time data is input into a graph-optimized SLAM framework.

[0034] This framework is used not only to estimate the robot's precise pose, such as position and orientation, in real time, but also to incrementally update and expand the environmental map, with particular attention to changes in equipment layout, such as the opening and closing status of power distribution cabinet doors or the placement of temporary equipment.

[0035] Based on this dynamically updated environmental awareness, the system makes online corrections to the coordinates of the final inspection points: if the SLAM process detects that a device has moved relative to the environment map, the system will recalculate and update the coordinates of the inspection point corresponding to that device; if new potential target devices are identified through continuous image recognition near existing devices or in the original blank areas, and confirmed by the aforementioned multimodal cross-validation process, the system will add the coordinates of these new points to the inspection point set.

[0036] This closed-loop process ensures that the inspection point map is not static but can adapt to environmental changes, continuously maintaining its accuracy and integrity, thus providing dynamic adaptability for inspection tasks.

[0037] Simultaneous localization and mapping (SLAM) technology is used not only for robot localization and obstacle map construction, but also for tracking and mapping detected devices, such as power distribution cabinets and meters, as semantic landmarks. SLAM backend optimization continuously refines the 3D position estimates of these semantic landmarks. The system maintains a database of inspection points, where the world coordinates of each point are associated with one or more SLAM semantic landmark IDs. Periodically, the system compares the latest landmark position output by the SLAM module with the associated old coordinates in the database. If the Euclidean distance between the two exceeds a preset tolerance (this tolerance can be set according to positioning accuracy and device importance), the system automatically updates the inspection point coordinates in the database with the new landmark position and records the update log. For newly added semantic landmarks not recorded in the database, they will be prompted to be added as new inspection points after confirmation.

[0038] The visible light camera is an industrial-grade camera with automatic white balance and wide dynamic range, while the infrared thermal imager is an uncooled infrared thermal imager. During image acquisition, the RGB and infrared images are ensured to be spatiotemporally aligned. It should be further noted that in the specific implementation process, the selected visible light camera, in the image acquisition hardware configuration and data preprocessing stages, is an industrial-grade camera with automatic white balance and wide dynamic range. Its sensor size, pixel count, and optical performance meet the requirements for capturing high-fidelity RGB images under typical industrial lighting conditions. The selected infrared thermal imager uses an uncooled microbolo calorimeter core, possessing radiation data storage capabilities and able to record the scene's temperature information. During synchronous acquisition, the system ensures that the RGB and infrared images acquired at the same time are strictly time-correlated through a unified hardware trigger signal or a high-precision timestamp mechanism.

[0039] To achieve spatial alignment, coordinate transformation parameters between the two sensors, such as rotation matrices and translation vectors, need to be pre-calculated and stored during the system calibration phase. After each acquisition, these calibration parameters are used to reproject the infrared image, aligning it with the RGB image in the spatial field of view. The spatiotemporally aligned image data set will be stored as a coherent whole data unit, providing a data foundation for subsequent fusion processing and joint analysis.

[0040] The deep learning visual recognition model employs a dual-branch network structure, processing RGB and infrared images separately, and outputting target detection results through a feature fusion layer. It's important to further clarify that in practice, the deep learning visual recognition model is constructed and operates using a dual-branch network architecture. One branch specifically receives and processes the RGB three-channel image acquired by a visible light camera, its main function being to extract the target's texture, color, shape, and other visual features. The other branch receives and processes the single-channel infrared image acquired by an infrared thermal imager in parallel; the core function of this branch is to extract the target's thermal radiation distribution and temperature field features. After completing their respective feature extractions, the high-level feature tensors output by both branches are fed into a shared feature fusion layer.

[0041] This fusion layer does not simply perform feature concatenation or weighted summation. Instead, it employs a fusion strategy based on channel and spatial attention mechanisms to dynamically evaluate and integrate feature information from two different modalities, emphasizing the complementary and discriminative components of infrared and visible light features. The unified feature tensor after fusion is ultimately input into the subsequent detection head network, outputting a target detection box, category confidence score, and semantic label that simultaneously contains visual appearance and thermal characteristic information. This dual-branch fusion design aims to overcome the limitations of single-modal perception and improve the robustness and accuracy of target device recognition, especially for targets exhibiting thermal anomalies, in complex industrial environments.

[0042] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for automatically mapping inspection points using a multi-sensor-based inspection robot. Further explanation is needed: In the specific implementation process, when the computer program stored in the computer-readable storage medium is executed by the processor, it first initializes the hardware units of the inspection robot system, including a visible light camera, an infrared thermal imager, and environmental perception sensors, and establishes time synchronization and spatial calibration relationships between the sensors. The program controls the robot to begin inspection operations according to a preset mode, simultaneously acquiring RGB image and infrared image data streams. For each set of acquired image data, the program calls a pre-loaded dual-branch deep learning model for parallel processing and feature fusion, performs object detection, and generates candidate boxes with semantic labels. Subsequently, the program filters these candidate boxes according to a predefined confidence mechanism, and initiates a quadrant coordinate mapping process for the selected candidate points, transforming their image coordinates to the world coordinate system.

[0043] After obtaining the initial world coordinates, the program initiates multimodal cross-validation logic: retrieves the infrared temperature data analysis results of the corresponding area, determines whether the coordinates are located in an abnormal temperature area, or whether they belong to a type of equipment that requires special attention, and decides whether to include them in the final inspection point set based on the validation results.

[0044] The program then inputs all confirmed points into the path planning module. This module comprehensively considers the robot's motion constraints and real-time perceived environmental obstacle information, recursively calculating and optimizing to generate a safe and executable inspection path. Throughout the process, the program continuously collects data and performs SLAM processing to dynamically monitor environmental changes and corrects and updates the drawn points and paths online. All generated path and point data are structured and stored and output to the human-machine interface for monitoring and operation.

[0045] An inspection robot system includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement a method for automatically mapping inspection points using a multi-sensor-based inspection robot. Further, in practical implementation, this inspection robot system integrates a visible light camera, an infrared thermal imager, a lidar, an ultrasonic sensor, and a chassis and gimbal mechanism for movement and attitude adjustment. The system processor is connected to each sensor and actuator via an internal bus. The computer program stored in the memory includes multiple modules such as system initialization, sensor control, data acquisition, data processing, decision planning, and control output. When the program starts, the processor first executes the initialization module to establish synchronization and calibration relationships between the sensors. During inspection operations, the processor periodically controls the sensors to synchronously collect environmental data according to program instructions.

[0046] The collected multimodal data is sent to the data processing module, where the processor calls a pre-stored deep learning model to fuse and identify the image data, generating candidate target information. Subsequently, the decision-making and planning module is activated. The processor uses all the logical methods defined by a multi-sensor-based inspection robot's automatic mapping of inspection points, including confidence-based filtering, quadrant coordinate mapping, and multimodal cross-validation, to ultimately determine the coordinates of all inspection points. The path planning submodule calculates the robot's trajectory based on these coordinate point sets and real-time perceived environmental obstacle information.

[0047] Finally, the control output module converts the planned path instructions into low-level drive signals, controlling the robot's chassis and gimbal to perform movement and scanning actions. Simultaneously, the processor continuously calls the SLAM processing thread during operation, using newly added sensor data to update the robot's pose estimation and environmental map, and dynamically corrects the stored inspection points and paths accordingly. This achieves a closed-loop autonomous inspection system integrating perception, decision-making, planning, execution, and online correction.

[0048] It should be further explained that, in the specific implementation process, a method for automatically mapping inspection points using a multi-sensor-based inspection robot involves simultaneously acquiring color images and thermal radiation images of the equipment to be inspected using a visible light camera and an infrared thermal imager mounted on the robot. The selected visible light camera has automatic white balance and wide dynamic range functions, enabling it to acquire clear images of the target's appearance under complex lighting conditions. The selected infrared thermal imager can acquire and store radiation data, accurately reflecting the temperature distribution of the target surface. During acquisition, hardware triggering or high-precision timestamps ensure that the two sets of images are strictly synchronized in time, and image registration is performed using pre-calibrated spatial transformation parameters to achieve spatial alignment, forming a spatiotemporally consistent multimodal data pair.

[0049] Subsequently, these image data are fed into a two-branch deep learning model for processing. One branch of the model specializes in processing color images, extracting appearance features such as texture and color; the other branch processes infrared images, extracting thermal distribution features. The features extracted by the two branches are integrated in a fusion layer, which uses an attention mechanism to dynamically weigh the importance of the two modalities, highlighting complementary information. The fused features are used to generate detection boxes for target devices and output the semantic category and confidence score for each target. Based on the detection results, the system further generates heatmaps reflecting the spatial distribution probability of the targets and semantic statistical maps for different categories.

[0050] For each identified candidate target, the system calculates pixel coordinates based on its detection bounding box and initiates a quadrant coordinate mapping process. This process establishes a spatial coordinate system with the camera's optical center as the origin and divides the field of view into four quadrants. Depending on the quadrant in which the target is located, the corresponding geometric mapping relationship is used to convert the two-dimensional pixel coordinates into three-dimensional world coordinates based on the robot. During this process, image edge distortion caused by the wide-angle lens is corrected, and for points located at the edge of the field of view or in blind spots, blind spot compensation strategies such as controlling robot movement or adjusting the gimbal angle are used to attempt to reacquire valid coordinates.

[0051] After obtaining the preliminary world coordinates, the system does not immediately adopt them as final indicators. Instead, it initiates a multimodal cross-validation process. This process spatially compares the preliminary coordinates with temperature anomaly areas identified through infrared image analysis. If the coordinates fall within the temperature anomaly area, they are directly confirmed as the final inspection point. If the coordinates do not fall within the anomaly area, but the target is identified as belonging to the category of easily heated equipment such as circuit breaker busbar connection points, the point is marked as a point to be verified and added to the confirmation queue for subsequent focused inspection. All other cases are discarded.

[0052] All verified final inspection points are aggregated into an ordered set and input into the path planning module. This module employs a model predictive control framework to generate a global inspection path using these points as path points, while satisfying the robot's kinematic constraints. During the robot's movement along the path, a local real-time obstacle avoidance thread continuously runs. It integrates real-time perception data from LiDAR, ultrasonic sensors, and other sources to detect sudden obstacles ahead and dynamically plan local detour trajectories, minimizing deviations from the global path while ensuring safety.

[0053] In the model predictive control framework, the robot's kinematic model is discretized into state-space equations. Kinematic constraints, such as the minimum turning radius ρ_min and the maximum velocity v_max, are transformed into linear constraints on control inputs, such as linear and angular velocities and their rates of change. In each control cycle, the optimizer, with the objective function of minimizing the deviation between the predicted trajectory and the global reference path over several future time steps, solves a quadratic programming problem under the conditions of the robot's dynamic model and the aforementioned linear constraints, obtaining a series of optimal control commands. After executing only the first control command, the next cycle begins with a re-prediction and optimization process, thus achieving rolling optimization and feedback correction.

[0054] While the robot performs its inspection tasks, the system continuously collects new sensor data and uses simultaneous localization and mapping (SMR) technology to constantly update its own pose and the environmental map. Based on the updated environmental information, the system dynamically corrects the mapped inspection points, such as updating the coordinates of moved equipment or adding inspection points for newly added equipment. This allows the entire inspection point map to continuously adapt to environmental changes, maintaining its accuracy and completeness.

[0055] The entire method achieves full autonomy of the inspection point process, from automatic identification and precise positioning to path planning and continuous optimization, through multimodal perception fusion, quadrant-based precise mapping, multi-level cross-validation, and closed-loop dynamic updates. This effectively improves the automation and reliability of equipment inspection in complex industrial environments.

[0056] The confidence threshold is not a fixed value; its setting depends on the performance evaluation results of the deep learning model on a validation dataset containing various typical scenarios. An initial confidence threshold is set for each category by statistically analyzing the recognition accuracy of different target categories under different lighting and occlusion conditions. In practical applications, the system allows users to fine-tune this threshold based on specific inspection task requirements and environmental conditions. For example, a higher threshold can be used for critical equipment with extremely high security requirements to minimize false alarms; in scenarios with higher inspection efficiency requirements, the threshold can be appropriately lowered to expand the detection range. Furthermore, the system records historical detection results. If a certain type of target consistently falls below the current threshold, a prompt will be triggered, allowing the user to assess whether the model or threshold needs adjustment.

[0057] By employing multimodal sensor fusion and deep learning recognition technology, automatic detection and precise coordinate mapping of inspection points are achieved, effectively reducing the reliance on manual pre-setting in traditional methods and improving the automation level and positioning accuracy of point mapping. Simultaneously, the integration of dynamic path planning and closed-loop environmental perception enhances the system's adaptability to complex industrial scenarios and improves inspection reliability.

[0058] Through quadrant coordinate transformation, multi-source data cross-validation, and continuous map update mechanisms, the system can autonomously cope with changes in equipment layout and environmental interference, improving the accuracy and practicality of inspection data.

[0059] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatic mapping of inspection points by a multi-sensor based inspection robot, characterized in that, The method comprises the following steps: S1: synchronously collecting RGB images and infrared images of the equipment to be inspected by a visible light camera and an infrared thermal imager carried by a robot; S2: fusing the RGB images and the infrared images by using a deep learning visual recognition model to identify target equipment and generate a target detection frame; S3: generating a spatial position thermal map and a semantic distribution map based on the target detection frame, and screening candidate inspection points according to a preset confidence threshold; S4: converting image pixel coordinates to a world coordinate system with the robot as the origin according to the spatial quadrant partition of the candidate inspection points in the camera field of view; S5: cross- verifying the obtained preliminary world coordinates with temperature abnormal regions based on infrared image analysis to determine the final inspection point coordinates; S6: integrating all the final inspection point coordinates to form an inspection point set, inputting a path planning algorithm and combining real-time environmental obstacle information to generate a final inspection path; The spatial quadrant partition is to establish a coordinate system with the optical center of the robot camera as the origin, with the optical axis direction, the horizontal direction and the vertical direction, and to divide the field of view into four quadrants; the coordinate mapping model establishes a mapping relationship of image pixel coordinates to world coordinates for different quadrants, and corrects image edge distortion; For the candidate inspection points located at the edge or blind area of the field of view, a blind area compensation algorithm is used in the coordinate conversion process, and the blind area compensation algorithm includes adjusting the robot pose or the camera viewing angle; The spatial position cross- verification includes judging whether the preliminary world coordinates are located in the temperature abnormal region, and if yes, determining the final inspection point; If not, recheck in combination with the semantic category, and retain the points belonging to the easy-to-heat equipment category.

2. The method for automatically drawing inspection points by a multi-sensor-based inspection robot according to claim 1, characterized in that: The path planning algorithm uses a model predictive control framework, the global path planning takes the final inspection point coordinates as path points and satisfies the kinematic constraints of the robot, and the local real-time obstacle avoidance adjusts the global path based on the environmental information sensed by the sensor. 3.The method for automatically drawing inspection points by a multi-sensor-based inspection robot according to claim 2, characterized in that: During the execution of the inspection path, the final inspection point coordinates are dynamically corrected and supplemented by continuously collecting sensor data and using simultaneous localization and mapping technology.

4. The method for automatically mapping inspection points by a multi-sensor-based inspection robot according to claim 1, wherein: The visible light camera is an industrial camera with automatic white balance and wide dynamic range function, and the infrared thermal imager is a non- cryogenic infrared thermal imager; when collecting images, ensure the spatio- temporal alignment of the RGB images and the infrared images.

5. The method for automatically mapping inspection points by a multi-sensor-based inspection robot according to claim 1, wherein: The deep learning visual recognition model is a double- branch network structure, which processes the RGB images and the infrared images respectively, and outputs the target detection result through a feature fusion layer.

6. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps of the method of any one of claims 1-5.

7. An inspection robot system comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1-5.

Images