Tower robot early warning inspection control system based on multiple sensors

Abstract

The present application relates to the technical field of inspection control, in particular to a tower robot early warning inspection control system based on multiple sensors, which comprises a multi-source sensor calibration unit, an environment dynamic compensation unit and a fusion early warning unit. The present application introduces a field reference calibration and environment dynamic compensation double feedforward correction link, so that the robot has real-time diagnosis and self-adaptive ability to the health state of its own sensing system and external interference environment, ensuring the reliability of the perception source; the fusion early warning unit as the brain realizes accurate judgment and active control, upgrades the robot from a passive data collector to an active intelligent diagnosis terminal on site, realizes accurate identification and quantitative evaluation of complex defects, on the one hand, significantly improves the automation level and detection accuracy of inspection, on the other hand, by giving the robot the ability to adapt to the environment and equipment state, provides a solid and reliable technical foundation for unmanned and intelligent operation and maintenance of power transmission lines.

Description

Technical Field

[0001] This invention relates to the field of inspection control technology, and more specifically, to a tower robot early warning inspection control system based on multiple sensors. Background Technology

[0002] With the rapid development of the national economy, my country has built the world's largest and highest-voltage power transmission and distribution network, as well as the most extensive mobile communication network. The number of power transmission towers and communication towers (collectively referred to as "towers," such as angle steel towers, steel pipe towers, and hybrid tower types) that form the core support of this network exceeds several million, and they are mostly distributed in complex outdoor environments. Ensuring the structural safety of these towers and the normal operation of their auxiliary equipment (such as insulators, fittings, and antennas) is a prerequisite for ensuring the stability of the power grid and uninterrupted communication. Among them, the tower inspection robot is an intelligent robot system used for automated inspection of high-altitude structures such as power transmission towers and communication towers. It typically has functions such as climbing, obstacle crossing, image acquisition, and sensor monitoring, and can replace or assist manual labor in completing high-altitude inspection tasks; In existing technologies, tower inspection robots typically employ single-modal or simple multimodal parallel detection methods for control and defect identification. However, for micro-cracks, early corrosion points, or slight overheating, the characteristic response of single-modal sensors is weak, resulting in a low signal-to-noise ratio. On the one hand, existing methods lack a systematic theoretical explanation of how these minute defects are collaboratively expressed and enhanced in multimodal data such as vision, infrared, and ultrasound, leading to a high false negative rate. On the other hand, due to the inherent differences in sampling time, spatial resolution, and physical dimensions (pixels, temperature, thickness) between visual, infrared, and ultrasonic data (spatiotemporal asynchrony, differences in feature dimensions), direct fusion can easily lead to information confusion or failure, failing to uncover the intrinsic correlation and complementarity between data. At the same time, the sensors themselves experience performance drift over time (e.g., lens damage, decreased sensitivity of infrared detectors), and the inspection site environment changes dynamically (e.g., slight tower swaying due to wind load, thermal image background interference caused by changes in sunlight). Existing technologies lack online calibration and dynamic environmental compensation mechanisms, resulting in a sharp decline in detection accuracy and poor robustness under non-ideal conditions. In view of this, a tower robot early warning inspection control system based on multiple sensors is proposed. Summary of the Invention

[0003] The purpose of this invention is to provide a tower robot early warning and inspection control system based on multiple sensors. Through an innovative progressive processing flow of online calibration, dynamic compensation and cross-fusion, the original, noisy and asynchronous multi-source data is transformed into comprehensive diagnostic information that can accurately characterize the specific defects of the tower. This systematically solves the problems of multi-source data fusion failure, difficulty in identifying minor defects and weak system adaptability.

[0004] To achieve the above objectives, the present invention provides a tower robot early warning and inspection control system based on multiple sensors, including a multi-source sensor calibration unit, an environmental dynamic compensation unit, and a fusion early warning unit; The sensor calibration unit is used to prioritize controlling the inspection robot to perform multi-source data acquisition tasks at preset track points of the calibration target group, and to obtain multi-source data standard values ​​including visual image data, infrared thermal imaging data and ultrasonic thickness measurement data. Control the current inspection robot to perform multi-source data acquisition tasks at the corresponding track points of the current actual iron tower, and simultaneously acquire real-time visual image data, real-time infrared thermal imaging data and real-time ultrasonic thickness measurement data; By comparing the standard values ​​of multi-source data with the real-time multi-source data, the deviation values ​​of each sensor are calculated, and the real-time multi-source data is corrected to obtain corrected multi-source data. The environmental dynamic compensation unit is used to control the inspection robot to collect the actual boundary condition data of the current tower, and input the actual boundary condition data into the preset optimization model to compensate and optimize the multi-source data to obtain optimized multi-source data. The fusion early warning unit is used for spatiotemporal registration optimization of multi-source data. It performs consistency verification of multi-source data through cross-validation rules and performs deep feature extraction and fusion through a multimodal feature fusion network to generate comprehensive inspection data that can fully reflect the current status of the tower. If the comparison of the comprehensive inspection data exceeds the safety threshold, an early warning signal is generated and the robot is controlled to execute a safety strategy.

[0005] As a further improvement to this technical solution, the inspection robot includes a walking mechanism that climbs along the iron tower track, a visual sensor for acquiring visual image data, an infrared sensor for acquiring infrared thermal imaging data, and an ultrasonic sensor for acquiring ultrasonic thickness measurement data, wherein: A gimbal that supports three degrees of freedom adjustment (pitch, yaw, and roll) is provided between the visual sensor and the walking mechanism.

[0006] As a further improvement to this technical solution, the inspection robot also includes a positioning and synchronization module. The positioning and synchronization module is used to control the gimbal to call the corresponding angle parameters to adjust the rotation according to the preset inspection task, so as to align the vision sensor with the part to be inspected. When the infrared sensor detects an abnormal heat point, it prompts the pan-tilt unit under the vision sensor to automatically turn to the heat point to take pictures and confirm the crack or dirt. If the infrared sensor detects a temperature rise at a connection point, it activates the ultrasonic sensor to collect ultrasonic thickness measurement data.

[0007] As a further improvement to this technical solution, the calibration target set is a portable calibration target set with known geometric characteristics and stable physical properties. The calibration target set includes a high-contrast patterned plate for visual calibration, a blackbody / emissivity reference plate for infrared calibration, and a standard test block of known thickness for ultrasonic calibration.

[0008] As a further improvement to this technical solution, the sensor calibration unit includes a standard data acquisition module, an actual data acquisition module, and a deviation correction module; The standard data acquisition module is used to control the robot to move along a preset track to multiple calibration points on the calibration target group based on the inspection method of the positioning synchronization module, perform synchronous multi-source data acquisition, and output multi-source data standard values, wherein: All calibration points are located in standard areas on undamaged, clean surfaces with known physical properties and calibrated as reference values. The actual data acquisition module is used to deploy the same robot as in the standard data acquisition module to the geometric position of the calibration point corresponding to the actual iron tower, and to collect real-time multi-source data based on the inspection method of the positioning synchronization module, and output actual multi-source data. The deviation correction module is used to define the deviation functions of each sensor, including visual brightness offset, infrared zero-point drift, and ultrasonic thickness measurement system error. The deviation functions are used to establish a correction mapping function to model and inversely compensate for the unified deviation of the three heterogeneous sensors, thereby obtaining corrected multi-source data.

[0009] As a further improvement to this technical solution, the boundary condition data includes mechanical vibration, thermal environment interference, contact instability, and attitude deviation.

[0010] As a further improvement to this technical solution, the environmental dynamic compensation unit includes an environmental status acquisition module and an optimized multi-source data storage module; The environmental status acquisition module is used to configure the following auxiliary sensors on the robot for acquiring environmental status: An IMU inertial measurement sensor for monitoring the vibration frequency and amplitude of the body; a contact force sensor mounted on an ultrasonic probe bracket for real-time feedback of applied pressure; a light intensity sensor for estimating the solar altitude angle and radiation intensity; and a wind speed and direction sensor. An environment vector is constructed after collecting the environmental status. The optimized multi-source data storage module is used to input environmental vectors into the optimization model to compensate and optimize the multi-source data, thereby obtaining optimized multi-source data.

[0011] As a further improvement to this technical solution, the optimization model includes a visual image deblurring compensation model, an infrared thermometry dynamic compensation model, and an ultrasonic thickness measurement compensation model, wherein: The visual image deblurring compensation model utilizes the robot's IMU to collect three-dimensional acceleration and angular velocity data in real time. After subtracting the influence of gravity from the acceleration data, time integration is performed to obtain the displacement trajectory. Simultaneously, the rotation component is calculated by combining the angular velocity data. Fast Fourier Transform is used to analyze the motion spectrum, automatically identifying low-frequency structural oscillations and high-frequency mechanical vibrations. Based on the motion trajectory within the exposure time period, a point spread matrix describing the image blur direction is dynamically generated. Principal component analysis is used to compress this matrix into a 64-dimensional feature vector. A U-Net++ structure containing symmetrical encoder and decoder paths is adopted, embedding motion attention gate modules on each cross-layer connection line between the encoder and decoder, wherein: Using the image feature map and 64-dimensional feature vector of the current layer as input, the feature vector is multiplied by the visual features to establish a visual image deblurring compensation model. During the decoding process, motion condition information of different scales is gradually injected to achieve progressive processing from global blur correction to local detail restoration. The infrared temperature measurement dynamic compensation model constructs the thermal balance equation of the tower surface, dynamically calculates the background field of environmental thermal interference, and uses an iterative compensation algorithm to remove the influence of solar radiation and convective heat dissipation from the measured radiation value. Finally, it extracts the real temperature rise signal that only reflects the abnormal thermal state of the component itself and outputs optimized infrared thermal imaging data. The ultrasonic thickness measurement compensation model establishes a quantitative mapping relationship between probe contact pressure, tilt angle, ambient temperature, and measurement error. During real-time inspection, it synchronously collects environmental vectors, dynamically calculates triple compensation factors to correct ultrasonic thickness measurement data, and combines confidence assessment to trigger an automatic retest mechanism to output optimized ultrasonic thickness measurement data.

[0012] As a further improvement to this technical solution, the fusion early warning unit includes a spatiotemporal registration module, a cross-validation module, a multimodal feature fusion module, and an early warning control module; The spatiotemporal registration module uses the inspection point number and gimbal angle provided by the robot positioning and synchronization module as indexes to establish the association of different sensor data. For the same inspection point, the optimized visual image data and optimized infrared thermal imaging data are aligned at the two-dimensional image level using feature-based image registration algorithms. The optimized ultrasonic thickness measurement data is mapped to the approximate area of ​​the corresponding visual / infrared image based on the three-dimensional coordinates of the probe contact. The cross-validation module is used to build a multimodal joint criterion rule library, and only when multiple physical dimensions are co-existing abnormalities are they confirmed as real defects; The multimodal feature fusion module is used to process optimized visual image data, optimized infrared thermal imaging data, and optimized ultrasonic thickness measurement data through independent feature extraction sub-networks. The optimized ultrasonic thickness measurement data is converted into a one-dimensional feature vector. A fusion layer based on an attention mechanism is used to calculate the cross-modal attention weight between visual and infrared features. The features are then weighted and fused using this weight. The fused features are concatenated with the ultrasonic feature vector and input into a fully connected classification / regression network. The output is comprehensive inspection data that records the defect type and severity. The early warning control module is used to convert comprehensive inspection data into a comprehensive health score representing the overall risk level of the current inspection point. If the comprehensive health score is greater than the safety threshold, an early warning signal for a high-risk state is output, and the pan-tilt unit is controlled to take pictures from multiple angles, repeatedly collect data, and perform secondary verification. If the early warning signal is continuously issued, the walking mechanism sends a braking command.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This multi-sensor-based tower robot early warning and inspection control system constructs a self-evolving closed loop of "perception-diagnosis-decision-control," introducing on-site reference calibration and dynamic environmental compensation correction. This enables the robot to have real-time diagnostic and adaptive capabilities for the health status of its own sensing system and external interference environments, forming an intelligent closed loop with self-perception, self-correction, and autonomous decision-making capabilities, ensuring the reliability of the perception source. Furthermore, it performs environmental filtering to ensure the purity of input information. The integrated early warning unit acts as the brain, achieving accurate judgment and proactive control, upgrading the robot from a passive data collector to an active on-site intelligent diagnostic terminal. This enables accurate identification and quantitative assessment of complex defects. On the one hand, it significantly improves the level of inspection automation and detection accuracy; on the other hand, by endowing the robot with the ability to adapt to the environment and equipment status, it provides a solid and reliable technical foundation for the unmanned and intelligent operation and maintenance of power transmission lines. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating the overall structural principle of the present invention; Figure 2 A detailed block diagram illustrating the overall structural principle of this invention.

[0015] The meanings of the labels in the diagram are as follows: 100. Multi-source sensor calibration unit; 200. Environmental dynamic compensation unit; 300. Fusion early warning unit. Detailed Implementation

[0016] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 Please see Figures 1-2 As shown, this embodiment provides a tower robot early warning and inspection control system based on multiple sensors, including a multi-source sensor calibration unit 100, an environmental dynamic compensation unit 200, and a fusion early warning unit 300. Step 1: The sensor calibration unit 100 is used to prioritize controlling the inspection robot to perform multi-source data acquisition tasks at the preset track points of the calibration target group, and to obtain multi-source data standard values ​​including visual image data, infrared thermal imaging data and ultrasonic thickness measurement data. Control the current inspection robot to perform multi-source data acquisition tasks at the corresponding track points of the current actual iron tower, and simultaneously acquire real-time visual image data, real-time infrared thermal imaging data and real-time ultrasonic thickness measurement data; By comparing the standard values ​​of multi-source data with the real-time multi-source data, the deviation values ​​of each sensor are calculated, and the real-time multi-source data is corrected to obtain corrected multi-source data. Therefore, by performing a rapid calibration process on a known and stable calibration target group before the inspection task begins, the system deviations of the vision, infrared, and ultrasonic sensors currently equipped on the inspection robot due to aging, contamination, or initial calibration drift can be quantified. In actual inspection, these deviation values ​​are used to reverse correct the raw data collected in real time to obtain corrected multi-source data, which is equivalent to equipping the robot with a "field reference value comparison template" to ensure the accuracy of the data source. Note: This calibration process is mainly used to discover and quantify significant sensor drift or malfunctions (such as severe lens contamination or infrared detector failure) and to provide preliminary linear compensation based on reference values ​​for the collected data. For nonlinear errors that are strongly correlated with the environment, comprehensive processing is required in conjunction with the subsequent environmental dynamic compensation unit. The inspection robot includes a walking mechanism that climbs along the tower track (composed of rollers driven by servo motors, allowing it to move along the track), a visual sensor for acquiring visual image data (typically a high-resolution industrial camera or visible light camera with autofocus and low-light enhancement capabilities), an infrared sensor for acquiring infrared thermal imaging data (specifically a non-contact infrared sensor that measures surface temperature distribution based on the infrared radiation emitted by the object itself, enabling the detection of overheating defects in connection parts such as power line joints, clamps, and surge arresters; localized heating caused by uneven voltage distribution due to contamination or damage to insulator strings; and abnormal heat conduction in tower metal components due to corrosion or cracks), and an ultrasonic sensor for acquiring ultrasonic thickness measurement data (utilizing the time difference of high-frequency sound waves propagating within the material to measure material thickness, particularly suitable for monitoring the wall thickness of metal structures; a non-destructive testing technology, used for assessing the degree of corrosion of the main tower materials and flange connections, analyzing the long-term degradation trend of angle steel and steel pipe walls, and detecting internal delamination or porosity caused by manufacturing defects or fatigue). A gimbal that supports three degrees of freedom of pitch, yaw and roll is set between the vision sensor and the walking mechanism. By adjusting the angle, a single camera can cover multiple facades and blind spots, avoiding blind spots and facilitating multi-angle comparison of the same component (such as observing missing bolts from the side), thereby improving the defect detection rate.

[0018] In addition, the inspection robot also includes a positioning and synchronization module. The positioning and synchronization module is used to control the gimbal to call the corresponding angle parameters to adjust the rotation according to the preset inspection tasks (such as insulator strings, bolt connection points, rusted areas, etc.), so that the vision sensor is aimed at the part to be inspected and high-definition images are captured at the best angle to ensure that key features are clearly identifiable. When the infrared sensor detects an abnormal heat point, it prompts the pan-tilt unit under the vision sensor to automatically turn to the heat point to take pictures and confirm the crack or dirt. If the infrared sensor detects a temperature rise at a connection point, ΔT>10℃ and remains stable, the ultrasonic sensor is activated to collect ultrasonic thickness measurement data.

[0019] The calibration target set is a portable calibration target set with known geometric characteristics and stable physical properties. The calibration target set includes a high-contrast patterned board for visual calibration, a blackbody / emissivity reference board for infrared calibration, and a standard test block of known thickness for ultrasonic calibration. A small-scale real calibration target set with physical stability, controllable parameters, and repeatable measurements is established, and several calibration points are set on its surface. The calibration points correspond to the key structural parts of the actual iron tower (such as bolt connections, welds, angle steel corners, etc.). It can be manufactured at a 1:5 or 1:10 scale according to the actual iron tower type, with consistent materials (such as Q345 steel) and surface treatment processes (galvanizing / rust simulation). This facilitates the subsequent quantification of the systematic deviation of the vision, infrared, and ultrasonic sensors carried by the current inspection robot from the ideal state, and realizes subsequent data correction.

[0020] It is worth noting that the sensor calibration unit 100 includes a standard data acquisition module, an actual data acquisition module, and a deviation correction module; The standard data acquisition module controls the robot to move along a preset track to multiple calibration points on the calibration target group based on the inspection method of the positioning synchronization module. It performs synchronous multi-source data acquisition, outputting multi-source data standard values. The track covers typical structural parts (angle steel connections, bolt joints, weld areas, etc.). Each calibration point has unique spatial coordinates and physical attribute labels (e.g., surface roughness Ra=3.2μm, emissivity=0.88, true wall thickness t0=8.0mm). Through the encoder + UWB + IMU fusion positioning system built into the positioning synchronization module, the robot is guided to accurately move to the first calibration point. A vision sensor captures high-definition images, an infrared sensor records temperature distribution maps, and an ultrasonic sensor emits pulses and receives echoes to calculate the thickness value. This completes the data acquisition for all calibration points, forming a set of multi-source data standard values, including: All calibration points are located in standard areas on undamaged, clean surfaces with known physical properties and calibrated as reference values. The actual data acquisition module is used to deploy the same robot as the standard data acquisition module to the corresponding calibration point of the actual iron tower. It uses digital twin registration or SLAM relocalization technology to ensure consistency and reproduce the acquisition under the same posture, distance and angle conditions. It collects real-time multi-source data based on the inspection method of the positioning synchronization module and outputs actual multi-source data, including actual visual image data, actual infrared thermal imaging data and actual ultrasonic thickness measurement data. The deviation correction module defines the deviation functions for each sensor, including visual brightness offset, infrared zero-point drift, and ultrasonic thickness measurement system error. It uses these deviation functions to establish a correction mapping function, modeling and compensating for the unified deviations of the three heterogeneous sensors. This yields corrected multi-source data, including corrected visual image data, corrected infrared thermal imaging data, and corrected ultrasonic thickness measurement data. This avoids the problem that traditional offline calibration cannot reflect real-time aging. Specifically: Define the visual brightness offset function: For a visual sensor, by comparing the image coordinates of feature points on the calibration target with the known world coordinates, calculate the homography transformation matrix, which is used to correct the geometric distortion caused by the slight offset of the camera installation, and output the deviation parameter. Define the infrared zero-point drift function: For an infrared sensor, by reading the difference between the measured temperature of the blackbody reference plate and its true temperature, an additive offset is calculated for temperature compensation, and the deviation parameter is output. Define the error function of the ultrasonic thickness measurement system: use a standard test block (embedded reference layer) of known thickness for rapid on-site calibration, establish an error response surface, and output deviation parameters; Then, all deviation parameters are periodically updated (e.g., every 5 minutes) through edge computing units (deployed on the robot body) to form a continuous time series, construct a unified deviation state vector, and design a family of differentiable and invertible compensation functions to make visual channel correction, infrared temperature field correction, and ultrasonic thickness correction possible. Considering that the responses of the three sensors are related under the same physical event (e.g., corrosion → visual dimming, abnormal infrared thermal conductivity, enhanced ultrasonic scattering), a multimodal graph structure can be constructed, where nodes represent sensor types and edges represent coupling relationships. A deviation propagation model is constructed by using GNN to learn from the deviation of one sensor to infer the potential deviations of other sensors. If a significant decrease in visual brightness and blurring of texture are found, lens contamination is inferred → which can help determine whether the infrared window is also blocked → providing early warning of infrared drift risk. Upon completion of this stage, it is equivalent to completing the "factory-level recalibration" of all sensors used by the robot during this mission, greatly improving the reliability of subsequent test data.

[0021] Even if the sensor itself is accurate, the data it collects in complex outdoor environments can be distorted by the slight swaying of the tower caused by wind loads, the vibration of the robot moving along the guide rail, and the change in the contact force between the detection probe and the tower surface. Therefore, in step two, the environmental dynamic compensation unit 200 is used to control the inspection robot to collect the actual boundary condition data of the current tower and input the actual boundary condition data into the preset optimization model to compensate and optimize the multi-source data, thereby obtaining optimized multi-source data. This can simulate and reverse the influence of boundary conditions on various types of data, such as compensating for image blurring caused by jitter, temperature measurement errors caused by changes in contact thermal resistance, and ultrasonic thickness measurement deviations caused by angle shifts. The output is "optimized multi-source data" that excludes the main environmental dynamic interference and reflects the true state of the tower itself. Boundary condition data include mechanical vibration (wind load causes slight swaying of the tower (amplitude can reach several centimeters), shaking of the robot walking mechanism → resulting in blurred images and increased ultrasonic signal noise), thermal environment interference (uneven sunlight causes background temperature difference, air convection affects infrared radiation measurement), contact instability (ultrasonic probe pressure fluctuations due to track bumps, poor coupling → increased thickness measurement error) and attitude deviation (gimbal angle offset causes the detection direction to deviate from vertical → causing oblique refraction effect).

[0022] To establish an environmental dynamic compensation mechanism based on a combination of physical modeling and data-driven approaches, the environmental dynamic compensation unit 200 includes an environmental status acquisition module and an optimized multi-source data storage module. The environmental status acquisition module is used to configure the following auxiliary sensors on the robot to collect environmental status data: An IMU inertial measurement sensor (accelerometer + gyroscope) is used to monitor the vibration frequency and amplitude of the machine body; a contact force sensor mounted on an ultrasonic probe bracket to provide real-time feedback of applied pressure; a light intensity sensor to estimate the solar altitude angle and radiation intensity; and a wind speed and direction sensor. An environment vector is constructed after collecting the environmental status. The multi-source data storage optimization module is used to input environmental vectors into the optimization model to compensate and optimize multi-source data, thereby obtaining optimized multi-source data, including optimized visual image data, optimized infrared thermal imaging data, and optimized ultrasonic thickness measurement data.

[0023] The optimization models include a visual image deblurring compensation model, an infrared thermometry dynamic compensation model, and an ultrasonic thickness measurement compensation model, among which: When the IMU detects high-frequency vibration (>5Hz) or large-amplitude displacement, relative displacement occurs during camera exposure, causing the image to blur along a specific direction. The visual image deblurring compensation model utilizes the robot's onboard inertial measurement unit (IMU) to collect three-dimensional acceleration and angular velocity data in real time. Note that the built-in gyroscope drift compensation algorithm eliminates sensor cumulative errors, ensuring the accuracy of motion measurement. After subtracting the influence of gravity from the acceleration data, time integration is performed to obtain the displacement trajectory. Simultaneously, the rotation component is calculated by combining the angular velocity data. The motion spectrum is analyzed using Fast Fourier Transform (FFT) to automatically identify low-frequency structural swaying (such as the slow swaying of a tower under wind) and high-frequency mechanical vibration (such as the instantaneous shaking of a robot walking). Based on the motion trajectory within the exposure time period, a point spread matrix describing the blur direction of the image is dynamically generated. Since the PSF matrix generated by IMU data is theoretically very large (e.g., 32×32=1024 dimensions), but the motion mode of the tower inspection robot is highly limited (track movement, limited vibration frequency range, mechanical amplitude limitation), the effective mathematical dimension describing the motion mode of the tower inspection robot is far less than 1024. This provides a physical justification for significant dimensionality reduction and forms the basis for all subsequent processing. Principal component analysis (PCA) is used to compress the matrix into a 64-dimensional feature vector, which serves as the motion condition input for the deblurring network. (Specifically, PCA is performed on a large number of measured PSF matrices to verify that the 64-dimensional feature vector can perform PCA on a large number of measured PSF matrices. 32 dimensions lead to information loss and underfitting, while 128 dimensions offer negligible performance improvement but significantly increase computational burden and overfitting risk. Therefore, 64 dimensions represent the optimal balance between information preservation and model efficiency.) A U-Net++ structure with symmetrical encoder and decoder paths is adopted. Motion attention gate modules are embedded in each cross-layer connection between the encoder and decoder. The encoder extracts features through multi-layer convolution and downsampling, while the decoder recovers a clear image through upsampling and feature fusion. (The number of feature channels in the encoder and decoder is typically 512; 64 dimensions are 1 / 8 of 512, facilitating the splicing and fusion of channel attention between the 64-dimensional feature vector and the image feature map. Furthermore, 64 dimensions are divisible by the typical number of attention heads, 8, facilitating uniform partitioning in multi-head attention mechanisms.) The motion attention gate module takes the current layer's image feature map and 64-dimensional feature vector as input. After the 64-dimensional feature vector is transformed through two fully connected network layers, it is fused with the image feature map via channel concatenation (where the two-dimensional spatial grid of the image feature map is compressed into a one-dimensional sequence, and the feature vector is projected onto the semantic space of the image feature map for concatenation). A spatial attention weight map is generated through 1x1 convolution (identifying which regions in the image are most blurred due to the current encoded motion mode with pixel-level precision). This can identify blurred areas that need to be focused on for repair with pixel-level precision, such as edge blurring caused by high-frequency vibration. In the feature space of the deepest layer of the network (lowest resolution, most global semantic information), the feature vector is multiplied with visual features by matrix multiplication (such as using affine transformation or matrix multiplication to interact the 64-dimensional motion feature vector with the image feature map to achieve global correction of the overall blur direction and intensity of the image). A visual image deblurring compensation model is established. During the decoding process, motion condition information of different scales is gradually injected to achieve progressive processing from global blur correction to local detail restoration. Specifically, during training: First, a high-definition image library of the tower surface was collected. The measured vibration data of the IMU was converted into corresponding blur kernels through physical simulation. Labeled blur-sharp image pairs were generated in batches. Extreme working conditions such as strong wind (level 8 wind) and rapid turning were simulated to ensure the robustness of the model. Pixel-level L1 loss, structural similarity (SSIM) loss and edge enhancement loss were used in combination. L1 loss ensured the accuracy of the overall tone, SSIM loss maintained the structural integrity, and edge loss focused on optimizing the sharpness of defect areas such as cracks and corrosion. Frequency domain constraints were introduced in the later stage of training to force the network to prioritize the repair of high-frequency details, which solved the problem of traditional methods failing in complex vibration environments. The infrared thermometry dynamic compensation model constructs a thermal balance equation for the tower surface, dynamically calculates the background field of environmental thermal interference, and uses an iterative compensation algorithm to remove the influence of solar radiation and convective heat dissipation from the measured radiation values. Ultimately, it extracts the true temperature rise signal that only reflects the abnormal thermal state of the component itself, outputting optimized infrared thermal imaging data. This achieves a leap from qualitative observation to quantitative analysis of infrared detection in complex outdoor environments. Specifically, it utilizes environmental vectors (real-time acquisition of direct solar radiation intensity, ambient air temperature, wind speed, and wind direction by light intensity sensors and wind speed and direction sensors; calculation of the orientation angle of each infrared image pixel on the tower surface relative to the sun at this moment to determine solar radiation absorption) to establish the thermal balance equation for the tower surface. The core of the tower surface thermal balance equation is a transient thermodynamic control equation based on the principle of energy conservation, as shown in the following formula: The change in thermal energy per unit time and unit volume (transient term) = Input (absorption term) - Output (heat dissipation term) + Internal term; The absorption term is direct and diffuse solar radiation (solar radiation absorption + defect heat generation), the forced convection heat dissipation and thermal radiation (effective wind speed convection heat dissipation + radiation heat dissipation), and the internal term is abnormal heat generation caused by defects (internal heat conduction of the structure). The influence of wind speed is calculated using an engineering convection heat transfer formula, and the radiative heat transfer between surfaces is calculated by introducing a viewing angle factor. The theoretical temperature field under defect-free conditions is solved by numerical iteration, and the theoretical background temperature field is calculated based on environmental parameters. The initial temperature is inverted from the measured radiation value, and the influence of environmental reflection is corrected by iterative correction of the radiative transfer equation. The difference between the compensated real temperature and the background temperature is output, which is to optimize the infrared thermal imaging data. The ultrasonic thickness measurement compensation model establishes a quantitative mapping relationship between probe contact pressure, tilt angle, ambient temperature, and measurement error. During real-time inspection, it synchronously collects environmental vectors (six-dimensional force, attitude, and temperature data), dynamically calculates a triple compensation factor to correct the ultrasonic thickness measurement data, and combines confidence assessment to trigger an automatic retest mechanism, outputting optimized ultrasonic thickness measurement data. Ultimately, this improves the thickness measurement accuracy under moving conditions, achieving stable and reliable measurement under complex conditions such as vibration, tilt, and temperature changes. Specifically: To address the issues of insufficient pressure leading to poor coupling and excessive pressure causing compression of materials or probe deformation, different normal forces were applied to a standard test block, and the signal-to-noise ratio (SNR) and time-of-flight (ToF) offset of the echo signal were recorded. An empirical function was then established to compensate for the probe contact pressure. To address the issues of beam refraction, increased sound path, and echo splitting caused by tilting, an IMU is used to obtain the probe's attitude angle relative to the measured surface, and material sound velocity correction is combined to compensate for the probe's tilt angle. To address the impact of temperature changes on material sound velocity and probe delay linear expansion, the material sound velocity versus temperature curve was calibrated. The thermal expansion of the probe delay layer was independently calibrated to introduce additional delay for environmental temperature compensation, and the final thickness was calculated.

[0024] Step 3: The fusion early warning unit 300 is used for spatiotemporal registration and optimization of multi-source data to ensure that all data points to the same physical location. The consistency of multi-source data is verified through cross-validation rules. For example, a rust spot discovered visually must be accompanied by local temperature anomalies on infrared (rust may cause increased contact resistance) or thickness reduction on ultrasonic measurements to be confirmed as a true defect. This greatly reduces false alarms. Deep feature extraction and fusion are performed through a multimodal feature fusion network to generate comprehensive inspection data that can fully reflect the current status of the tower. If the comparison of the comprehensive inspection data exceeds the safety threshold, an early warning signal is generated and the robot is controlled to execute safety strategies to achieve accurate early warning and autonomous control (such as stopping forward and focusing on re-inspection).

[0025] The fusion early warning unit 300 includes a spatiotemporal registration module, a cross-validation module, a multimodal feature fusion module, and an early warning control module; Because visual cameras, infrared thermal imagers, and ultrasonic sensors have different acquisition frequencies (e.g., 30Hz, 15Hz, 5Hz) and different hardware processing delays, direct fusion would lead to spatiotemporal misalignment. Therefore, the spatiotemporal registration module uses the inspection point number and gimbal angle provided by the robot positioning synchronization module as an index to establish the association between different sensor data. For the same inspection point, the optimized visual image data and optimized infrared thermal imaging data are aligned at the two-dimensional image level using feature-based image registration algorithms. The optimized ultrasonic thickness measurement data is mapped to the approximate area of ​​the corresponding visual / infrared image based on the three-dimensional coordinates of its probe contact (calculated from the robot pose and gimbal angle). The cross-validation module is used to build a multimodal joint criterion rule base. Only when multiple physical dimensions are co-existing anomalies are met is it confirmed as a real defect. Among them, the multimodal joint criterion rule base establishes a criterion system of "necessary conditions + sufficient conditions". Taking corrosion defects as an example: Visually necessary conditions: abnormal roughening of surface texture (grayscale co-occurrence matrix contrast change >30%), color shift towards reddish-brown (HSV space hue change >15°); Infrared spectral density requirement: The corrosion layer acts as a thermal resistance layer, leading to abnormal heat conduction (local temperature gradient > 2℃ / cm). Ultrasonic testing is sufficient if corrosion causes material loss (thickness reduction > 0.3 mm, or a thinning rate > 5% compared to historical data at the same location). Judgment logic: Only when the necessary visual condition is met and at least one other modality sufficient condition is met can a rust defect be identified as a real rust defect. As for the other joint judgment rule base, it can be customized according to the actual situation and output a confidence score. The weight coefficients are obtained by training a large number of known defect samples, reflecting the ability of different sensors to identify various types of defects. It can be iteratively optimized based on on-site feedback. For loose bolts: Visual necessary condition: significant change in the visual position of the bolt head or nut; Infrared sufficient condition: abnormal temperature rise at the connection (ΔT > 15℃); Ultrasonic sufficient condition: (not applicable). Criterion logic: if both the visual necessary condition and the infrared sufficient condition are met, then it is confirmed as a high risk of loose bolts. The multimodal feature fusion module is used to extract optimized visual image data, optimized infrared thermal imaging data, and optimized ultrasonic thickness measurement data through independent feature extraction sub-networks. The optimized ultrasonic thickness measurement data is converted into a one-dimensional feature vector. A fusion layer based on an attention mechanism is used to calculate the cross-modal attention weight between visual and infrared features. The features are then weighted and fused using this weight. The fused features are concatenated with the ultrasonic feature vector and input into a fully connected classification / regression network. The output is comprehensive inspection data that records the defect type and severity. The early warning control module is used to convert comprehensive inspection data into a comprehensive health score representing the overall risk level of the current inspection point. If the comprehensive health score is greater than the safety threshold, an early warning signal for a high-risk state is output, and the pan-tilt unit is controlled to take pictures from multiple angles, repeatedly collect data, and perform secondary verification. If the early warning signal is continuously issued, the walking mechanism sends a braking command to prevent further approach to the dangerous area.

[0026] In summary: This invention addresses the issue of online sensor health management and data traceability mechanisms for complex field conditions. It solves the problem that the performance of inspection robot sensors inevitably degrades during long-term operation in harsh environments, but traditional methods lack convenient and quantitative on-site assessment and correction methods, leading to questionable data reliability. This invention designs a portable calibration target set and a rapid on-site calibration process, constructing a mobile reference system. By performing rapid measurements on targets with known true values, the instantaneous system deviation of the sensor within the current task cycle is directly quantified, and all subsequent inspection data is corrected accordingly. This is equivalent to equipping the robot with a traveling instrument calibrator, achieving proactive assurance of data accuracy and full traceability. This invention addresses the problem of severely distorted measurement data when using sophisticated laboratory sensors directly in outdoor environments with vibration, temperature changes, and wind disturbances, based on a dynamic decoupling and compensation model for real-time multi-physics sensing of environmental interference. Existing compensation methods are often static and single-factor-based, failing to handle complex coupled interference. This invention models the environmental interference itself as system input, deploying auxiliary sensors to collect environmental vectors in real time. Subsequently, specialized physics-data hybrid compensation models are constructed for different sensing principles (e.g., image deblurring based on IMU motion trajectory, infrared background field stripping based on thermal balance equations, and ultrasonic error correction based on pressure-angle-temperature mapping). This achieves targeted and refined stripping of complex environmental interference, outputting data stripped of environmental noise that reflects the intrinsic state of the components, thus solving the core problem of inaccurate outdoor testing data. Based on a multimodal defect collaborative decision-making system that integrates rigorous physical logic and deep learning, this invention addresses the common problem that multi-sensor information fusion often relies on data stacking or simple weighting, lacking physical constraints and prone to misjudgments that violate physical laws. Furthermore, rule-based expert systems are not intelligent enough to handle unknown defects. This invention establishes a multimodal joint criterion rule base, rooted in the physical mechanisms of defect occurrence (e.g., corrosion always precedes morphological changes and may be accompanied by thermal resistance or thickness loss), performing hard filtering to significantly reduce false alarms. Data that passes physical verification is then fused to achieve more refined classification and quantification, combining high reliability with strong adaptability.

[0027] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A tower robot early warning and inspection control system based on multiple sensors, characterized in that, It includes a multi-source sensor calibration unit (100), an environmental dynamic compensation unit (200), and a fusion early warning unit (300). The sensor calibration unit (100) is used to control the inspection robot to perform multi-source data acquisition tasks on the preset track points of the calibration target group, and to obtain multi-source data standard values ​​including visual image data, infrared thermal imaging data and ultrasonic thickness measurement data. Control the current inspection robot to perform multi-source data acquisition tasks at the corresponding track points of the current actual iron tower, and simultaneously acquire real-time visual image data, real-time infrared thermal imaging data and real-time ultrasonic thickness measurement data; By comparing the standard values ​​of multi-source data with the real-time multi-source data, the deviation values ​​of each sensor are calculated, and the real-time multi-source data is corrected to obtain corrected multi-source data. The environmental dynamic compensation unit (200) is used to control the inspection robot to collect the actual boundary condition data of the current tower, and input the actual boundary condition data into the preset optimization model to compensate and optimize the multi-source data to obtain optimized multi-source data; The fusion early warning unit (300) is used for spatiotemporal registration optimization of multi-source data. It performs consistency verification of multi-source data through cross-validation rules and performs deep feature extraction and fusion through a multimodal feature fusion network to generate comprehensive inspection data that can fully reflect the current status of the tower. If the comparison of the comprehensive inspection data exceeds the safety threshold, an early warning signal is generated and the robot is controlled to execute a safety strategy.

2. The tower robot early warning and inspection control system based on multiple sensors according to claim 1, characterized in that: The inspection robot includes a walking mechanism that climbs along the iron tower track, a visual sensor that collects visual image data, an infrared sensor that collects infrared thermal imaging data, and an ultrasonic sensor that collects ultrasonic thickness measurement data, wherein: A gimbal that supports three degrees of freedom adjustment (pitch, yaw, and roll) is provided between the visual sensor and the walking mechanism.

3. The tower robot early warning and inspection control system based on multiple sensors according to claim 2, characterized in that: The inspection robot also includes a positioning and synchronization module, which is used to control the gimbal to call the corresponding angle parameters to adjust the rotation according to the preset inspection task, so as to align the vision sensor with the part to be inspected. When the infrared sensor detects an abnormal heat point, it prompts the pan-tilt unit under the vision sensor to automatically turn to the heat point to take pictures and confirm the crack or dirt. If the infrared sensor detects a temperature rise at a connection point, it activates the ultrasonic sensor to collect ultrasonic thickness measurement data.

4. The tower robot early warning and inspection control system based on multiple sensors according to claim 3, characterized in that: The calibration target set is a calibration target set with known geometric characteristics and stable physical properties. The calibration target set includes a high-contrast patterned plate for visual calibration, a blackbody / reference plate with known emissivity for infrared calibration, and a standard test block with known thickness for ultrasonic calibration.

5. The tower robot early warning and inspection control system based on multiple sensors according to claim 3, characterized in that: The sensor calibration unit (100) includes a standard data acquisition module, an actual data acquisition module, and a deviation correction module; The standard data acquisition module is used to control the robot to move along a preset track to multiple calibration points on the calibration target group based on the inspection method of the positioning synchronization module, perform synchronous multi-source data acquisition, and output multi-source data standard values, wherein: All calibration points are located in standard areas on undamaged, clean surfaces with known true physical properties and are calibrated as reference values. The actual data acquisition module is used to deploy a robot identical to the standard data acquisition module to the geometric position of the calibration point corresponding to the actual iron tower, and to collect real-time multi-source data based on the inspection method of the positioning synchronization module, and output actual multi-source data. The deviation correction module is used to define the deviation functions of each sensor, including visual brightness offset, infrared zero-point drift, and ultrasonic thickness measurement system error. The deviation functions are used to establish a correction mapping function to model and inversely compensate for the unified deviation of the three heterogeneous sensors, thereby obtaining corrected multi-source data.

6. The tower robot early warning and inspection control system based on multiple sensors according to claim 5, characterized in that: The boundary condition data includes mechanical vibration, thermal environmental disturbance, contact instability, and attitude deviation.

7. The tower robot early warning and inspection control system based on multiple sensors according to claim 6, characterized in that: The environmental dynamic compensation unit (200) includes an environmental status acquisition module and an optimized multi-source data storage module; The environmental status acquisition module is used to configure the following auxiliary sensors on the robot for acquiring environmental status: An IMU inertial measurement sensor for monitoring the vibration frequency and amplitude of the body; a contact force sensor mounted on an ultrasonic probe bracket for real-time feedback of applied pressure; a light intensity sensor for estimating the solar altitude angle and radiation intensity; and a wind speed and direction sensor. An environment vector is constructed after collecting the environmental status. The optimized multi-source data storage module is used to input environmental vectors into the optimization model to compensate and optimize the multi-source data, thereby obtaining optimized multi-source data.

8. The tower robot early warning and inspection control system based on multiple sensors according to claim 7, characterized in that: The optimization model includes a visual image deblurring compensation model, an infrared thermometry dynamic compensation model, and an ultrasonic thickness measurement compensation model, wherein: The visual image deblurring compensation model utilizes the robot's IMU to collect three-dimensional acceleration and angular velocity data in real time. After subtracting the influence of gravity from the acceleration data, time integration is performed to obtain the displacement trajectory. Simultaneously, the rotation component is calculated by combining the angular velocity data. Fast Fourier Transform is used to analyze the motion spectrum, automatically identifying low-frequency structural oscillations and high-frequency mechanical vibrations. Based on the motion trajectory within the exposure time period, a point spread matrix describing the image blur direction is dynamically generated. Principal component analysis is used to compress this matrix into a 64-dimensional feature vector. A U-Net++ structure containing symmetrical encoder and decoder paths is adopted, embedding motion attention gate modules on each cross-layer connection line between the encoder and decoder, wherein: Taking the image feature map and 64-dimensional feature vector of the current layer as input, the 64-dimensional feature vector is transformed by two layers of fully connected network and then concatenated with the image feature map through channel concatenation. A spatial attention weight map is generated through 1x1 convolution. The feature vector is multiplied with the visual features to establish a visual image deblurring compensation model. During the decoding process, motion condition information of different scales is gradually injected to achieve progressive processing from global blur correction to local detail restoration. The infrared temperature measurement dynamic compensation model constructs the thermal balance equation of the tower surface, dynamically calculates the background field of environmental thermal interference, and uses an iterative compensation algorithm to remove the influence of solar radiation and convective heat dissipation from the measured radiation value. Finally, it extracts the real temperature rise signal that only reflects the abnormal thermal state of the component itself and outputs optimized infrared thermal imaging data. The ultrasonic thickness measurement compensation model establishes a quantitative mapping relationship between probe contact pressure, tilt angle, ambient temperature, and measurement error. During real-time inspection, it synchronously collects environmental vectors, dynamically calculates triple compensation factors to correct ultrasonic thickness measurement data, and combines confidence assessment to trigger an automatic retest mechanism to output optimized ultrasonic thickness measurement data.

9. The tower robot early warning and inspection control system based on multiple sensors according to claim 1, characterized in that: The fusion early warning unit (300) includes a spatiotemporal registration module, a cross-validation module, a multimodal feature fusion module, and an early warning control module; The spatiotemporal registration module uses the inspection point number and gimbal angle provided by the robot positioning and synchronization module as indexes to establish the association of different sensor data. For the same inspection point, the optimized visual image data and optimized infrared thermal imaging data are aligned at the two-dimensional image level using feature-based image registration algorithms, while the optimized ultrasonic thickness measurement data are mapped to the corresponding visual / infrared image area according to the three-dimensional coordinates of the probe contact. The cross-validation module is used to build a multimodal joint criterion rule library, and only when multiple physical dimensions are co-existing abnormalities are they confirmed as real defects; The multimodal feature fusion module is used to process optimized visual image data, optimized infrared thermal imaging data, and optimized ultrasonic thickness measurement data through independent feature extraction sub-networks. The optimized ultrasonic thickness measurement data is converted into a one-dimensional feature vector. A fusion layer based on an attention mechanism is used to calculate the cross-modal attention weight between visual and infrared features. The features are then weighted and fused using this weight. The fused features are concatenated with the ultrasonic feature vector and input into a fully connected classification / regression network. The output is comprehensive inspection data that records the defect type and severity. The early warning control module is used to convert comprehensive inspection data into a comprehensive health score representing the overall risk level of the current inspection point. If the comprehensive health score is greater than the safety threshold, an early warning signal for a high-risk state is output, and the pan-tilt unit is controlled to take pictures from multiple angles, repeatedly collect data, and perform secondary verification. If the early warning signal is continuously issued, the walking mechanism sends a braking command.

Images