A multi-modal data fusion driven sewer monitoring device marching method

Abstract

The application relates to the technical field of sewer pipeline detection and multi-modal data fusion, and discloses a multi-modal data fusion driven sewer pipeline monitoring equipment advancing method, which comprises the following steps: acquiring original collection data through a sewer pipeline monitoring equipment; the sewer pipeline monitoring equipment is integrated with a visual collection unit, an infrared thermal imaging unit, an adaptive pipe diameter ground penetrating radar unit and an advancing mechanism; the original collection data is subjected to fusion processing to construct a standardized multi-modal data set; fusion feature extraction is carried out on the standardized multi-modal data set to output sewer pipeline disease environment fusion data; the disease environment fusion data comprises a disease type, a disease position coordinate, a disease confidence and a real-time pipe diameter calculation value; the sewer pipeline monitoring equipment is controlled to advance through the disease environment fusion data; and the advancing path is optimized based on the disease type during the advancing. According to the technical scheme, the monitoring equipment can be adapted to sewer pipelines with different diameters and different diseases.

Description

Technical Field

[0001] This invention relates to the field of drainage pipeline inspection and multimodal data fusion technology, and more specifically, to a method for the movement of drainage pipeline monitoring equipment driven by multimodal data fusion. Background Technology

[0002] Drainage pipes are a core component of urban infrastructure, and their health directly affects urban flood control, drainage, and ecological environment safety. During long-term service, pipes are susceptible to geological subsidence, water erosion, chemical corrosion, and other factors, resulting in cracks, corrosion, leakage, structural damage, and other defects. If these defects are not detected and repaired in a timely manner, they may lead to serious consequences such as road collapse and water pollution.

[0003] Existing drainage pipeline inspection equipment suffers from the following main shortcomings: First, data acquisition dimensions are limited, with most equipment relying solely on single video or radar data, making it difficult to comprehensively depict the visual state, temperature anomalies, and structural defects inside the pipeline. Second, pipe diameter adaptability is poor; traditional ground-penetrating radar sensors are fixed in installation and cannot adjust the detection distance according to different pipe diameters, resulting in unstable detection accuracy. Third, data fusion is low; most equipment uses simple stitching methods to process multi-source data, failing to fully leverage the complementary advantages of different modal data, and is prone to detection blind spots and misjudgments. Fourth, panoramic detection capabilities are insufficient; ordinary video acquisition has limited field of view, making it difficult to achieve comprehensive coverage of the pipeline's inner wall without blind spots.

[0004] In recent years, the development of multimodal data fusion and adaptive mechanical design technologies has provided new pathways to solve the aforementioned problems. In existing technologies, some pipeline inspection equipment has begun to attempt multimodal data integration, but it lacks dedicated fusion algorithms for panoramic video, infrared thermal imaging, and ground-penetrating radar. Meanwhile, adaptive robotic arm designs are mostly applied to industrial assembly scenarios and have not yet been deeply integrated with pipeline inspection equipment, failing to meet the inspection needs of ground-penetrating radar in pipelines of different diameters. Furthermore, the movement strategies of drainage pipeline monitoring equipment in pipelines do not adjust to different defects in the pipeline, easily exacerbating the degree of defects and causing secondary damage to the pipeline.

[0005] Therefore, there is an urgent need for technical solutions to integrate multimodal data for drainage pipeline monitoring, adjust the movement strategy of drainage pipeline monitoring equipment according to different defects in the pipeline, adapt to the detection needs of pipelines with different diameters, and make it suitable for comprehensive detection and defect diagnosis of urban drainage pipelines. Summary of the Invention

[0006] To achieve the above objectives, this application provides a method for the movement of drainage pipeline monitoring equipment driven by multimodal data fusion, comprising the following steps:

[0007] Raw data is acquired through drainage pipeline monitoring equipment. The drainage pipeline monitoring equipment integrates a visual acquisition unit 401, an infrared thermal imaging unit 403, an adaptive pipe diameter ground-penetrating radar unit 402, and a traveling mechanism 404. The adaptive pipe diameter ground-penetrating radar unit 402 includes a ground-penetrating radar sensor and an adaptive robotic arm. The adaptive robotic arm adopts a multi-section telescopic structure and is equipped with a laser distance sensor, a ground-penetrating radar sensor, a PID controller, and a torque adjustment module. The raw data acquired includes: visual information of the drainage pipeline, temperature distribution data of the inner wall of the pipeline, radar signals inside the pipeline, position coordinates of the drainage pipeline monitoring equipment, and attitude data of the robotic arm.

[0008] The raw collected data is fused and processed to construct a standardized multimodal dataset;

[0009] The standardized multimodal dataset is subjected to fusion feature extraction to output fusion data of the drainage pipeline's disease environment; the fusion data of the disease environment includes disease type, disease location coordinates, disease confidence level and real-time measured value of pipeline diameter;

[0010] The movement of drainage pipeline monitoring equipment is controlled by integrating disease and environmental data; during movement, the movement path is optimized based on the type of disease; the movement path optimization includes: controlling the detection mode of the robotic arm and controlling the movement path of the drainage pipeline monitoring equipment.

[0011] Among them, visual information of drainage pipes is used to show the texture of the inner wall of drainage pipes and the morphology of defects; temperature distribution data of the inner wall of pipes is used to highlight the temperature difference between defective areas and normal areas; and radar signals inside the pipes are used to show the three-dimensional structural data of the pipe structural layer thickness and internal defects.

[0012] Constructing a standardized multimodal dataset involves the following steps:

[0013] Panoramic video preprocessing is performed on the visual information of drainage pipes to generate a standardized panoramic image sequence;

[0014] Infrared thermal imaging data preprocessing is performed on the temperature distribution data of the inner wall of the pipeline to form an infrared thermal imaging temperature value map.

[0015] Ground-penetrating radar data preprocessing is performed on the radar signals inside the pipeline, converting them into two-dimensional structural feature maps;

[0016] By synchronizing timestamps and mapping spatial coordinates, standardized panoramic image sequences, infrared thermal imaging temperature value maps, and two-dimensional structural feature maps are spatially aligned, and the location coordinates of drainage pipeline monitoring equipment are added to unify the data format and generate a standardized multimodal dataset.

[0017] Furthermore, the fusion feature extraction includes:

[0018] An improved ResNet-50 network is used to encode the features of standardized panoramic image sequences and output semantic feature maps to capture the contours and details of pipe surface defects. A three-layer convolutional neural network is used to process the infrared thermal imaging temperature value map and output a temperature anomaly feature map to highlight the temperature anomaly region features corresponding to the defects. A CNN+fully connected layer is used to extract features from the two-dimensional structural feature map to obtain a structural feature map that reflects the thickness and structural features of the pipe structure and defects.

[0019] An attention-driven dynamic weighted fusion module is constructed, which takes semantic feature map, temperature anomaly feature map and structural feature map as input, and generates a fusion feature map of the same dimension; the fusion feature map is decoded and outputs fusion data of the disease environment of drainage pipeline.

[0020] Furthermore, the travel path of the drainage pipeline monitoring equipment is controlled through the adaptive control of the travel system for defects; the adaptive control of the travel system for defects refers to the travel mechanism 404 of the drainage pipeline monitoring equipment dynamically adjusting its speed, trajectory and travel parameters in combination with the characteristics of defect distribution and the pipeline environment to achieve multi-party collaboration.

[0021] The detection mode of the robotic arm combines the type and confidence level of the defect with the real-time calculated pipe diameter as a benchmark. It uses a PID control algorithm to drive the robotic arm to adjust the deployment angle and length in real time to adapt to different pipe diameter environments. The detection mode includes: robotic arm defect adaptive adjustment mode and sensor parameter defect adaptive optimization.

[0022] Among them, sensor parameter disease adaptive optimization refers to: the sensor adjusts its working parameters according to the type of disease to maximize the accuracy and effectiveness of data acquisition;

[0023] The robotic arm's disease-adaptive adjustment modes include: fine detection mode, area scanning mode, focused tracking mode, and safe avoidance mode.

[0024] Among them, the fine detection mode includes: adjusting the sampling frequency of the laser distance sensor and the adjustment accuracy of the PID controller to ensure that the radar sensor fits the pipe wall along the diseased area and collects data on the crack depth and extension trajectory;

[0025] The planar scanning mode includes: controlling the robotic arm to move the ground-penetrating radar sensor laterally at a specified step size, performing a grid-like scan of the diseased area, simultaneously recording the structural thickness changes at different locations, and generating a disease severity distribution map;

[0026] The focused tracking mode includes: acquiring the location coordinates of the center point of the diseased area, driving the radar sensor to move closer to the center point of the diseased area, and enhancing the signal strength to identify the aperture and orientation of the center point;

[0027] The safety avoidance mode includes: shortening the extension length of the robotic arm, adjusting the angle of the radar sensor, and collecting the three-dimensional structural data of the diseased area at an angle to avoid signal distortion caused by directly facing the diseased area.

[0028] Furthermore, path optimization based on disease type control includes:

[0029] When the defect type is crack: If the defect confidence level reaches a specified threshold, the robotic arm's defect adaptive adjustment mode activates the fine detection mode; sensor parameter defect adaptive optimization includes: adjusting the panoramic camera frame rate to improve image resolution, enhance crack details, and improve crack edge contrast; focusing the infrared thermal imager's temperature measurement range to improve temperature resolution, capture subtle temperature differences at the crack, retain only temperature data around the crack, and shorten the temperature sampling interval; the travel system defect adaptive control includes: if there are dense cracks in the current area, controlling the drainage pipe monitoring equipment to slow down, increasing the 404 grounding pressure of the travel mechanism, and preventing slippage;

[0030] When the type of defect is corrosion: if the defect confidence level reaches a specified threshold, the robotic arm's defect adaptive adjustment mode switches to planar scanning mode; sensor parameter defect adaptive optimization includes: adjusting the center frequency of the ground penetrating radar to match different corrosion layer thicknesses; the infrared thermal imager starts regional constant temperature monitoring, records the average temperature of the corrosion area at a specified preset frequency, calculates the temperature difference between the corrosion area and the normal area, and generates a corrosion degree heat map; the travel system defect adaptive control includes: if a large area of ​​corrosion is identified, controlling the travel path of the drainage pipe monitoring equipment to trigger a reciprocating scanning mode to ensure that multimodal data acquisition covers the boundary of the corrosion area;

[0031] When the defect type is leakage, the robotic arm's defect adaptive adjustment mode activates the focus tracking mode; sensor parameter defect adaptive optimization includes: the high-definition industrial camera activates the low-light enhancement mode, and histogram equalization is used to eliminate fog blur caused by leakage, thereby improving image clarity; the travel system defect adaptive control includes: if leakage points are dense, the drainage pipe monitoring equipment pauses its movement, rotates in place, and uses the high-definition industrial camera and ground-penetrating radar sensor to jointly locate the relative position of the leakage point and the detection area, generating a "leak point-area" correlation map;

[0032] When the defect type is structural damage, the robotic arm's defect adaptive adjustment mode switches to the safe avoidance mode; sensor parameter defect adaptive optimization includes: increasing the radar sensor's transmission power to ensure penetration of the debris at the defect site to obtain deep structural data; the travel system defect adaptive control includes: controlling the drainage pipe monitoring equipment to detour and adjusting the direction of movement to prevent the equipment from getting stuck in the gap at the defect site.

[0033] Furthermore, the travel path optimization is achieved through adaptive control, including: adaptive adjustment control of the robotic arm, control of the equipment's travel path and speed, and adaptive control of sensor parameters.

[0034] Among them, the adaptive adjustment control of the robotic arm includes: using the real-time measured value of the pipe diameter as the core control target, and combining the real-time distance data fed back by the laser distance sensor to construct a PID closed-loop control system;

[0035] After the laser distance sensor detects the distance, it determines whether the distance is within the specified range. If it is within the range, the current posture of the robotic arm is maintained; otherwise, the equipment control system quickly calculates the extension adjustment amount, drives the robotic arm to extend and retract, adjusts the unfolding angle and length, and triggers the laser distance sensor to detect the distance again.

[0036] Equipment travel path and speed control includes: generating the optimal travel trajectory based on the location coordinates of defects; reducing travel speed when densely defected areas are identified; triggering detour planning if large structural defects are identified; and maintaining an efficient travel speed in normal areas without obvious defects.

[0037] Sensor parameter adaptive control includes: constructing a multimodal data feedback closed loop, transmitting the fused data of the disease environment and the original collected data to the equipment control system in real time to generate control commands, and dynamically adjusting the sensor operating parameters.

[0038] According to the present invention, the problem of blind spots in conventional drainage pipeline inspection can be overcome, significantly improving the comprehensiveness and accuracy of identifying internal pipeline defects. Simultaneously, the present invention combines an adaptive pipe diameter robotic arm structure with the characteristic design of the monitoring equipment to create a travel route that allows the monitoring equipment to adapt to drainage pipelines of different diameters and with different defects. This effectively overcomes the problems of traditional rigid structures having difficulty navigating through defective pipelines, equipment being easily contaminated, and the inspection process potentially exacerbating pipeline defects. It provides efficient and reliable technical equipment support for the accurate diagnosis, risk warning, and intelligent operation and maintenance of urban drainage networks, demonstrating outstanding practicality and significant engineering application value. Attached Figure Description

[0039] Figure 1 This is a step diagram of a method for the movement of a drainage pipeline monitoring device driven by multimodal data fusion according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the PID closed-loop control logic of the mechanism arm of the drainage pipeline monitoring equipment provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the standardized multimodal dataset generation process provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a drainage pipeline monitoring device provided according to an embodiment of the present invention;

[0043] Reference numerals: 401-Visual acquisition unit, 402-Adaptive diameter ground-penetrating radar unit, 403-Infrared thermal imaging unit, 404-Traveling mechanism. Detailed Implementation

[0044] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] The multimodal data fusion-driven method for the movement of drainage pipeline monitoring equipment provided by this invention is as follows: Figure 1 As shown, it includes the following steps:

[0046] Step S100: Obtain raw data through drainage pipe monitoring equipment;

[0047] The drainage pipeline monitoring equipment provided by this invention, such as Figure 4 As shown, it integrates a visual acquisition unit 401, an infrared thermal imaging unit 403, and an adaptive diameter ground-penetrating radar unit 402 with an adaptive robotic arm.

[0048] 1) The visual acquisition unit 401 uses a multi-camera ring array layout to capture visual information such as the texture of the inner wall and the morphology of defects in the drainage pipe; the visual acquisition unit 401 can be composed of multiple high-definition industrial cameras, with a resolution of no less than 2560×1440 for each camera and a frame rate of 15-20fps; after the data is acquired by a single camera, a 360° panoramic video without blind spots is generated through an image stitching algorithm.

[0049] 2) The infrared thermal imaging unit 403 uses a high-resolution infrared thermal imager to collect temperature distribution data of the inner wall of the pipe, highlighting the temperature difference between the diseased areas such as corrosion and leakage and the normal areas.

[0050] 3) The adaptive pipe diameter ground-penetrating radar unit 402 combines a ground-penetrating radar sensor with an adaptive robotic arm to collect radar signals that reflect the three-dimensional structural data of the pipe structure, such as the thickness of the structural layers and internal defects. The adaptive robotic arm adopts a multi-section telescopic structure and is equipped with a laser distance sensor, a ground-penetrating radar sensor, a PID controller and a torque adjustment module. It can adaptively extend and retract within the pipe diameter range of DN300-DN2000mm, so that the radar sensor always maintains the optimal detection distance with the inner wall of the pipe to obtain the optimal three-dimensional structural data. At the same time, the robotic arm responds to the PID control command to adjust its posture. It uses its own onboard sensors (such as angle sensors and displacement sensors) to acquire sensor data in real time and combines the feedback data of the PID control command to obtain the robotic arm posture data.

[0051] The drainage pipeline monitoring equipment is also equipped with a PID-controlled travel mechanism 404, which controls speed, direction and slope according to control commands, and has a built-in IMU and GPS to obtain the location coordinates of the drainage pipeline monitoring equipment.

[0052] The PID control command is generated by the equipment control system based on the fusion data of the disease environment and the original collected data. The equipment control system can be deployed on the drainage pipeline monitoring equipment or implemented on a server outside the drainage pipeline monitoring equipment. It sends PID control commands to the drainage pipeline monitoring equipment through wired or wireless networks.

[0053] The drainage pipeline monitoring equipment constructs a multimodal detection system through the above units. The raw data acquired includes: visual information of the drainage pipeline, temperature distribution data of the inner wall of the pipeline, radar signals inside the pipeline, position coordinates of the drainage pipeline monitoring equipment, and posture data of the robotic arm.

[0054] The drainage pipeline monitoring equipment also integrates an edge processing unit, which acquires raw data from the visual acquisition unit 401, the adaptive pipe diameter ground penetrating radar unit 402, and the infrared thermal imaging unit 403 via an Ethernet interface, and pushes it to the equipment control system in real time via wired or wireless network through the MQTT protocol.

[0055] This invention provides embodiments:

[0056] The visual acquisition unit 401 uses six Basler acA2500-14uc high-definition industrial cameras, which are evenly distributed in a ring at the front of the equipment. The overlapping field of view of adjacent cameras is ≥30°. The resolution of a single camera is set to 2560×1440, the frame rate is 15fps, and data is transmitted via gigabit Ethernet. The SIFT feature matching algorithm and fast stitching technology are used to generate 360° panoramic video in real time, with a stitching error of ≤1 pixel.

[0057] The infrared thermal imaging unit 403 uses a high-resolution infrared thermal imager, model FLIR A655sc, with a temperature measurement range of -40℃ to 150℃, a temperature resolution of 0.03℃, and a spatial resolution of 640×512. The device integrates a temperature calibration module, calibrated using a blackbody radiation source to ensure a temperature measurement error ≤ ±0.5℃.

[0058] The adaptive pipe diameter ground-penetrating radar unit 402 consists of a ground-penetrating radar sensor with a center frequency of 1.5 GHz and a detection depth of 0.1-1.5 m, and a multi-section telescopic robotic arm. The robotic arm is made of aluminum alloy and includes three telescopic sections with a maximum telescopic stroke of 800 mm. It is equipped with a laser distance sensor (measurement accuracy ±1 mm) and a torque adjustment module. Through a PID control algorithm, the robotic arm can adjust its deployment angle and length in real time according to the pipe diameter, maintaining a constant distance of 3-5 cm between the radar sensor and the inner wall of the pipe, adapting to pipe diameters ranging from DN300 to DN2000 mm.

[0059] Step S110: The raw collected data is fused and processed to construct a standardized multimodal dataset;

[0060] The process of constructing a standardized multimodal dataset is as follows: Figure 3 As shown, it includes:

[0061] 1) Perform panoramic video preprocessing on the visual information of drainage pipes, including: removing image noise by Gaussian filtering, optimizing the uneven lighting problem by histogram equalization, correcting camera distortion by perspective transformation, and completing multi-camera image stitching based on SIFT feature matching to generate a standardized panoramic image sequence.

[0062] 2) Perform infrared thermal imaging data preprocessing on the pipe inner wall temperature distribution data, including: visualizing the pipe inner wall temperature distribution data to generate a grayscale image; using median filtering to eliminate thermal noise; establishing the mapping relationship between grayscale values ​​and actual temperature through temperature calibration experiments, converting the grayscale image into a temperature value image, and completing the normalization process by combining the mean and variance of the normal area temperature of the pipe to form an infrared thermal imaging temperature value image.

[0063] 3) Perform ground-penetrating radar data preprocessing on the radar signals inside the pipeline, including: removing clutter interference from the radar signals through wavelet transform, separating the effective signals from environmental noise using a background removal algorithm, spatially calibrating the radar signals based on the robotic arm's posture data, and converting the one-dimensional radar signals into two-dimensional structural feature maps.

[0064] Furthermore, by synchronizing timestamps and mapping spatial coordinates, the standardized panoramic image sequence, infrared thermal imaging temperature value map, and two-dimensional structural feature map are spatially aligned, the device's location coordinates are added, and the data format (e.g., JSON format) is unified to construct a standardized multimodal dataset.

[0065] The present invention provides an embodiment in which, in the panoramic video preprocessing stage, a 5×5 Gaussian filter is used to remove Gaussian noise, histogram equalization is used to enhance image contrast, perspective transformation is used to correct camera distortion (distortion correction error ≤ 0.3%), and finally, multi-camera image stitching is completed based on the RANSAC algorithm to generate a standardized panoramic image sequence with a resolution of 4096×2048.

[0066] In the infrared thermal imaging data preprocessing stage, 3×3 median filtering was used to eliminate salt-and-pepper noise. A temperature calibration experiment was conducted to establish the mapping relationship between grayscale values ​​and temperature values ​​(T=0.021G+19.8, where T is the temperature value and G is the grayscale value). The mean temperature (set to 25℃) and variance (set to 1.2℃) of the normal area of ​​the pipeline were calculated. 2 The temperature map is normalized to highlight abnormal temperature areas.

[0067] In the ground-penetrating radar data preprocessing stage, wavelet transform (db4 wavelet basis, decomposition level 3) is used to remove clutter interference, effective signals are separated from environmental noise by background removal algorithm, and the radar signal is spatially calibrated based on the robotic arm attitude data, converting the one-dimensional radar echo signal into a two-dimensional structural feature map with a resolution of 512×512.

[0068] When performing spatiotemporal alignment, the device's built-in GPS and IMU data are used as timestamps to achieve time synchronization of panoramic video, infrared data, and radar data (synchronization error ≤10ms); spatial coordinate alignment is completed through camera extrinsic parameter calibration and radar coordinate system mapping to ensure the positional consistency of multi-source data.

[0069] Step S120: Extract fusion features from the standardized multimodal dataset and output the fusion data of the drainage pipeline's disease environment; the fusion data of the disease environment includes disease type, disease location coordinates, disease confidence level and real-time calculated value of pipeline diameter.

[0070] The process of feature extraction fusion includes the following stages:

[0071] 1) Feature extraction stage, specifically including the following steps:

[0072] An improved ResNet-50 network is used to encode the features of standardized panoramic image sequences and output semantic feature maps. The first three layers of the network extract shallow texture features of 128×128×64 dimensions, and the last two layers extract deep semantic features of 16×16×2048 dimensions, accurately capturing the outline and details of defects on the pipe surface.

[0073] The infrared thermal imaging temperature value map is processed by a 3-layer convolutional neural network with kernel sizes of 3×3, 3×3 and 1×1, and ReLU activation function. The final output is a 16×16×1024-dimensional temperature anomaly feature map, which focuses on the temperature change region to highlight the temperature anomaly region features corresponding to corrosion, leakage and other diseases.

[0074] By using CNN+fully connected layers to extract features from the two-dimensional structural feature map, the CNN layer is responsible for capturing local structural features, and the fully connected layer realizes feature dimension compression and integration, resulting in a 16×16×1024 dimension structural feature map, which clearly represents the internal structural state of the pipeline and reflects the structural features such as the thickness and defects of the pipeline structure.

[0075] 2) Construct an attention mechanism-driven dynamic weighted fusion module, design intra-modal contrast loss and inter-modal complementary loss, input semantic feature maps, temperature anomaly feature maps and structural feature maps into the attention fusion module, and adjust the weight ratio in real time according to the data quality (clarity, signal-to-noise ratio, signal strength) scores of each modality.

[0076] The sharpness score is achieved using the entropy method: the image sharpness is quantified by calculating the information entropy of the feature map, and the higher the entropy value, the better the sharpness; the signal-to-noise ratio is achieved by the ratio of signal power to noise power; the signal strength is represented by the average pixel value of the feature map, which can reflect the effective information content of the data.

[0077] Based on the three scores, a weighted summation method is used to obtain the comprehensive quality score of each modality. The fusion weights are dynamically allocated according to the comprehensive quality score, with higher scores having larger weight proportions. Subsequently, the three types of features are fused into a unified 16×16×2048 fusion feature map through matrix weighted summation.

[0078] The fused feature map is decoded to output the disease type, disease confidence level (≥0.7 for effective identification), and real-time calculated pipe diameter.

[0079] 3) Add a modal conflict resolution unit. When there are contradictions in the pipeline status reflected by different modal data, call the preset domain rules and historical data to make logical judgments and correct the deviation of the fusion result.

[0080] In practical applications, when the pipeline conditions reflected by different modal data are contradictory, a pre-defined domain rule base is invoked in conjunction with historical detection data (association patterns of multi-modal data on similar pipe diameters and similar defects) to perform logical judgment and correct the deviation of the fused feature map. Common contradictions include: structural defects are identified in the two-dimensional structural feature map but surface defects are not reflected in the standardized panoramic image sequence; and the locations of infrared temperature anomaly areas and radar structural defect areas do not match. Based on this contradiction, the pre-defined domain rule base can determine that: the structural defect is located below the inner wall of the pipe and is not visually directly apparent; and temperature anomaly areas are preferentially associated with corrosion or leakage defects.

[0081] After correction by the modal conflict resolution unit, the final output is the fused data of the drainage pipeline's defect environment. The defect environment fusion data includes the pipeline defect type, defect confidence level (≥0.7 for valid identification), and real-time calculated pipeline diameter; among which, pipeline defect types include cracks, corrosion, leakage, and structural damage.

[0082] The fusion data of the disease environment also includes the location coordinates of the disease and the real-time measured value of the pipe diameter; the location coordinates of the disease are calculated by combining the location coordinates of the equipment with the pixel position corresponding to the disease type; the real-time measured value of the pipe diameter is generated by co-calculating the location size reflected by the standardized panoramic image sequence with the radar detection range.

[0083] Step S130: Control the movement of drainage pipeline monitoring equipment by integrating disease and environmental data.

[0084] During the movement of the drainage pipeline monitoring equipment, the movement path is optimized based on the type of defect. The movement path optimization includes controlling the detection mode of the robotic arm and controlling the movement path of the drainage pipeline monitoring equipment.

[0085] 1) The travel path of the drainage pipeline monitoring equipment is achieved through the adaptive control of the travel system for defects. The adaptive control of the travel system for defects refers to the travel mechanism 404 of the drainage pipeline monitoring equipment dynamically adjusting its speed, trajectory and travel parameters in combination with the characteristics of defect distribution and the pipeline environment to achieve multi-party collaboration.

[0086] 2) The detection mode combines the type and confidence level of the defect with the real-time calculated pipe diameter as a benchmark. A PID control algorithm drives the multi-section telescopic structure of the robotic arm to adjust its deployment angle and length in real time, ensuring a constant distance between the radar sensor and the inner wall of the pipe. This mode adapts to different pipe diameter environments. The robotic arm's detection modes include: robotic arm defect adaptive adjustment mode and sensor parameter defect adaptive optimization mode.

[0087] Sensor parameter disease adaptive optimization refers to the adjustment of operating parameters of each sensor according to the type of disease to maximize the accuracy and effectiveness of data acquisition.

[0088] The robotic arm's adaptive adjustment modes for fault detection include: fine detection mode, area scanning mode, focused tracking mode, and safe avoidance mode. Specifically:

[0089] The fine detection mode includes: adjusting the sampling frequency of the laser distance sensor from the current value (e.g., 10Hz) to 20Hz, and optimizing the adjustment accuracy of the PID controller from the current value (e.g., ±2mm) to ±1mm, to ensure that the radar sensor dynamically fits the pipe wall along the path of the diseased area and collects data on the crack depth and extension trajectory.

[0090] The planar scanning mode includes: controlling the robotic arm to move the ground-penetrating radar sensor in a specified lateral movement step (e.g., 5 mm / step) to perform a grid-like scan of the diseased area, simultaneously recording the structural thickness changes at different locations, and generating a disease severity distribution map;

[0091] The focused tracking mode includes: acquiring the location coordinates of the center point of the diseased area, driving the radar sensor to approach the center point of the diseased area to a specified distance (e.g., 2-3cm) lower than the normal distance (e.g., 3-5cm), while enhancing the signal strength to identify the aperture and orientation of the center point;

[0092] The safety avoidance mode includes: shortening the extension length of the robotic arm to avoid collisions between the robotic arm and the diseased area, adjusting the angle of the radar sensor to collect the three-dimensional structural data of the diseased area at an angle, and avoiding signal distortion caused by directly facing the diseased area.

[0093] Optimization of the travel path of drainage pipeline monitoring equipment based on disease type includes:

[0094] 1) The type of defect is cracking:

[0095] If the disease confidence level reaches a specified threshold (e.g., ≥0.8), the robotic arm's disease adaptive adjustment mode will activate the fine detection mode.

[0096] A laser distance sensor tracks the crack edge, and a PID controller adjusts the robotic arm joint angle in real time based on the tangent angle of the crack's direction, ensuring that the radar sensor maintains a detection accuracy of 1mm along the crack's extension direction. For example, when detecting a longitudinal crack 200mm long and 3mm wide in a DN800mm pipe, the robotic arm adjusts its angle every 0.5s to ensure that the deviation between the radar scan line and the crack centerline is ≤2mm, and to fully acquire gradual data on the crack depth from 1mm to 5mm.

[0097] The sensor parameter disease adaptive optimization includes: adjusting the panoramic camera frame rate to 25fps (normally 15fps) to improve image resolution, enhance crack details, and improve crack edge contrast; focusing the infrared thermal imager temperature measurement range on 20-40℃ (normally -40 to 150℃) to improve temperature resolution to 0.02℃, capturing subtle temperature differences at the cracks, and retaining only the temperature data around the cracks to reduce redundant calculations and shorten the temperature sampling interval.

[0098] The adaptive control of the travel system for defects includes: if there are dense cracks in the current area (e.g., ≥2 cracks in a 1m pipe section), the speed of the drainage pipe monitoring equipment is reduced, and the ground pressure of the tracks is increased to prevent the equipment from slipping due to unevenness of the pipe inner wall. For example, in a DN300mm small-diameter pipe, if 5 transverse cracks are detected in a 10m pipe section, the travel speed is reduced from 0.3m / s to 0.08m / s, and the 404 travel mechanism activates anti-slip protrusions (3mm high) to prevent the equipment from slipping at the cracks; at the same time, the equipment pauses for 0.5s for each crack detected to ensure that the panoramic camera captures a complete image of the crack.

[0099] 2) The type of disease is corrosion defect:

[0100] If the confidence level of the defect reaches a specified threshold (e.g., ≥0.8), and the area of ​​the abnormal infrared temperature region reaches a specified level (e.g., percentage ≥3%) or the temperature deviation reaches a specified level (e.g., ≥5℃), the robotic arm's defect adaptive adjustment mode switches to planar scanning mode: Considering the planar distribution characteristics of the corrosion area, the robotic arm adopts a composite motion mode combining lateral stepping and longitudinal scanning. (Example: A DN1500mm pipe with an inner area of ​​0.5m²...) 2 Taking the corroded area as an example, the robotic arm moves horizontally in a step of 5mm and scans vertically in a range of 30cm. It generates a structural thickness matrix of the corroded area (resolution 1mm×1mm) through 120 scans. Combined with infrared temperature data, the severity of corrosion is distinguished (mild: thickness loss ≤10%, temperature deviation 3-5℃; severe: thickness loss ≥30%, temperature deviation ≥8℃).

[0101] The adaptive optimization of sensor parameters for corrosion includes: adjusting the center frequency of the ground-penetrating radar, dynamically switching frequencies within the 1.5-2.0 GHz range to match different corrosion layer thicknesses (light corrosion: 1.5 GHz frequency, heavy corrosion: 2.0 GHz frequency) to enhance the penetration capability of the corrosion layer on the inner wall of the pipe; and initiating constant temperature monitoring of the area using an infrared thermal imager, recording the average temperature of the corrosion area at a specified preset frequency (e.g., 5 seconds), calculating the temperature difference between the corrosion area and the normal area, and generating a corrosion degree heat map (red: temperature difference ≥ 8℃, yellow: 5-8℃, green: < 5℃), providing raw data for analyzing the correlation between corrosion rate and temperature. For example, for a ring-shaped corrosion area covering 1 / 3 of the circumference of a DN2000mm pipe, a spiral movement trajectory is planned: gradually approaching the center from the edge of the corrosion area, moving 50cm per circle, and covering the entire corrosion area through 3 circles of scanning, avoiding omissions of corrosion boundaries due to movement in a single direction.

[0102] The adaptive control of the travel system for defects includes: if a large area of ​​corrosion is identified (e.g., the corrosion area accounts for ≥1 / 4 of the pipe circumference), the travel path of the drainage pipe monitoring equipment is controlled to trigger a reciprocating scanning mode, that is: multiple back-and-forth movements are made within a specified range (e.g., 50cm) before and after the corrosion area to ensure that multimodal data acquisition covers the boundary of the corrosion area.

[0103] 3) The type of defect is leakage:

[0104] Because leaks are often accompanied by water vapor infiltration, causing false reflections in radar signals, the robotic arm's defect adaptive adjustment mode activates a focused tracking mode: the robotic arm enhances the signal-to-noise ratio by shortening the distance between the radar sensor and the pipe wall (from 3-5cm to 2-3cm), and simultaneously adjusts the sensor angle to 15° with the pipe wall to avoid direct interference from water vapor in signal reception. For example, when a 5mm diameter leak is detected in a DN500mm pipe, after locating the leak, the robotic arm drives the radar sensor to scan three times at different angles (0°, 15°, and 30°). By superimposing the signals, water vapor interference is eliminated, and the depth of the leak channel is accurately identified.

[0105] Sensor parameter defect adaptive optimization includes: enabling low-light enhancement mode for high-definition industrial cameras to cope with the humid and foggy environment inside pipes caused by leakage, and eliminating fog blur caused by leakage through histogram equalization to improve image clarity.

[0106] The adaptive control of the travel system for defects includes: if there are dense leaks (e.g., ≥n leaks within a 3m area), the drainage pipe monitoring equipment stops traveling, rotates 360° in place, and uses a high-definition industrial camera and ground-penetrating radar sensor to locate the relative position of the leaks and the detection area, generating a "leak point-area" correlation map.

[0107] 4) The type of damage is structural breakage / defect:

[0108] Faced with severe structural defects such as pipe rupture and deformation, the robotic arm prioritizes equipment safety and ensures data integrity. In this case, the robotic arm's defect adaptive adjustment mode switches to a safety avoidance mode: for example, when a 10cm diameter collapse area is detected in a DN1000mm pipe, the robotic arm automatically shortens its extension length by 8cm and rotates the radar sensor to 60° to collect structural data of the collapse area from an oblique angle, avoiding direct contact between the robotic arm and the damaged edge. If there is a sharp edge at the collapse point, the robotic arm retracts the radar sensor into the main body of the equipment and completes the initial positioning only through a high-definition industrial camera and an infrared thermal imager.

[0109] Sensor parameter defect adaptive optimization includes: increasing the radar sensor's transmission power to ensure penetration of the debris at the damaged area to obtain deep structural data.

[0110] The adaptive control of the travel system for defects includes: when severe structural damage is detected (such as pipe breakage or collapse, with a pipe diameter deviation of ≥20% at the break point), the drainage pipe monitoring equipment is controlled to detour, for example, by moving away from the damaged area by a specified distance and adjusting its direction of movement to avoid the equipment getting stuck in the gap at the break point. For example, if a 20cm long break is detected in a DN800mm pipe, the travel path will automatically deviate from the break point by 30cm.

[0111] Path optimization is achieved through adaptive control, including:

[0112] 1) Adaptive Adjustment Control of the Robotic Arm: Using the real-time calculated pipe diameter as the core control objective, and combining it with real-time distance data from a laser distance sensor, a PID closed-loop control system is constructed (proportional coefficient Kp=0.8, integral coefficient Ki=0.2, derivative coefficient Kd=0.1). For example... Figure 2 As shown, after the laser distance sensor detects the distance, it determines whether the distance is within a specified range (e.g., whether it is within the optimal range of 3-5cm). If it is within the range, the current posture of the robotic arm is maintained; otherwise, the PID controller quickly calculates the extension adjustment amount, drives the robotic arm to extend and retract, adjusts the extension angle and length, and triggers the laser distance sensor to detect the distance again.

[0113] 2) Equipment Travel Path and Speed ​​Control: Integrating obstacle recognition results from visual information, structural detection data from radar, and the location coordinates of defects output by the fusion algorithm, the optimal travel trajectory is generated using the A-Star path planning algorithm. The cost function of the A-Star algorithm comprehensively considers factors such as path length, obstacle avoidance, and defect coverage priority, assigning higher coverage weights to areas with defect confidence ≥ 0.8. When the fusion result identifies a dense defect area, the travel speed is automatically reduced to extend the data acquisition time and improve the accuracy of defect detail capture; if a large structural defect (area ≥ 0.1m²) is identified... 2 If the defect is detected, a detour plan is triggered to avoid the defective area and prevent equipment collision damage; in normal areas without obvious defects, a certain speed (such as 0.3m / s) is maintained to achieve an efficient travel speed and balance detection efficiency and accuracy.

[0114] 3) Sensor Parameter Adaptive Control: A multimodal data feedback closed loop is constructed to transmit fused data of the defect environment and raw acquisition data to the equipment control system in real time, dynamically adjusting sensor operating parameters. When the fusion results show a high defect density, the number of images acquired is increased; based on the temperature range of infrared temperature anomaly characteristics, the infrared thermal imager's temperature measurement range (-40℃~60℃ or 60℃~150℃) is adaptively switched to improve temperature measurement accuracy; based on the signal strength of radar structural characteristics, the radar sensor's transmission power is adjusted to ensure clear structural feature signals can be obtained under different pipe materials and defect depth scenarios.

[0115] In this invention, the drainage pipeline monitoring equipment is controlled to move through the pipeline. After collecting more detailed data for different types and degrees of damage, the original data can be retrieved again, a new standardized multimodal dataset can be added, and damage environment fusion data can be obtained. The drainage pipeline monitoring equipment is then controlled to move through the pipeline, forming a complete closed loop.

[0116] Meanwhile, in order to improve the effectiveness of the entire process, simulation testing is provided to optimize the hardware parameters and algorithm models of the equipment. Under different pipe diameters and different disease scenarios, multiple rounds of iteration are completed based on indicators such as detection accuracy, adaptability, and stability to ensure that the equipment meets the actual engineering needs.

[0117] First, a pipeline simulation testing platform was built to simulate pipeline environments with different pipe diameters (such as DN300, DN800, DN1500, and DN2000mm) and different types of defects (cracks, corrosion, leakage, and structural defects) to test the equipment's data acquisition accuracy and pipe diameter adaptability. Then, based on the simulation test data, hardware parameters such as the robotic arm's extension accuracy and sensor installation positions were optimized, as well as algorithm rules such as fusion weights and control parameters, to achieve adaptive control. Finally, through multiple rounds of simulation iterations, the detection accuracy of the equipment in complex pipeline environments was verified to be ≥92%, and the pipe diameter adaptability error was ≤3%, meeting the requirements of engineering applications.

[0118] This invention integrates multi-source heterogeneous sensor data, including panoramic video, infrared thermal imaging, and ground-penetrating radar (GPR), to construct a multimodal, complementary pipeline internal state perception system. This overcomes the blind spots inherent in traditional drainage pipeline inspection processes, significantly improving the comprehensiveness and accuracy of defect identification. Furthermore, by combining the characteristics of an adaptive pipe diameter robotic arm structure and monitoring equipment with a customized path design, the monitoring equipment can adapt to drainage pipes of different diameters. It also employs customized detection strategies for different defect types and degrees, effectively overcoming the difficulties of traditional rigid structures navigating through defective pipelines, equipment contamination, and the increased risk of pipeline damage during inspection. This provides efficient and reliable technical equipment support for the accurate diagnosis, risk warning, and intelligent operation and maintenance of urban drainage networks, demonstrating outstanding practicality and significant engineering application value.

[0119] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for the movement of drainage pipeline monitoring equipment driven by multimodal data fusion, characterized in that, Includes the following steps: Raw data is acquired through drainage pipeline monitoring equipment; the drainage pipeline monitoring equipment integrates a visual acquisition unit, an infrared thermal imaging unit, an adaptive pipe diameter ground penetrating radar unit, and a traveling mechanism; the adaptive pipe diameter ground penetrating radar unit includes a ground penetrating radar sensor and an adaptive robotic arm. The adaptive robotic arm adopts a multi-section telescopic structure and is equipped with a laser distance sensor, a ground-penetrating radar sensor, a PID controller, and a torque adjustment module. The raw data collected includes: visual information of the drainage pipe, temperature distribution data of the pipe's inner wall, radar signals inside the pipe, position coordinates of the drainage pipe monitoring equipment, and robotic arm posture data. Among these, the visual information of the drainage pipe is used to show the texture of the drainage pipe's inner wall and the morphology of defects; the temperature distribution data of the pipe's inner wall is used to highlight the temperature difference between the defective area and the normal area; and the radar signals inside the pipe are used to show the three-dimensional structural data of the pipe's structural layers and internal defects. The original collected data is fused and processed to construct a standardized multimodal dataset. The construction of the standardized multimodal dataset includes: synchronizing the standardized panoramic image sequence, infrared thermal imaging temperature value map and two-dimensional structural feature map spatially by synchronizing the timestamps and mapping the spatial coordinates, adding the location coordinates of the drainage pipeline monitoring equipment, unifying the data format, and generating a standardized multimodal dataset. A standardized multimodal dataset is used for fusion feature extraction to output fused data on the disease environment of drainage pipelines. This fused data includes disease type, disease location coordinates, disease confidence level, and real-time calculated pipe diameter. The fusion feature extraction includes: using an improved ResNet-50 network to encode the standardized panoramic image sequence and output a semantic feature map to capture the contours and details of surface diseases on the pipeline; processing the infrared thermal imaging temperature map using a three-layer convolutional neural network to output a temperature anomaly feature map, highlighting the temperature anomaly region features corresponding to the disease; extracting features from the two-dimensional structural feature map using a CNN+fully connected layer to obtain a structural feature map reflecting the thickness and structural features of the pipeline structure; constructing an attention-driven dynamic weighted fusion module, inputting the semantic feature map, the temperature anomaly feature map, and the structural feature map to generate a unified-dimensional fusion feature map; and decoding the fusion feature map to output the fused data on the disease environment of the drainage pipelines. The movement of the drainage pipeline monitoring equipment is controlled by the fusion data of the disease environment; during the movement, the movement path is optimized based on the type of disease; the movement path optimization includes: controlling the detection mode of the robotic arm and controlling the movement path of the drainage pipeline monitoring equipment. The travel path of the drainage pipeline monitoring equipment is achieved through adaptive control of the travel system for defects; the adaptive control of the travel system for defects refers to the travel mechanism of the drainage pipeline monitoring equipment dynamically adjusting its speed, trajectory and travel parameters in combination with the characteristics of defect distribution and the pipeline environment; The detection mode of the robotic arm combines the type and confidence level of the defect with the real-time calculated pipe diameter as a benchmark. It uses a PID control algorithm to drive the robotic arm to adjust the deployment angle and length in real time to adapt to different pipe diameter environments. The detection mode includes: robotic arm defect adaptive adjustment mode and sensor parameter defect adaptive optimization.

2. The method for moving drainage pipeline monitoring equipment driven by multimodal data fusion according to claim 1, characterized in that, The construction of the standardized multimodal dataset includes the following steps: The visual information of the drainage pipe is subjected to panoramic video preprocessing to generate a standardized panoramic image sequence; The temperature distribution data of the inner wall of the pipe is preprocessed with infrared thermal imaging data to form an infrared thermal imaging temperature value map. The radar signals inside the pipeline are preprocessed using ground-penetrating radar data and converted into a two-dimensional structural feature map.

3. The method for moving drainage pipeline monitoring equipment driven by multimodal data fusion according to claim 1, characterized in that, The aforementioned sensor parameter disease adaptive optimization refers to: the sensor adjusting its operating parameters according to the type of disease to maximize the accuracy and effectiveness of data acquisition; The robotic arm's disease adaptive adjustment modes include: fine detection mode, area scanning mode, focused tracking mode, and safe avoidance mode.

4. The method for moving drainage pipeline monitoring equipment driven by multimodal data fusion according to claim 3, characterized in that... , The fine detection mode includes: adjusting the sampling frequency of the laser distance sensor and the adjustment accuracy of the PID controller to ensure that the radar sensor fits the pipe wall along the diseased area and collects data on the crack depth and extension trajectory. The planar scanning mode includes: controlling the robotic arm to move the ground-penetrating radar sensor laterally at a specified step length, performing a grid-like scan of the diseased area, simultaneously recording the structural thickness changes at different locations, and generating a disease severity distribution map; The focused tracking mode includes: acquiring the position coordinates of the center point of the diseased area, driving the radar sensor to approach the center point of the diseased area, and enhancing the signal strength to identify the aperture and orientation of the center point; The safety avoidance mode includes: shortening the extension length of the robotic arm, adjusting the angle of the radar sensor, and collecting the three-dimensional structural data of the diseased area at an angle to avoid signal distortion caused by directly facing the diseased area.

5. The method for moving drainage pipeline monitoring equipment driven by multimodal data fusion according to claim 3, characterized in that... The path optimization based on disease type control includes: When the defect type is a crack: if the defect confidence level reaches a specified threshold, the robotic arm's defect adaptive adjustment mode switches to a fine detection mode; sensor parameter defect adaptive optimization includes: adjusting the panoramic camera frame rate to improve image resolution, enhance crack details, and enhance crack edge contrast; focusing the infrared thermal imager's temperature measurement range to improve temperature resolution, capture subtle temperature differences at the crack, and retain only temperature data around the crack, while shortening the temperature sampling interval; the travel system defect adaptive control includes: if there are dense cracks in the current area, controlling the drainage pipe monitoring equipment to slow down, increasing the grounding pressure of the travel mechanism, and avoiding slippage; When the disease type is corrosion defect: if the disease confidence level reaches a specified threshold, the robotic arm's disease adaptive adjustment mode switches to a planar scanning mode; the sensor parameter disease adaptive optimization includes: adjusting the ground penetrating radar center frequency to match different corrosion layer thicknesses; the infrared thermal imager starts regional constant temperature monitoring, records the average temperature of the corrosion area at a specified preset frequency, calculates the temperature difference between the corrosion area and the normal area, and generates a corrosion degree heat map; the travel system disease adaptive control includes: if a large area of ​​corrosion is identified, controlling the travel path of the drainage pipe monitoring equipment to trigger a reciprocating scanning mode to ensure that multimodal data acquisition covers the boundary of the corrosion area. When the defect type is leakage, the robotic arm's defect adaptive adjustment mode switches to focus tracking mode; the sensor parameter defect adaptive optimization includes: the high-definition industrial camera turns on low-light enhancement mode, and histogram equalization is used to eliminate fog blur caused by leakage, thereby improving image clarity; the travel system defect adaptive control includes: if leakage points are dense, the drainage pipe monitoring equipment pauses its travel, rotates in place, and uses the high-definition industrial camera and ground-penetrating radar sensor to jointly locate the relative position of the leakage point and the detection area, generating a "leak point-area" correlation map; When the type of defect is structural damage, the robotic arm's defect adaptive adjustment mode switches to a safe avoidance mode; the sensor parameter defect adaptive optimization includes: increasing the radar sensor's transmission power to ensure penetration of the debris at the defect site to obtain deep structural data; the travel system defect adaptive control includes: controlling the drainage pipe monitoring equipment to detour and adjusting its direction of movement to prevent the equipment from getting stuck in the gap at the defect site.

6. The method for moving drainage pipeline monitoring equipment driven by multimodal data fusion according to claim 1, characterized in that... The path optimization is achieved through adaptive control, including: adaptive adjustment control of the robotic arm, control of the equipment's path and speed, and adaptive control of sensor parameters.

7. The method for moving drainage pipeline monitoring equipment driven by multimodal data fusion according to claim 6, characterized in that... The adaptive adjustment control of the robotic arm includes: using the real-time measured pipe diameter as the core control target, and combining it with the real-time distance data fed back by the laser distance sensor to construct a PID closed-loop control system; After the laser distance sensor detects the distance, it determines whether the distance is within the specified range. If it is within the range, the current posture of the robotic arm is maintained; otherwise, the equipment control system quickly calculates the extension adjustment amount, drives the robotic arm to extend and retract, adjusts the unfolding angle and length, and triggers the laser distance sensor to detect the distance again. The equipment's travel path and speed control includes: generating an optimal travel trajectory based on the coordinates of the defect location; reducing the travel speed when densely affected areas are identified; triggering detour planning if a large structural defect is identified; and maintaining an efficient travel speed in normal areas without obvious defects. The adaptive control of sensor parameters includes: constructing a multimodal data feedback closed loop, transmitting the fused data of the disease environment and the original collected data to the equipment control system in real time to generate control commands, and dynamically adjusting the sensor operating parameters.

Images