A sewer pipeline disease segmentation method based on multi-modal data fusion

Abstract

The present application relates to the technical field of sewer pipeline detection, and discloses a sewer pipeline disease segmentation method based on multi-modal data fusion, comprising the following steps: S1, collecting multi-modal data; S2, pre-processing the multi-modal data to obtain standardized data; S3, extracting the features of the standardized data to obtain multi-modal features; S4, performing self-adaptive weighted fusion operation on the shallow features and the deep features to obtain a multi-modal fusion feature map; S5, inputting the multi-modal fusion feature map into a segmentation decoding network to obtain a preliminary disease segmentation result; the segmentation decoding network is generated by training through a hybrid loss function; and S6, performing morphological processing and connected domain analysis on the preliminary disease segmentation result to obtain a final disease segmentation result. The present application plays a complementary role between different modal data, solves the problem of insufficient feature representation of single modal data under complex working conditions, and improves the accuracy and stability of sewer pipeline disease segmentation.

Description

Technical Field

[0001] This invention relates to the field of drainage pipeline inspection technology, and more specifically, to a method for segmenting drainage pipeline defects based on multimodal data fusion. Background Technology

[0002] Drainage pipes, a crucial component of urban infrastructure, are primarily responsible for sewage discharge and rainwater drainage. Over long-term use, drainage pipes are susceptible to various factors such as geological subsidence, water erosion, and chemical corrosion, leading to problems like cracks, corrosion, damage, and joint detachment. If these issues are not detected and repaired promptly, they can result in a series of consequences, including sewage leakage, groundwater pollution, and road collapse, severely impacting urban operational safety and residents' quality of life.

[0003] In the past, drainage pipe inspections relied mainly on manual inspections down into manholes or closed-circuit television (CCTV). Manual inspections were inefficient and dangerous; CCTV inspections visualized the internal condition of pipes, but the results depended on human judgment, making them subjective, prone to missed detections, and difficult to quantify accurately. With the continuous development of computer vision and deep learning technologies, image processing can now be used to automatically segment pipe defects.

[0004] Currently, most deep learning-based pipeline defect segmentation methods rely primarily on single-modal visible light image data. However, in actual pipeline inspection, situations such as uneven lighting, obscuring the inner wall of the pipeline by stains, and unclear defect features are frequently encountered. In these cases, relying solely on visible light image data is insufficient to fully present the feature information of the defect, resulting in suboptimal accuracy and robustness of the segmentation model. For example, for tiny cracks in the inner wall of the pipeline, visible light images struggle to clearly show the crack edges in insufficient light; similarly, visible light images often fail to distinguish between corroded areas and ordinary stains within the pipeline.

[0005] To overcome the limitations of single-modal data in pipeline defect detection, techniques combining multimodal data for detection have emerged. However, most current multimodal data fusion methods simply stitch features together or employ weighted fusion strategies, failing to fully utilize the complementary advantages between different modalities. This results in insufficient feature fusion and loss of deep semantic information. For example, directly stitching visible light image features and laser point cloud depth features not only increases feature dimensionality and computational cost but also fails to effectively highlight features crucial for defect segmentation. Traditional weighted fusion methods, with their fixed weights, cannot adaptively adjust based on the quality and feature importance of different modalities, leading to unsatisfactory segmentation results under complex operating conditions.

[0006] Therefore, an efficient feature fusion method is needed to effectively combine the complementary characteristics of multimodal data to improve the accuracy and stability of drainage pipeline defect segmentation. Summary of the Invention

[0007] To achieve the above objectives, this application provides a method for segmenting drainage pipeline defects based on multimodal data fusion, comprising the following steps:

[0008] S1. Collect multimodal data;

[0009] The multimodal data includes: visible light images, laser point cloud data, and infrared thermal imaging data;

[0010] S2. Preprocess the multimodal data to obtain standardized data;

[0011] The standardized data includes: standardized visible light images, depth maps, and temperature value maps;

[0012] S3. Build an enhanced multi-branch feature extraction network to extract features from standardized data and obtain multimodal features;

[0013] The multimodal features include: shallow features and deep features;

[0014] S4. Multimodal feature fusion is achieved through a three-dimensional attention-guided strategy. An adaptive weighted fusion operation is performed on shallow and deep features to obtain a multimodal fused feature map.

[0015] S5. Input the multimodal fusion feature map into the segmentation decoding network to obtain preliminary disease segmentation results; the segmentation decoding network is generated through training with a hybrid loss function;

[0016] S6. Perform morphological processing and connected component analysis on the preliminary disease segmentation results to obtain the final disease segmentation results.

[0017] Furthermore, the visible light image is acquired using a high-definition industrial camera, with a resolution greater than or equal to 1920×1080 and a frame rate ranging from 10 to 20 frames per second.

[0018] The laser point cloud data density is 500-1000 points per square meter, and the ranging accuracy is ±1 to ±5 millimeters;

[0019] The infrared thermal imaging data has a temperature resolution of 0.01℃ to 0.1℃ and a spatial resolution of ≥320×240.

[0020] Furthermore, S2, preprocessing the multimodal data to obtain standardized data specifically includes the following steps:

[0021] S21. After performing Gaussian filtering, histogram equalization, and perspective transformation on the visible light image in sequence, normalize it to the [0,1] interval to obtain a standardized visible light image.

[0022] S22. Perform statistical filtering and voxelization downsampling on the laser point cloud data in sequence. Use the RANSAC algorithm to fit the cylindrical model and extract the inner wall point cloud data. Project the data to generate a depth map.

[0023] S23. Perform median filtering and temperature calibration on the infrared thermal imaging data in sequence to obtain a temperature value map. Normalize the temperature value map to obtain the temperature difference.

[0024] Furthermore, the enhanced multi-branch feature extraction network includes: a visible light feature extraction branch, a laser point cloud depth feature extraction branch, and an infrared thermal feature extraction branch;

[0025] The backbone network of the branch networks of the enhanced multi-branch feature extraction network adopts the EfficientNet-B3 structure.

[0026] The last residual block of the branch network of the enhanced multi-branch feature extraction network is embedded in the DynamicASPP module;

[0027] The shallow features include: texture features, edge features, and temperature features;

[0028] The deep features include: semantic features and three-dimensional structural features;

[0029] The visible light feature extraction branch extracts texture features and semantic features from the standardized visible light image; the texture features include: crack edges and corrosion spots; the semantic features include: the type and location of the disease;

[0030] The laser point cloud depth feature extraction branch extracts three-dimensional structural features from the depth map; the three-dimensional structural features include: the depth of the depression in the damaged area and the width of the crack.

[0031] The infrared thermal feature extraction branch extracts temperature features from the temperature value map; the temperature features include: higher temperature in the corrosion area and abnormal temperature in the crack area.

[0032] Furthermore, the three-dimensional attention guidance strategy specifically refers to a modal-channel-spatial three-dimensional attention guidance strategy;

[0033] S4, achieving multimodal feature fusion through a three-dimensional attention-guided strategy, and performing adaptive weighted fusion of shallow and deep features to obtain a multimodal fused feature map, specifically includes the following steps:

[0034] S41. Learn the dynamic weights of each modality in the multimodal features through a fully connected layer to obtain the modal weights;

[0035] S42. Perform average pooling on shallow features and global max pooling on deep features, and learn channel attention weights through a multi-layer perceptron mechanism.

[0036] S43. Perform weighted processing on the multimodal features according to the channel attention weights to obtain the multimodal features after channel weighted processing;

[0037] S44. Based on the modal weights, the multimodal features after channel weighting are concatenated along the channel dimension to obtain the feature map;

[0038] S45. Generate a spatial attention weight map through convolution operation, and enhance the feature map with spatial dimension weighting through the spatial attention weight map to obtain a spatially weighted enhanced feature map.

[0039] The spatially weighted enhanced feature map includes: a shallow fusion feature map and a deep fusion feature map;

[0040] S46. By using gated cross-attention, the shallow fusion feature map and the deep fusion feature map are fused to obtain a multimodal fusion feature map.

[0041] Furthermore, the segmentation and decoding network supports an improved U-Net++;

[0042] The improved U-Net++ includes: embedding adaptive gated skip connections, replacing convolutional layers with deformable convolutional v2, and adding a multi-scale output fusion mechanism;

[0043] The embedded adaptive gated skip connection specifically refers to: when the encoded features are input into the decoding stage, the validity of the features is determined by a gated unit;

[0044] The replacement of convolutional layers with deformable convolution v2 specifically refers to replacing the last two convolutional layers of the segmentation decoding network with deformable convolution v2.

[0045] The aforementioned multi-scale output fusion mechanism specifically refers to adding auxiliary output heads at the 1 / 2, 1 / 4, and 1 / 8 resolution stages of the segmentation decoding network.

[0046] Furthermore, the hybrid loss function includes: Dice loss, IoU loss, class balancing FocalLoss, boundary-aware loss, and contrast loss;

[0047] The expression for the hybrid loss function is:

[0048]

[0049] in, For dynamic weights, For Dice's loss, For IoU loss, FocalLoss for category balancing For boundary loss weights, For boundary-aware loss, To compare loss weights, To compare the losses.

[0050] Furthermore, the expression for the category balance FocalLoss is:

[0051]

[0052] in, Indicates category balance factor, To predict the probability of class c for the model, =2;

[0053] The expression for boundary-aware loss is:

[0054]

[0055] in, To predict the boundary pixel set, For the true boundary pixel set, x is the number of elements in the pixel set, and x is The individual pixel coordinates, y is The single pixel coordinates in This is the minimum squared value of the Euclidean distance.

[0056] Furthermore, the expression for the contrast loss is:

[0057]

[0058] in, For sample features, For the same type of positive sample features, Features of outlier negative samples For inter-class separation threshold, Let max(·,0) be the positive part function and N be the number of samples.

[0059] Furthermore, step S6, performing morphological processing and connected component analysis on the preliminary disease segmentation results to obtain the final disease segmentation results, specifically includes the following steps:

[0060] S61. The preliminary disease segmentation results are subjected to corrosion treatment, and the preliminary disease segmentation results after corrosion treatment are subjected to expansion treatment to obtain the disease segmentation results after morphological treatment.

[0061] S62. Calculate the area, perimeter, and rectangularity of connected components in the morphologically processed disease segmentation results. Set area and rectangularity thresholds, remove false connected components with areas smaller than the area thresholds, and obtain the final disease segmentation results.

[0062] The beneficial effects of this invention are as follows:

[0063] 1. This invention integrates several multimodal data, including visible light images, laser point clouds, and infrared thermal imaging, leveraging the complementary effects of different modal data. Visible light images can present the texture features of diseases, laser point clouds can provide three-dimensional structural features, and infrared thermal imaging can reflect temperature difference features. This solves the problem of insufficient feature representation of single-modal data under complex working conditions, and improves the accuracy and stability of disease segmentation.

[0064] 2. This invention employs a multimodal feature fusion method based on an attention mechanism. Specifically, through channel attention and spatial attention mechanisms, the model can adaptively determine the importance weights of different modal features. This helps to highlight features that play a key role in disease segmentation while reducing interference from invalid features and noise. This method achieves deep fusion of multimodal features and overcomes the problems of feature redundancy and insufficient fusion in traditional fusion methods.

[0065] 3. This invention uses an improved hybrid loss function that combines the advantages of Dice loss, IoU loss and FocalLoss. This method effectively addresses the problem of imbalanced disease data samples, improves the segmentation accuracy of small target diseases and difficult-to-distinguish disease areas, and makes the segmentation boundaries more precise.

[0066] 4. The method of the present invention can be used for the segmentation of defects in drainage pipes of different diameters and materials; after segmentation, the defects can be quantitatively analyzed, such as determining the crack width, corrosion area, and damage depth; these analysis results can provide accurate data for drainage pipe maintenance decisions and have great potential for engineering applications. Attached Figure Description

[0067] Figure 1 This is a step diagram of a drainage pipeline defect segmentation method based on multimodal data fusion provided in an embodiment of the present invention. Detailed Implementation

[0068] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0069] like Figure 1 As shown, the present invention provides a method for segmenting drainage pipeline defects based on multimodal data fusion, which includes the following steps:

[0070] S1. Collect multimodal data;

[0071] The multimodal data includes: visible light images, laser point cloud data, and infrared thermal imaging data;

[0072] Specifically, visible light images of the pipe's inner wall are captured using a high-definition industrial camera to reflect the texture and color characteristics of the defects; laser point cloud data of the pipe's inner wall is acquired using 2D LiDAR to present the pipe's three-dimensional structural features and the depth information of the defects; and infrared thermal imaging data of the pipe's inner wall is recorded using an infrared thermal imager to show the temperature difference between the defective and normal areas.

[0073] The visible light images are acquired using a high-definition industrial camera, with a resolution greater than or equal to 1920×1080 and a frame rate ranging from 10 to 20 frames per second.

[0074] The laser point cloud data density is 500-1000 points per square meter, and the ranging accuracy is ±1 to ±5 millimeters;

[0075] The infrared thermal imaging data has a temperature resolution of 0.01℃ to 0.1℃ and a spatial resolution of ≥320×240.

[0076] S2. Preprocess the multimodal data to obtain standardized data;

[0077] The collected multimodal data has problems such as noise, distortion and redundancy, so it is necessary to preprocess the data of each modality separately.

[0078] The standardized data includes: standardized visible light images, depth maps, and temperature value maps;

[0079] S2, preprocessing the multimodal data to obtain standardized data, specifically includes the following steps:

[0080] S21. After performing Gaussian filtering, histogram equalization, and perspective transformation on the visible light image in sequence, normalize it to the [0,1] interval to obtain a standardized visible light image.

[0081] Among them, Gaussian filtering is used to remove noise from visible light images, histogram equalization makes the contrast of visible light images more obvious, and perspective correction corrects the distortion of visible light images.

[0082] S22. Perform statistical filtering and voxelization downsampling on the laser point cloud data in sequence. Use the RANSAC algorithm to fit the cylindrical model and extract the inner wall point cloud data. Project the data to generate a depth map.

[0083] Statistical filtering specifically refers to: calculating the distance distribution of points in the neighborhood of each point and removing outliers whose distance exceeds a set threshold; and removing noise generated when LiDAR collects laser point cloud data through statistical filtering.

[0084] Voxel downsampling specifically refers to dividing point cloud data into a three-dimensional voxel grid, with only one representative point retained in each grid. By using voxel downsampling, the amount of point cloud data is reduced, thereby improving the speed of subsequent processing.

[0085] The cylindrical model was fitted using the RANSAC algorithm, and the point cloud data of the inner wall was extracted. The projection was then used to generate a depth map, which can reflect the concave and convex structural features of the inner wall of the pipe.

[0086] S23. Perform median filtering and temperature calibration on the infrared thermal imaging data in sequence to obtain a temperature value map. Normalize the temperature value map to obtain the temperature difference.

[0087] Among them, median filtering removes salt-and-pepper noise from infrared thermal imaging data;

[0088] Temperature calibration processing specifically refers to: conducting temperature calibration experiments, determining the correspondence between the grayscale values ​​of infrared thermal imaging data and the actual temperature values, and converting the grayscale image into a temperature value map;

[0089] Normalizing the temperature map specifically refers to: calculating the average temperature and variance of the normal area of ​​the pipeline based on the temperature map, and normalizing the temperature image to highlight the temperature difference between the damaged area and the normal area.

[0090] In this embodiment, a self-developed pipeline inspection robot was used to inspect DN800 concrete drainage pipes in a city. The robot is equipped with a high-definition industrial camera with a resolution of 2560×1440 and a frame rate of 15fps. It is also equipped with a 2D lidar with a ranging range of 0.1-50m and a point cloud density of 800 points / square meter. In addition, it is also equipped with an infrared thermal imager with a temperature resolution of 0.05℃ and a spatial resolution of 640×512.

[0091] In visible light image preprocessing, a 5×5 Gaussian filter kernel is used to denoise the image. Then, histogram equalization is used to improve the image contrast. Perspective transformation is then used to correct the image distortion. Finally, the image data is normalized to the range [0,1].

[0092] In the laser point cloud preprocessing stage, a statistical filtering method was used, setting the number of neighboring points to 50 and the distance threshold to 1.5 times the standard deviation to remove outliers. Next, voxel downsampling was employed, with voxel sizes set to 0.01 m × 0.01 m × 0.01 m. Then, the RANSAC algorithm was used to fit a cylindrical model, setting the cylinder radius between 0.39 m and 0.41 m, thereby extracting the point cloud data of the pipe's inner wall, which was then further projected to generate a depth map.

[0093] During infrared thermal imaging preprocessing, a 3×3 median filter is used to eliminate noise. A temperature calibration experiment is conducted to obtain the correspondence between grayscale values ​​and temperature values, using the formula T = 0.02G + 20, where T represents the temperature value and G represents the grayscale value, thus constructing a temperature map. Finally, the temperature map is normalized to obtain the temperature differences.

[0094] S3. Build an enhanced multi-branch feature extraction network to extract features from standardized data and obtain multimodal features;

[0095] The enhanced multi-branch feature extraction network includes: a visible light feature extraction branch, a laser point cloud depth feature extraction branch, and an infrared thermal feature extraction branch;

[0096] The backbone network of the branch networks of the enhanced multi-branch feature extraction network adopts the EfficientNet-B3 structure.

[0097] The last residual block of the branch network of the enhanced multi-branch feature extraction network is embedded in the DynamicASPP module;

[0098] Among them, DynamicASPP is the Dynamic Hole Space Pyramid Pooling Module;

[0099] The multimodal features include: shallow features and deep features;

[0100] The shallow features include: texture features, edge features, and temperature features;

[0101] The deep features include: semantic features and three-dimensional structural features;

[0102] The visible light feature extraction branch extracts texture features and semantic features from the standardized visible light image; the texture features include: crack edges and corrosion spots; the semantic features include: the type and location of the disease;

[0103] The laser point cloud depth feature extraction branch extracts three-dimensional structural features from the depth map; the three-dimensional structural features include: the depth of the depression in the damaged area and the width of the crack.

[0104] The infrared thermal feature extraction branch extracts temperature features from the temperature value map; the temperature features include: higher temperature in the corrosion area and abnormal temperature in the crack area;

[0105] The backbone network adopts EfficientNet-B3 to improve feature extraction efficiency and modality adaptability; DynamicASPP modules are embedded at the end of each branch to dynamically adjust the hole rate, realize full-scale feature capture, and effectively extract shallow detail features and deep semantic features of each modality.

[0106] The visible light feature extraction branch works as follows: a standardized visible light image, i.e., a pre-processed visible light image, is input into this branch. The image will then pass through a convolutional layer, a pooling layer, a residual module, and a DynamicASPP module. Through these processes, shallow texture features of the image, such as crack edges and corrosion spots, can be extracted. Deep semantic features, such as the type and location of the disease, can also be extracted.

[0107] The process for the laser point cloud depth feature extraction branch is as follows: input the depth map into this branch, and extract the three-dimensional structural features of the pipe inner wall, such as how deep the depression in the damaged area is and how wide the crack is.

[0108] The process for the infrared thermal feature branch is as follows: input the temperature value map into this branch, and extract the temperature features of the diseased area, such as the high temperature in the corrosion area and the abnormal temperature in the crack area.

[0109] In this embodiment, the enhanced multi-branch feature extraction network is based on EfficientNet-B3, with dilated convolutions of dilation rate 2 added to the residual module; the hidden layer dimension of the multilayer perceptron in the channel attention submodule is set to 256; and the spatial attention submodule uses 1×1 convolutions to generate the weight map.

[0110] The shallow feature size extracted from the visible light image branch is 128×128×64, and the deep feature size is 16×16×2048; the same applies to the depth map branch, where the shallow feature size is 128×128×64 and the deep feature size is 16×16×2048; the shallow feature size extracted from the infrared temperature map branch is also 128×128×64, and the deep feature size is also 16×16×2048.

[0111] S4. Multimodal feature fusion is achieved through a three-dimensional attention-guided strategy. An adaptive weighted fusion operation is performed on shallow and deep features to obtain a multimodal fused feature map.

[0112] The three-dimensional attention guidance strategy specifically refers to: a modal-channel-spatial three-dimensional attention guidance strategy;

[0113] S4, achieving multimodal feature fusion through a three-dimensional attention-guided strategy, and performing adaptive weighted fusion of shallow and deep features to obtain a multimodal fused feature map, specifically includes the following steps:

[0114] S41. Learn the dynamic weights of each modality in the multimodal features through a fully connected layer to obtain the modal weights;

[0115] Specifically, the expression for learning the dynamic weights of each modality in multimodal features through fully connected layers is as follows:

[0116]

[0117] in, The global feature statistics for the m-th mode are composed of "feature map mean + feature map variance", which comprehensively reflects the data quality and effective information content of this mode. Among them, m=1 is visible light, m=2 is laser point cloud, and m=3 is infrared. , The weight matrix for the fully connected layer is set to 256, and adaptive learning is performed during training. This is for batch normalization operations to avoid gradient vanishing; Use the sigmoid activation function to output mode weights. The value range is [0,1], and it satisfies the formula. ; The modal weights are used to adapt the effectiveness of different modes under complex working conditions in real time.

[0118] The specific adaptation logic is as follows: When there is insufficient light, the infrared mode weights are adjusted. Automatically increased, up to 0.45; when obscured by dirt, the modal weights of the laser point cloud are adjusted. Automatically boosted, reaching a maximum weight of 0.42; under normal, unobstructed lighting conditions, the visible light mode weight is dominant. Approximately 0.36, enabling adaptive mode selection under complex operating conditions.

[0119] S42. Perform average pooling on shallow features and global max pooling on deep features, and learn channel attention weights through a multi-layer perceptron mechanism.

[0120] By performing average pooling on shallow features and global max pooling on deep features, channel statistics for each feature map can be obtained.

[0121] S43. Perform weighted processing on the multimodal features according to the channel attention weights to obtain the multimodal features after channel weighted processing;

[0122] Channel weighting enhances useful channel features for disease segmentation and weakens ineffective channel features.

[0123] S44. Based on the modal weights, the multimodal features after channel weighting are concatenated along the channel dimension to obtain the feature map;

[0124] S45. Generate a spatial attention weight map through convolution operation, and enhance the feature map with spatial dimension weighting through the spatial attention weight map to obtain a spatially weighted enhanced feature map.

[0125] A spatial attention weight map is generated through convolution operations. The value of each pixel in the weight map represents the importance of the feature at the corresponding location. This spatial attention weight map is used to enhance the spliced ​​feature map in the spatial dimension, highlighting the features of the diseased area and reducing the interference of the background area.

[0126] The spatially weighted enhanced feature map includes: a shallow fusion feature map and a deep fusion feature map;

[0127] S46. By using gated cross-attention, the shallow fusion feature map and the deep fusion feature map are fused to obtain a multimodal fusion feature map.

[0128] The fusion of shallow and deep fusion feature maps using gated cross-attention specifically involves: first, upsampling the shallow fusion feature map to 16×16 using bilinear interpolation, making it the same size as the deep fusion feature map; then, using a gating function to determine the validity of details such as "edges and textures" in the shallow features, and only fusing the valid details with the deep semantic features. The final fused feature map has a size of 16×16×(6144+6144)=16×16×4096, and includes both detailed texture and semantic structure information, thus solving the problem of "shallow noise interfering with deep features" in the original fusion method.

[0129] Specifically, the expression for fusing shallow and deep fusion feature maps using gated cross-attention is as follows:

[0130]

[0131] The image shows a shallow fusion feature map of multimodal data after modality-channel-spatial weighting, measuring 128×128×192. The gate function is activated by sigmoid, which adaptively selects effective features; This is a multimodal deep fusion feature map after modality-channel-spatial weighting, 16×16×6144; This indicates element-wise multiplication.

[0132] In this embodiment, the weights learned by the channel attention module are used to weight the different modal feature channels. Specifically, the average weight of the visible light feature channel is 0.35, the average weight of the depth feature channel is 0.32, and the average weight of the infrared feature channel is 0.33. At the same time, the feature response of the diseased area is improved by more than 30% through the spatial attention weight map. Then, the shallow fusion feature map is upsampled to a size of 16×16 and then stitched with the deep fusion feature map to finally obtain a multimodal fusion feature map with a size of 16×16×4096.

[0133] S5. Input the multimodal fusion feature map into the segmentation decoding network to obtain preliminary disease segmentation results; the segmentation decoding network is generated through training with a hybrid loss function;

[0134] The segmentation and decoding network supports an improved U-Net++;

[0135] The improved U-Net++ includes: embedding adaptive gated skip connections, replacing convolutional layers with deformable convolutional v2, and adding a multi-scale output fusion mechanism;

[0136] Among them, the deformable convolution v2 is DCNv2;

[0137] The improved U-Net++ gradually increases the spatial resolution of the feature map through upsampling operations, and at the same time uses skip connections to fuse the shallow features extracted in the encoding stage and the features in the decoding stage, thereby improving the accuracy of the segmentation boundary.

[0138] The embedded adaptive gated skip connection specifically refers to: when the encoded features are input into the decoding stage, the validity of the features is determined by a gated unit;

[0139] The replacement of convolutional layers with deformable convolution v2 specifically refers to replacing the last two convolutional layers of the segmentation decoding network with deformable convolution v2.

[0140] The aforementioned multi-scale output fusion mechanism specifically refers to adding auxiliary output heads at the 1 / 2, 1 / 4, and 1 / 8 resolution stages of the segmentation decoding network.

[0141] The hybrid loss functions include: Dice loss, IoU loss, class-balanced FocalLoss, boundary-aware loss, and contrastive loss;

[0142] The expression for the hybrid loss function is:

[0143]

[0144] in, For dynamic weights, For Dice's loss, For IoU loss, FocalLoss for category balancing For boundary loss weights, For boundary-aware loss, To compare loss weights, To compare the losses.

[0145] Improvement in the early stages of training Weights, improved in the later stages of training Weight;

[0146] , The main loss component has no fixed value and changes dynamically with the prediction results.

[0147] In this embodiment, the hyperparameter is set as: boundary loss weight. Ensure that the main loss accounts for ≥70% and avoid interference from auxiliary losses.

[0148] The expression for the category balance FocalLoss is:

[0149]

[0150] in, Indicates category balance factor, To predict the probability of class c for the model, =2;

[0151] set up =2 Maintains the ability to focus on difficult-to-differentiate samples;

[0152] Category balance factor Where N is the total number of samples, K is the number of categories, and Nc is the number of samples in the c-th category;

[0153] FocalLoss, a category-balanced approach, addresses the underfitting problem of a minority of disease categories, such as cracks, corrosion, damage, and interface detachment, based on the varying sample sizes. For example, if the proportion of interface detachment is less than 5%, a category-balanced factor is introduced to assign higher weights to these minority categories. This improves the mIoU of interface detachment-related diseases by 10%.

[0154] The expression for boundary-aware loss is:

[0155]

[0156] in, To predict the boundary pixel set, For the true boundary pixel set, x is the number of elements in the pixel set, and x is The individual pixel coordinates, y is The single pixel coordinates in This is the minimum squared value of the Euclidean distance.

[0157] Boundary-aware loss directly optimizes the completeness and accuracy of segmentation boundaries, and uses chamfer distance to calculate the difference between the predicted boundary and the real boundary. By forcing the model to focus on boundary details through boundary-aware loss, the average deviation between the segmented boundary and the real boundary is reduced from 2.3 pixels to 1.1 pixels.

[0158] The expression for the contrast loss is:

[0159]

[0160] in, For sample features, For the same type of positive sample features, Features of outlier negative samples For inter-class separation threshold, Let max(·,0) be the positive part function and N be the number of samples.

[0161] In this embodiment, the margin is set to 0.5.

[0162] The contrast loss improves the feature differentiation between disease categories, avoiding confusion between "corrosion and stains" and "micro-cracks and pipe textures"; the loss is calculated only when the intra-class distance is greater than the inter-class distance plus margin. Ultimately, this results in more compact features for similar diseases and better separation of dissimilar features, increasing the diagonal proportion of the category confusion matrix by more than 5%.

[0163] S6. Perform morphological processing and connected component analysis on the preliminary disease segmentation results to obtain the final disease segmentation results.

[0164] The internal environment of pipelines is complex, and the initial defect segmentation results may contain problems such as false segmentation areas and small-area noise. Therefore, post-processing is required for optimization.

[0165] S6, performing morphological processing and connected component analysis on the preliminary disease segmentation results to obtain the final disease segmentation results, specifically includes the following steps:

[0166] S61. The preliminary disease segmentation results are subjected to corrosion treatment, and the preliminary disease segmentation results after corrosion treatment are subjected to expansion treatment to obtain the disease segmentation results after morphological treatment.

[0167] The initial disease segmentation results are subjected to corrosion treatment to remove small areas of noise; the corrosion-treated initial disease segmentation results are then subjected to expansion treatment to restore the shape of the diseased area to its completeness and prevent the corrosion treatment from reducing the size of the diseased area.

[0168] S62. Calculate the area, perimeter, and rectangularity of connected components in the morphologically processed disease segmentation results. Set area and rectangularity thresholds, remove false connected components with areas smaller than the area thresholds, and obtain the final disease segmentation results.

[0169] In this embodiment, the area threshold is set to 50 pixels and the rectangularity threshold is set to 0.3; false connected components with an area smaller than the threshold are removed, leaving only the defective connected components that meet the conditions.

[0170] The specific methods for training and validating the model in this invention include:

[0171] Multimodal data of drainage pipes of different diameters, including those made of concrete, cast iron, and PVC, were collected. The data covered common types of drainage pipe defects, such as cracks, corrosion, damage, and joint detachment. Subsequently, professionals performed precise, pixel-level manual annotation of the defective areas, ultimately constructing a multimodal dataset of drainage pipe defects.

[0172] The dataset is divided into training, validation, and test sets in a 7:2:1 ratio. The training set is used to allow the network to learn parameters, the validation set is used to tune the network's hyperparameters, and the test set is used to evaluate the model's final segmentation performance.

[0173] Stochastic gradient descent was selected as the optimizer, with an initial learning rate of 0.001. Cosine annealing decay strategy was used to adjust the learning rate. During training, 8 samples were processed in each batch, and a total of 100 training rounds were conducted. To prevent overfitting, an early stopping strategy was applied during training, i.e., training was stopped and the best-performing model was saved when the mean intersection-union ratio of the validation set did not improve for 10 consecutive rounds.

[0174] After the model training was completed, validation was carried out on the test set. The average intersection-union ratio, Dice coefficient, and pixel accuracy were selected as indicators to evaluate the model performance. The validation results showed that the segmentation effect of the method of the present invention was better than that of the traditional single-modal segmentation method.

[0175] Specifically, the core improvements of this invention are all designed specifically for the unique characteristics of pipeline network defect segmentation, while ordinary defect segmentation does not require such complex designs. A detailed comparison between pipeline network defect segmentation and ordinary defect segmentation includes: pipeline network defect segmentation detects defects attached to cylindrical curved surfaces; while ordinary defect segmentation detects defects on planar / simple curved surfaces, such as skin lesions, road cracks, and scratches on metal parts. The scenarios for pipeline network defect segmentation are characterized by being enclosed, narrow, with uncontrollable lighting, no natural light sources, diverse pipe materials such as concrete, cast iron, and PVC, and large differences in pipe diameter; while the scenarios for ordinary defect segmentation are mostly open or controllable environments, such as roads and laboratories, with relatively simple structures, mostly planar, such as medical imaging equipment. The core objective of pipeline network defect segmentation is not only pixel-level segmentation but also support for quantitative analysis, such as crack width, corrosion area, and damage depth, to provide data support for pipeline maintenance; while the core objective of ordinary defect segmentation is mainly category segmentation, with some scenarios not requiring quantification, such as medical imaging which only needs to locate the lesion area. Therefore, the pipeline network scenario is characterized by "dynamic complex curved surfaces + closed and uncontrollable environment + quantitative requirements", while the ordinary scenario is characterized by "static simple structure + controllable environment + classification / location requirements".

[0176] Due to the differences in application scenarios and detection objects between pipeline network defect segmentation and ordinary defect segmentation, the "multimodal dependence" of pipeline network defects stems from the inability of a single modality to cope with complex surfaces and environmental interference, while the "single-modal sufficiency" of ordinary defects arises from the simplicity of the scenario and features. The "coexistence of multiple sources of interference" in pipeline networks leads to much higher robustness requirements than in ordinary scenarios, necessitating the use of attention mechanisms and multimodal fusion to offset interference. The "high annotation cost + imbalanced sample + diversity requirements" of pipeline network datasets stem from the complexity of the scenario, while the "low cost + balanced distribution" of ordinary datasets stems from the simplicity of the scenario.

[0177] The unique design features provided by this invention include: Multimodal fusion + 3D attention: addressing the "multi-source interference + modal complementarity requirements" of pipeline networks, which can be met by ordinary segmentation using a single modality; DynamicASPP + DCNv2: adapting to the "large scale differences + irregular shape" of pipeline network defects, where ordinary segmentation results in uniform defect scale / shape, eliminating the need for dynamic receptive fields and deformation convolution; Dynamic weighted hybrid loss: solving the "imbalanced sample + high boundary accuracy requirements + easy feature confusion" of pipeline networks, where ordinary segmentation does not need to consider all three simultaneously; Lightweight backbone network: meeting the needs of embedded deployment, where ordinary segmentation has no hardware constraints and can use larger models.

[0178] In summary, the pipeline defect segmentation method proposed in this invention is a comprehensive challenge of "complex environment + complex features + strong engineering requirements", while ordinary defect segmentation is a routine task of "simple environment + simple features + single requirements". The technical difference between the two is essentially the difference in scenario complexity and engineering requirements.

[0179] In this embodiment, a multimodal disease dataset for drainage pipes was constructed, containing 10,000 sets of multimodal data. Specifically, the dataset is divided as follows: 7,000 sets for training, 2,000 sets for validation, and 1,000 sets for testing. During model training, the SGD optimizer was selected, with an initial learning rate of 0.001, a cosine annealing decay period of 10 epochs, a batch size of 8, and a total of 100 training epochs. An early stopping strategy was employed, with a waiting period of 10 epochs.

[0180] This embodiment validated the model on the test set and compared it with the traditional single-modal U-Net segmentation model and the simple feature splicing and fusion model. The comparison results of pipe misalignment defect segmentation are shown in Table 1:

[0181]

[0182] Table 1

[0183] The table shows the comparison results of pipe misalignment defect segmentation. In addition, this embodiment also conducted experiments on three common defects in pipes: cracks, corrosion, and damage, obtaining corresponding segmentation results. Experiments show that the method of this invention achieves a segmentation mIoU of 0.896 for cracks, 0.912 for corrosion areas, and 0.938 for damaged areas; these indicators are all higher than the comparison model. Furthermore, the segmentation boundaries of this invention are clearer, and there are fewer false segmentation areas.

[0184] The drainage pipeline defect segmentation method proposed in this invention is based on multimodal data fusion and can be installed in the embedded system of a pipeline inspection robot to achieve real-time automatic segmentation and quantitative analysis of drainage pipeline defects. This method is applicable to the inspection of drainage pipelines of different diameters and materials, and can operate stably even under complex conditions such as uneven lighting and dirt obstruction. It provides an efficient and accurate technical solution for the intelligent inspection and maintenance of urban drainage pipe networks, and has great potential in industrial applications and the market.

[0185] This invention integrates multimodal data from visible light images, laser point clouds, and infrared thermal imaging, leveraging the complementary effects of different modalities. Visible light images reveal the texture features of disease lesions, laser point clouds provide three-dimensional structural features, and infrared thermal imaging reflects temperature differences. This addresses the problem of insufficient feature representation by single-modal data under complex conditions, improving the accuracy and stability of disease segmentation. The invention employs a multimodal feature fusion method based on an attention mechanism. Specifically, through channel attention and spatial attention mechanisms, the model adaptively determines the importance weights of different modal features. This helps highlight features crucial for disease segmentation while reducing invalid features and noise interference. This method achieves deep fusion of multimodal features, overcoming the problems of feature redundancy and insufficient fusion in traditional fusion methods. The invention utilizes an improved hybrid loss function, combining the advantages of Dice loss, IoU loss, and FocalLoss. This method effectively addresses the problem of imbalanced disease data samples, improving the segmentation accuracy of small-target diseases and difficult-to-distinguish disease areas, and also making the segmentation boundaries more precise. The method of this invention can be used to segment drainage pipes of different diameters and materials for defects. After segmentation, the defects can be quantitatively analyzed, such as determining crack width, corrosion area, and damage depth. These analysis results can provide accurate data for drainage pipe maintenance decisions and have great potential for engineering applications.

[0186] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for segmenting drainage pipeline defects based on multimodal data fusion, characterized in that, Includes the following steps: S1. Collect multimodal data; The multimodal data includes: visible light images, laser point cloud data, and infrared thermal imaging data; S2. Preprocess the multimodal data to obtain standardized data; The standardized data includes: standardized visible light images, depth maps, and temperature value maps; S3. Build an enhanced multi-branch feature extraction network to extract features from standardized data and obtain multimodal features; The multimodal features include: shallow features and deep features; The enhanced multi-branch feature extraction network includes: a visible light feature extraction branch, a laser point cloud depth feature extraction branch, and an infrared thermal feature extraction branch; The backbone network of the branch networks of the enhanced multi-branch feature extraction network adopts the EfficientNet-B3 structure. The last residual block of the branch network of the enhanced multi-branch feature extraction network is embedded in the DynamicASPP module; The shallow features include: texture features, edge features, and temperature features; The deep features include: semantic features and three-dimensional structural features; The visible light feature extraction branch extracts texture features and semantic features from the standardized visible light image; the texture features include: crack edges and corrosion spots; the semantic features include: the type and location of the disease; The laser point cloud depth feature extraction branch extracts three-dimensional structural features from the depth map; the three-dimensional structural features include: the depth of the depression in the damaged area and the width of the crack. The infrared thermal feature extraction branch extracts temperature features from the temperature value map; the temperature features include: higher temperature in the corrosion area and abnormal temperature in the crack area; S4. Multimodal feature fusion is achieved through a three-dimensional attention-guided strategy. An adaptive weighted fusion operation is performed on shallow and deep features to obtain a multimodal fused feature map. The three-dimensional attention guidance strategy specifically refers to: a modal-channel-spatial three-dimensional attention guidance strategy; S4, achieving multimodal feature fusion through a three-dimensional attention-guided strategy, and performing adaptive weighted fusion of shallow and deep features to obtain a multimodal fused feature map, specifically includes the following steps: S41. Learn the dynamic weights of each modality in the multimodal features through a fully connected layer to obtain the modal weights; S42. Perform average pooling on shallow features and global max pooling on deep features, and learn channel attention weights through a multi-layer perceptron mechanism. S43. Perform weighted processing on the multimodal features according to the channel attention weights to obtain the multimodal features after channel weighted processing; S44. Based on the modal weights, the multimodal features after channel weighting are concatenated along the channel dimension to obtain the feature map; S45. Generate a spatial attention weight map through convolution operation, and enhance the feature map with spatial dimension weighting through the spatial attention weight map to obtain a spatially weighted enhanced feature map. The spatially weighted enhanced feature map includes: a shallow fusion feature map and a deep fusion feature map; S46. By fusing the shallow fusion feature map and the deep fusion feature map through gated cross attention, a multimodal fusion feature map is obtained. S5. Input the multimodal fusion feature map into the segmentation decoding network to obtain preliminary disease segmentation results; the segmentation decoding network is generated through training with a hybrid loss function; The hybrid loss functions include: Dice loss, IoU loss, class-balanced FocalLoss, boundary-aware loss, and contrastive loss; The expression for the hybrid loss function is: in, For dynamic weights, For Dice's loss, For IoU loss, FocalLoss for category balancing For boundary loss weights, For boundary-aware loss, To compare loss weights, To compare the losses; S6. Perform morphological processing and connected component analysis on the preliminary disease segmentation results to obtain the final disease segmentation results.

2. The method for segmenting drainage pipe defects according to claim 1, characterized in that, The visible light images are acquired using a high-definition industrial camera, with a resolution greater than or equal to 1920×1080 and a frame rate ranging from 10 to 20 frames per second. The laser point cloud data density is 500-1000 points per square meter, and the ranging accuracy is ±1 to ±5 millimeters; The infrared thermal imaging data has a temperature resolution of 0.01℃ to 0.1℃ and a spatial resolution of ≥320×240.

3. The method for segmenting drainage pipe defects according to claim 1, characterized in that, S2, preprocessing the multimodal data to obtain standardized data, specifically includes the following steps: S21. After performing Gaussian filtering, histogram equalization, and perspective transformation on the visible light image in sequence, normalize it to the [0,1] interval to obtain a standardized visible light image. S22. Perform statistical filtering and voxelization downsampling on the laser point cloud data in sequence. Use the RANSAC algorithm to fit the cylindrical model and extract the inner wall point cloud data. Project the data to generate a depth map. S23. Perform median filtering and temperature calibration on the infrared thermal imaging data in sequence to obtain a temperature value map. Normalize the temperature value map to obtain the temperature difference.

4. The method for segmenting drainage pipe defects according to claim 1, characterized in that, The segmentation and decoding network supports an improved U-Net++; The improved U-Net++ includes: embedding adaptive gated skip connections, replacing convolutional layers with deformable convolutional v2, and adding a multi-scale output fusion mechanism; The embedded adaptive gated skip connection specifically refers to: when the encoded features are input into the decoding stage, the validity of the features is determined by a gated unit; The replacement of convolutional layers with deformable convolution v2 specifically refers to replacing the last two convolutional layers of the segmentation decoding network with deformable convolution v2. The aforementioned multi-scale output fusion mechanism specifically refers to adding auxiliary output heads at the 1 / 2, 1 / 4, and 1 / 8 resolution stages of the segmentation decoding network.

5. The method for segmenting drainage pipe defects according to claim 1, characterized in that, The expression for the category balance FocalLoss is: in, Indicates category balance factor, To predict the probability of class c for the model, =2; The expression for boundary-aware loss is: in, To predict the boundary pixel set, For the true boundary pixel set, x is the number of elements in the pixel set, and x is The individual pixel coordinates, y is The single pixel coordinates in This is the minimum squared value of the Euclidean distance.

6. The method for segmenting drainage pipe defects according to claim 1, characterized in that, The expression for the contrast loss is: in, For sample features, For the same type of positive sample features, Features of outlier negative samples For inter-class separation threshold, Let max(·,0) be the positive part function and N be the number of samples.

7. The method for segmenting drainage pipe defects according to claim 1, characterized in that, S6, performing morphological processing and connected component analysis on the preliminary disease segmentation results to obtain the final disease segmentation results, specifically includes the following steps: S61. The preliminary disease segmentation results are subjected to corrosion treatment, and the preliminary disease segmentation results after corrosion treatment are subjected to expansion treatment to obtain the disease segmentation results after morphological treatment. S62. Calculate the area, perimeter, and rectangularity of connected components in the morphologically processed disease segmentation results. Set area and rectangularity thresholds, remove false connected components with areas smaller than the area thresholds, and obtain the final disease segmentation results.

Images