Sewer pipe defect detection method and device based on multi-scale image feature fusion

Abstract

The application discloses a sewage pipeline defect detection method and device based on multi-scale image feature fusion, and comprises the following steps: S1, acquiring historical sewage pipeline defect image data sets; S2, pre-processing and data expansion processing are performed on the historical pipeline defect image data sets; S3, an improved YOLOv8 model is trained according to the historical sewage pipeline defect image data sets after the pre-processing and data expansion processing, and a lightweight pipeline defect detection model is obtained; and S4, real-time collected sewage pipeline internal images are input into the lightweight pipeline defect detection model for pipeline defect detection. According to the technical scheme, different scale features can be efficiently extracted, and details of multi-scale targets can be better captured, so that the ability of small target detection and complex background processing is improved.

Description

Technical Field

[0001] This invention belongs to the field of pipeline inspection technology, and in particular relates to a method and device for detecting defects in sewage pipelines based on multi-scale image feature fusion. Background Technology

[0002] Sewage pipelines, as a crucial component of urban infrastructure, are primarily responsible for collecting and transporting domestic sewage, industrial wastewater, and rainwater. Their operational status directly impacts urban development, public health, and environmental quality. Therefore, ensuring the efficient operation of sewage pipeline systems is vital for sustainable urban development. With continuous economic growth and accelerated urbanization, my country's urban sewage pipe network systems have made significant progress in construction and development. However, in recent years, sewage pipeline management has generally suffered from a problem of "emphasizing construction while neglecting maintenance," leading to defects such as pipe structural damage, corrosion, and blockages. As pipelines age and sewage volumes continue to increase, these problems become increasingly severe, not only affecting daily life but also potentially causing safety hazards such as road collapses and urban flooding. Therefore, regular inspections of the internal condition of sewage pipelines are particularly important to promptly assess the type and location of defects and take appropriate countermeasures, such as pipeline maintenance, repair, or replacement of severely damaged sections.

[0003] CCTV (Closed-Circuit Television) systems have become a widely used inspection method in the inspection and maintenance of sewage pipelines. Typically, robots equipped with cameras and lighting move inside the pipelines, recording video in real time to assess their structural condition. This technology is widely recognized for its ease of operation and low cost. Through CCTV video, technicians can observe defects on the inner surface of the pipeline, such as cracks, collapses, and deposits. Although CCTV video analysis is widely used, traditional manual interpretation methods are not only time-consuming and labor-intensive but also easily influenced by the operator's experience and subjective judgment, leading to biased results. For example, a crack might be misjudged as a fracture, and the accuracy of manual inspection results is easily affected by the inspector's experience. While technicians can refer to standards such as the Pipeline Assessment and Certification Program (PACP) as supplementary tools, this human-based inspection method is insufficient to meet the demands of efficient and accurate modern inspection. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for detecting defects in sewage pipelines based on multi-scale image feature fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for detecting defects in sewage pipes based on multi-scale image feature fusion, comprising:

[0007] Step S1: Obtain a dataset of historical sewage pipe defect images;

[0008] Step S2: Preprocess and augment the historical pipeline defect image dataset;

[0009] Step S3: Train an improved YOLOv8 model based on the preprocessed and augmented historical sewage pipeline defect image dataset to obtain a lightweight pipeline defect detection model;

[0010] Step S4: Input the real-time acquired images of the inside of the sewage pipe into the lightweight pipe defect detection model to detect pipe defects.

[0011] Preferably, in step S2, the preprocessing includes grayscale conversion, contrast enhancement, and noise reduction, and the data augmentation includes random cropping, horizontal and vertical rotation of the historical sewage pipe defect image dataset.

[0012] As a preferred option, the improved YOLOv8 model includes: the C2f-FAM module, the HS-BiFPN module, and the Upsample module. Among them, the C2f-FAM module introduces EMSConvP multi-scale convolution for efficient extraction of features at different scales; the HS-BiFPN module replaces the original neck of the YOLOv8 model and achieves flexible fusion of feature maps at different scales through cross-level feature fusion operations; and the DySample module upsamples to dynamically optimize the scale of the feature maps according to the size of the target and the complexity of the scene.

[0013] The present invention also provides a sewage pipeline defect detection device based on multi-scale image feature fusion, comprising:

[0014] The acquisition module is used to acquire a dataset of historical sewage pipe defect images;

[0015] The processing module is used to preprocess and augment the historical pipeline defect image dataset.

[0016] The training module is used to train an improved YOLOv8 model based on a dataset of historical sewage pipe defect images after preprocessing and data augmentation, resulting in a lightweight pipe defect detection model.

[0017] The detection module is used to input real-time acquired images of the inside of sewage pipes into a lightweight pipe defect detection model for pipe defect detection.

[0018] Preferably, preprocessing includes grayscale conversion, contrast enhancement, and noise reduction, and data augmentation includes random cropping, horizontal and vertical rotation of the historical sewage pipe defect image dataset.

[0019] As a preferred option, the improved YOLOv8 model includes: the C2f-FAM module, the HS-BiFPN module, and the Upsample module. Among them, the C2f-FAM module introduces EMSConvP multi-scale convolution for efficient extraction of features at different scales; the HS-BiFPN module replaces the original neck of the YOLOv8 model and achieves flexible fusion of feature maps at different scales through cross-level feature fusion operations; and the DySample module upsamples to dynamically optimize the scale of the feature maps according to the size of the target and the complexity of the scene.

[0020] This invention proposes a lightweight pipeline defect detection model based on multi-scale feature fusion, building upon YOLOv8. First, the C2f module in the backbone network is replaced with a C2f-FAM module to enhance multi-scale feature extraction capabilities. Second, HS-BiFPN is used to replace the original neck structure, improving the model's ability to recognize multi-scale targets and rich feature representations. Finally, DySample is introduced to optimize the upsampling operation, improving the model's target capture capability in complex environments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 This is a flowchart of the sewage pipeline defect detection method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the improved YOLOv8 model according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of EMSConvP according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of HS-BiFPN according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of Cross-Level Feature Fusion according to an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of DySample according to an embodiment of the present invention; wherein, (a) is a schematic diagram of the DySample structure; and (b) is a schematic diagram of the static range factor version and the dynamic range factor version. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Example 1:

[0031] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for detecting defects in sewage pipelines based on multi-scale image feature fusion, comprising:

[0032] Step S1: Obtain a dataset of historical sewage pipe defect images;

[0033] Step S2: Preprocess and augment the historical pipeline defect image dataset;

[0034] Step S3: Train an improved YOLOv8 model based on the preprocessed and augmented historical sewage pipeline defect image dataset to obtain a lightweight pipeline defect detection model;

[0035] Step S4: Input the real-time acquired images of the inside of the sewage pipe into the lightweight pipe defect detection model to detect pipe defects.

[0036] As one embodiment of the present invention, in step S2, the preprocessing includes grayscale conversion, contrast enhancement and noise reduction, and the data augmentation includes random cropping, horizontal, vertical and random rotation of the historical sewage pipe defect image dataset.

[0037] As one embodiment of the present invention, such as Figure 2As shown, the improved YOLOv8 model includes the C2f-FAM module, the HS-BiFPN module, and the Upsample module. The C2f-FAM module introduces EMSConvP multi-scale convolution for efficient feature extraction at different scales. The HS-BiFPN module replaces the original neckline of the YOLOv8 model, achieving flexible fusion of feature maps at different scales through cross-level feature fusion operations. This allows each scale's feature map to more fully utilize contextual information from other scales, generating fine-grained feature representations and improving the model's multi-scale object detection capabilities. The DySample module dynamically optimizes the feature map scale based on the target size and scene complexity through upsampling. This adaptive adjustment mechanism enables the model to more effectively capture details when facing targets of various sizes and backgrounds, improving its ability to detect small targets and handle complex backgrounds.

[0038] Furthermore, the working process of the C2f-FAM module is as follows:

[0039] In the task of detecting sewage pipelines, since the dataset usually contains a large number of small, low-resolution target objects, although traditional convolutional layers expand the receptive field through downsampling operations, they are prone to losing key feature information, thus affecting the detection accuracy. Inspired by EMCAD

[29] , this embodiment of the invention proposes an improved module C2f-FAM, which enhances the ability to extract features at different scales by introducing EMSConvP multi-scale convolution. In this module, C2f-FAM replaces the standard convolution (Conv) at the second position in the traditional Bottleneck structure with EMSConvP.

[0040] The EMSConvP module consists of two parts: multi-scale convolution and channel fusion layers. First, features from different receptive fields are extracted using multi-scale convolution kernels to enhance the expressive power of multi-scale targets. Then, 1×1 convolutions are used to adjust the number of channels and effectively fuse multi-scale features.

[0041] Multi-scale convolution employs kernels of different sizes (1×1, 3×3, 5×5, and 7×7) to group the input feature maps by channel. The number of channels in each group is dynamically calculated based on the number of input channels and the number of convolution kernels, and the effectiveness of the convolution operation is ensured by limiting the number of channels in each group to no less than 16. Therefore, in this embodiment of the invention, only the third and fourth C2f modules in the backbone network are replaced. During feature extraction, each group of feature maps performs convolution operations independently, with the structure as follows: Figure 3 As shown.

[0042] After multi-scale convolution, the feature maps are merged through a concatenation operation, and the number of channels is adjusted and features are fused using 1×1 convolution. This design achieves the fusion of multi-scale features while preserving global semantic information, thereby enhancing the model's sensitivity to features at different scales. Compared to traditional full-channel convolution, this module reduces computational complexity and the number of parameters through grouped convolution, thus reducing computational overhead and memory usage, and improving computational efficiency while maintaining performance.

[0043] The C2f-FAM module enhances multi-scale feature extraction capabilities in pipeline detection by introducing EMSConvP, particularly improving detection performance for small targets. This module effectively reduces computational complexity and parameter count through grouped convolutions, thereby optimizing computational efficiency and memory usage while maintaining high detection accuracy, significantly improving the model's detection capabilities.

[0044] Furthermore, the working process of the HS-BiFPN module is as follows:

[0045] In order to address the challenges of variable dimensions, noise interference, and environmental changes in pipeline defect detection, a lightweight neck structure—HS-BiFPN—is proposed, drawing inspiration from the existing multi-scale feature fusion method HS-FPN

[23] . Compared with the original neck structure of YOLOv8, HS-BiFPN effectively improves detection accuracy while reducing computational load and parameter count. The structure is as follows: Figure 4 As shown.

[0046] HS-BiFPN mainly consists of two parts: a feature selection module and a cross-level feature fusion module. The specific process is as follows: First, feature maps of different scales generated by the backbone network undergo effective feature selection by the feature selection module. Then, these feature maps of different scales are fused by the cross-level feature fusion module to generate features with richer semantic information. This fusion helps to more accurately capture subtle features in the pipeline image, thereby improving the model's detection capability.

[0047] Feature Selection Module: In the CA module, global average pooling and global max pooling operations are first performed on the input feature map to calculate the average and maximum values ​​for each channel, respectively, to extract global features. Global average pooling helps to uniformly extract information from the feature map, while global max pooling focuses on extracting the most representative data from each channel, thereby minimizing information loss. Through this pooling method, the CA module can effectively capture the important features of each channel and reduce the interference of redundant information. Next, the sigmoid activation function is used to generate weight values ​​for each channel, thereby weighting each channel and strengthening the expression of key features. Subsequently, the calculated weights are multiplied with the original feature map channel by channel to generate a weighted feature map. This process allows the model to focus on more discriminative features and suppress redundant or irrelevant features, thereby improving the expressive power of the feature map. In addition, to ensure the smooth progress of cross-level feature fusion and achieve effective fusion between feature maps of different scales, a 1×1 convolution operation was adopted to perform dimension matching on feature maps of different scales and adjust the number of channels of each feature map to a uniform 256. This allows feature maps to be fused in the same dimensional space in the subsequent feature fusion stage, thereby avoiding information loss or inconsistency caused by channel mismatch.

[0048] Cross-Level Fusion: In deep neural networks, the multi-scale feature maps generated by the backbone network typically contain semantic information at different levels. High-level features usually originate from deeper layers of the network and possess high semantic richness, capable of capturing global contextual information and abstract target features. However, due to the lower spatial resolution of high-level feature maps, their ability to accurately locate targets is relatively weak, especially in the detection of small targets, exhibiting certain limitations. In contrast, low-level features usually originate from shallower layers of the network, possessing higher spatial resolution and capable of accurately locating target positions and capturing detailed information. However, low-level features are relatively limited in semantic expression, making it difficult to effectively distinguish different targets in complex scenes. Therefore, relying solely on high-level or low-level features each has its shortcomings, and the fusion of the two can effectively compensate for their respective weaknesses. By combining the advantages of high-level and low-level features, the expressive power of the feature maps can be enhanced, thereby improving the accuracy and robustness of target detection.

[0049] To this end, embodiments of the present invention propose a Cross-Level Feature Fusion (CLF) module, such as... Figure 5As shown, in the top-down path, high-level features are first used as weights to fuse low-level features, enhancing their semantic information. Specifically, given a high-level feature map and a low-level feature map as input, the CLF module first upsamples the high-level features using a 3×3 convolution kernel and a transposed convolution with a stride of 2. Then, the CA module processes the high-level features, generating weight values ​​for each channel, and uses these weights to filter redundant information in the low-level features. After this filtering process, the low-level features are fused with the high-level features to generate new feature maps N3, N4, and N5. Next, these fused feature maps enter the bottom-up path for another cross-level feature fusion. In this process, the low-level features are downsampled using a 3×3 convolution kernel and a convolution with a stride of 2, and the CA module generates corresponding weights to filter the high-level features. The filtered high-level features are then refused with the low-level features. Through this series of top-down and bottom-up cross-level fusion operations, not only is information loss effectively reduced, but the expressive power of the feature maps is also enhanced, ultimately generating feature maps with higher semantic information and spatial resolution. In this way, the model can better capture the details of multi-scale targets, thereby improving detection accuracy in complex scenes.

[0050] Furthermore, the Upsample module works as follows:

[0051] Upsampling is a widely used technique in image processing and deep learning, aiming to convert low-resolution images or feature maps into high-resolution outputs. By enlarging the image size, upsampling can effectively improve the resolution of images or feature maps, thereby enhancing the model's ability to capture detailed information. Its core objective is to recover more refined feature representations, providing richer feature support for subsequent processing and analysis. In YOLOv8, the traditional upsampling method uses nearest-neighbor interpolation. This method is widely used due to its fast computation speed and simple implementation, but its limitation is that it only expands by copying the values ​​of the nearest pixels and cannot generate new pixel information. Therefore, nearest-neighbor interpolation is relatively weak in terms of detail preservation and feature representation, especially in complex pipeline defect detection tasks, where noise interference and environmental complexity make it difficult to effectively capture subtle features and fail to meet practical needs. To overcome these problems, this embodiment of the invention uses DySample as the upsampling operator.

[0052] The input feature map, upsampled feature map, generated offset, and original sampling grid of DySample are denoted as X, X', O, and G, respectively. For example... Figure 6 As shown in (a), DySample first constructs a sampling set S from the input feature map X using a point sampling generator, and then resamples the sampling set using a grid sampling function to finally generate the upsampled feature map X'. This process can be represented by the following formula (1):

[0053]

[0054] DySample provides two versions of the generator: a static range factor version and a dynamic range factor version, such as... Figure 6 As shown in (b). In the static range factor version, the offset O is generated through a linear layer and pixel shuffle operation, and combined with the original sampling grid G ​​to finally obtain the sampling set S. This process is shown in equations (2) and (3). In the dynamic range factor version, a range factor is first generated, and then the offset O is modulated using this factor. The generation of the dynamic offset depends on the sigmoid function (σ). The adjusted offset O is added to the sampling grid G ​​to generate the sampling set S. This approach enables the dynamic range factor version to have stronger adaptability during upsampling, and can flexibly adjust the sampling position according to the local features and contextual information of the input image, thereby improving the detail preservation ability of the upsampling result.

[0055]

[0056] DySample's key innovation lies in its dynamic sampling mechanism. By combining the generated offset with the original grid position, it dynamically adjusts the sampling position, thereby significantly improving the quality of the upsampled image. Especially in detail preservation and handling complex textures, DySample demonstrates superior performance compared to traditional upsampling methods. By introducing the modulation of static and dynamic range factors, DySample can flexibly adjust the sampling process based on local information of the input features, thus achieving more accurate image reconstruction.

[0057] This invention employs an improved pipeline defect detection model. First, a C2f-FAM module is proposed, which enhances multi-scale feature extraction and small target detection capabilities by introducing the multi-scale convolution module EMSConvP. Next, HS-BiFPN is designed, which effectively fuses feature maps from different scales, achieving complementary fusion between low-resolution and high-resolution feature maps, thereby capturing features of targets at different scales more comprehensively. Furthermore, a DySample dynamic upsampling module is introduced, which can adjust the resolution of different feature maps as needed to improve the model's adaptability to targets at different scales, thereby enhancing the ability to detect small targets and handle complex backgrounds.

[0058] Example 2:

[0059] This invention also provides a sewage pipeline defect detection device based on multi-scale image feature fusion, comprising:

[0060] The acquisition module is used to acquire a dataset of historical sewage pipe defect images;

[0061] The processing module is used to preprocess and augment the historical pipeline defect image dataset.

[0062] The training module is used to train an improved YOLOv8 model based on a dataset of historical sewage pipe defect images after preprocessing and data augmentation, resulting in a lightweight pipe defect detection model.

[0063] The detection module is used to input real-time acquired images of the inside of sewage pipes into a lightweight pipe defect detection model for pipe defect detection.

[0064] As one embodiment of the present invention, the preprocessing includes grayscale conversion, contrast enhancement and noise reduction, and the data augmentation includes random cropping, horizontal and vertical rotation of the historical sewage pipe defect image dataset.

[0065] As one embodiment of the present invention, the improved YOLOv8 model includes: a C2f-FAM module, an HS-BiFPN module, and an Upsample module; wherein, the C2f-FAM module introduces EMSConvP multi-scale convolution for efficient extraction of features at different scales; the HS-BiFPN module replaces the original neck of the YOLOv8 model and achieves flexible fusion of feature maps at different scales through cross-level feature fusion operations; and the scale of the feature maps is dynamically optimized according to the size of the target and the complexity of the scene through upsampling by the DySample module.

[0066] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for detecting defects in sewage pipes based on multi-scale image feature fusion, characterized in that, include: Step S1: Obtain a dataset of historical sewage pipe defect images; Step S2: Preprocess and augment the historical pipeline defect image dataset; Step S3: Train an improved YOLOv8 model based on the preprocessed and augmented historical sewage pipeline defect image dataset to obtain a lightweight pipeline defect detection model; Step S4: Input the real-time acquired images of the inside of the sewage pipe into the lightweight pipe defect detection model to detect pipe defects. In step S2, preprocessing includes grayscale conversion, contrast enhancement, and noise reduction. Data augmentation includes random cropping, horizontal and vertical rotation, and random rotation of the historical sewage pipe defect image dataset. The improved YOLOv8 model includes the C2f-FAM module, the HS-BiFPN module, and the Upsample module. The C2f-FAM module enhances the extraction capability of features at different scales by introducing EMSConvP multi-scale convolutions. C2f-FAM replaces the standard convolution (Conv) at the second position in the traditional Bottleneck structure with EMSConvP. The HS-BiFPN module replaces the original neck of the YOLOv8 model. HS-BiFPN consists of two parts: a feature selection module and a cross-level feature fusion module. First, feature maps of different scales generated by the backbone network are filtered by the feature selection module. Then, these feature maps of different scales are fused by the cross-level feature fusion module to generate features containing semantic information. The Upsample module dynamically optimizes the scale of the feature maps based on the size of the target and the complexity of the scene through upsampling by the DySample module.

2. A sewage pipeline defect detection device based on multi-scale image feature fusion, implementing the sewage pipeline defect detection method based on multi-scale image feature fusion as described in claim 1, characterized in that, include: The acquisition module is used to acquire a dataset of historical sewage pipe defect images; The processing module is used to preprocess and augment the historical pipeline defect image dataset. The training module is used to train an improved YOLOv8 model based on a dataset of historical sewage pipe defect images after preprocessing and data augmentation, resulting in a lightweight pipe defect detection model. The detection module is used to input real-time acquired images of the inside of sewage pipes into a lightweight pipe defect detection model for pipe defect detection. Preprocessing includes grayscale conversion, contrast enhancement, and noise reduction. Data augmentation includes random cropping, horizontal and vertical rotation, and random rotation of the historical sewage pipe defect image dataset. The improved YOLOv8 model includes the C2f-FAM module, the HS-BiFPN module, and the Upsample module. The C2f-FAM module introduces EMSConvP multi-scale convolution to extract features at different scales. The HS-BiFPN module replaces the original neck of the YOLOv8 model and achieves the fusion of feature maps at different scales through cross-level feature fusion operations. The DySample module upsamples the feature map scale dynamically to optimize the scale according to the size of the target and the complexity of the scene.

Images