A deep learning-based urban pipeline defect identification method

Abstract

The present application relates to a kind of city pipeline defect identification method based on deep learning, it is related to defect identification technical field, the present application obtains the data set for training pipeline defect identification model;Pipeline defect identification model is trained by constructing and using the data set;Wherein, at least one adjacent maximum pooling layer and multi-branch feature extraction block are inserted in pipeline defect identification model Semantics enhancement block, semantics enhancement block combines maximum pooling layer output and filter feature map or combines maximum pooling layer output and down-sampling filter feature map to obtain defect attention, using defect attention weighted processing maximum pooling layer output to obtain defect semantic enhancement pipeline image feature map;When applied, pipeline image is collected, and the pipeline image collected is filtered to obtain filter feature map, pipeline image and filter feature map are submitted to pre-trained pipeline defect identification model and are subjected to defect identification, according to defect identification result, alarm is carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline defect identification, and in particular to a method for identifying urban pipeline defects based on deep learning. Background Art

[0002] Cities are home to a vast network of complex water, gas, and heat pipelines. According to incomplete statistics, the scale of urban pipelines is rapidly increasing, and most pipelines are overloaded and severely aged. Inadequate monitoring and maintenance of pipeline defects can easily lead to serious consequences, including safety incidents such as gas explosions, sudden pipe bursts, and water leaks, as well as pipeline structural failures and environmental hazards. Therefore, automating defect detection through advanced computer vision technology is of great significance for efficient pipeline maintenance and management. In existing technologies, semantic segmentation models based on the Unet architecture can be applied to pipeline defect identification tasks. However, the semantic segmentation model based on the Unet architecture faces challenges in pipeline defect identification:

[0003] Pipeline defects cover a wide range of scales, with a large number of small-scale targets (such as millimeter-level cracks and tiny perforations) that have a low pixel ratio and little feature information, as well as large-scale dents and corrosion. Unet's encoding process compresses spatial information through downsampling (to obtain high-level semantic features). Although the decoding process fuses low-level detail features through jump connections, the semantic distinction of low-level features is insufficient, and the spatial resolution of high-level features is low, resulting in poor localization and segmentation accuracy of small-scale defects, prone to missed detections, and insufficient segmentation accuracy. Furthermore, there is an imbalance between the number of pipeline defect categories and their impact: the training set contains more samples of minor corrosion that have a smaller impact on the pipeline, and fewer samples of local deformation that has a greater impact on pipeline safety. The model tends to favor the majority type of defect and has weak recognition capabilities for rare defects. Consequently, the model's defect recognition capabilities are mismatched with its defect impact capabilities. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems and address the above shortcomings, the present invention provides a method, device and storage medium for urban pipeline defect identification based on deep learning.

[0005] In a first aspect, the present invention provides a method for identifying urban pipeline defects based on deep learning, comprising:

[0006] Obtaining a dataset for training a pipeline defect recognition model, the dataset comprising a pipeline image, a filtered feature map of the pipeline image, a defect semantic segmentation mask, and a defect type label;

[0007] Construct and use the data set to train a pipeline defect recognition model; wherein, the pipeline defect recognition model includes: a five-layer multi-branch feature extraction block; the first four layers of multi-branch feature extraction blocks are respectively followed by a maximum pooling layer for downsampling; the last four layers of multi-branch feature extraction blocks are followed by a convolution block; using the last convolution block and multiple upsampling to upsample the outputs of the convolution blocks connected to the last three layers of multi-branch feature extraction blocks, and then combine them with the outputs of the convolution blocks of the next layer to obtain multiple groups of target feature maps, the multiple groups of target feature maps are respectively processed by depthwise separable convolution and upsampled and then added, the addition results are mapped by depthwise separable convolution to obtain predicted defect semantic segmentation masks, and the addition results are processed by a fully connected layer to obtain predicted defect type probability; a semantic enhancement block is set between at least one adjacent maximum pooling layer and multi-branch feature extraction block, the semantic enhancement block combines the output of the maximum pooling layer and the filtered feature map or combines the output of the maximum pooling layer and the downsampled filtered feature map to obtain defect attention, and uses the defect attention to weight the output of the maximum pooling layer to obtain a pipeline image feature map with defect semantic enhancement;

[0008] When applied, pipeline images are collected, the collected pipeline images are filtered to obtain filtered feature maps, the pipeline images and filtered feature maps are submitted to a pre-trained pipeline defect recognition model for defect recognition, and an alarm is issued based on the defect recognition results.

[0009] Furthermore, the datasets used to train the pipeline defect recognition model include:

[0010] Collect pipeline images of various materials, shapes, sizes, and imaging environments, and annotate the pipeline images using defect semantic segmentation masks;

[0011] For the collected pipeline image, the collected pipeline image is filtered using vertical and horizontal edge detection filters, vertical and horizontal line detection filters, and diagonal line detection filters, and multiple sets of corresponding filtering feature maps are obtained from the pipeline image;

[0012] Different defects have different impacts on pipeline safety and economy. Defects are assigned weights based on their impact on pipeline safety and economy.

[0013] Furthermore, the edge detection filter consists of two rectangular filter areas of the same size and closely adjacent to each other, and the two rectangular filter areas are assigned opposite weights. In the vertical edge detection filter, the rectangular filter areas are arranged horizontally and adjacent to each other on the left and right; in the horizontal edge detection filter, the rectangular filter areas are arranged vertically and adjacent to each other on the top and bottom.

[0014] The line detection filter consists of three rectangular filter areas of equal size and closely adjacent to each other. The middle rectangular filter area and the rectangular filter areas on both sides are assigned opposite weights. In the vertical line detection filter, the rectangular filter areas are arranged horizontally and adjacent to each other on the left and right. In the horizontal line detection filter, the rectangular filter areas are arranged vertically and adjacent to each other on the top and bottom.

[0015] The diagonal line detection filter is composed of four rectangular filter areas. In the quadrants corresponding to the four rectangular filter areas, two rectangular filter areas at one diagonal corner and two rectangular filter areas at another diagonal corner are assigned opposite weights.

[0016] Furthermore, the multi-branch feature extraction block includes the following three parallel branches:

[0017] The first depth-wise separable convolution branch includes: two depth-wise separable convolutions with a first kernel size, and the depth-wise separable convolution keeps the feature map size unchanged; the second depth-wise separable convolution branch includes: two depth-wise separable convolutions with a second kernel size, the second kernel size is larger than the first kernel size, and the second depth-wise separable convolution branch captures a wider range of contextual information than the first depth-wise separable convolution branch, while also keeping the feature map size unchanged; the pooling and point convolution branch includes maximum pooling with a third kernel size and a step size of 1, which also keeps the feature map size unchanged, and then uses point convolution to adjust the number of channels of the feature map after maximum pooling; the multi-branch feature extraction block connects the outputs of the three branches in a fixed order in the channel dimension to obtain the final output.

[0018] Furthermore, the semantic enhancement block uses depthwise separable convolution to process the received maximum pooling layer output so that the maximum pooling layer output is aligned with the filtered feature map or the downsampled filtered feature map in the channel dimension. The output of the depthwise separable convolution is added to the filtered feature map or the downsampled filtered feature map, and then the addition result is processed by point convolution and activation function to obtain the defect attention. The defect attention is multiplied point by point with the maximum pooling layer output to obtain the pipeline image feature map with defect semantic enhancement.

[0019] Furthermore, the process of training the pipeline defect recognition model includes:

[0020] Defects are assigned weights based on their impact on pipeline safety and availability, with the greater the impact, the greater the weight assigned. Various defects are proportionally selected from the training set based on the weights assigned to the defects to train the pipeline defect recognition model.

[0021] During the training of the pipeline defect recognition model, the learning rate is set, the Adam optimizer is used to adjust the model parameters, and the weighted sum of the weighted intersection-over-union entropy loss and the weighted classification cross entropy loss between the predicted defect semantic segmentation mask and the true defect semantic segmentation mask is used as the loss function. The parameters of the pipeline defect recognition model are adjusted with the goal of minimizing the loss function.

[0022] Furthermore, the formula for the weighted intersection-over-union entropy loss is:

[0023] ,

[0024] in, is the predicted defect semantic segmentation mask for the c-th defect, is the true defect semantic segmentation mask of the c-th defect, is the weight corresponding to the c-th type of defect, and the total number of types of C defects;

[0025] The formula for the weighted classification cross entropy loss is:

[0026] ;

[0027] in, is the predicted probability of the c-th type defect, is the true type label of the c-th defect.

[0028] Furthermore, the semantic enhancement module is arranged between the second-layer maximum pooling layer and the third-layer multi-branch feature extraction block, or, the semantic enhancement module is arranged both between the second-layer maximum pooling layer and the third-layer multi-branch feature extraction block and between the third-layer maximum pooling layer and the fourth-layer multi-branch feature extraction block.

[0029] In a second aspect, the present invention provides a deep learning-based urban pipeline defect identification system, comprising: multiple interconnected network devices, any network device comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit storing a computer program, and the processing unit implementing the deep learning-based urban pipeline defect identification method by running the computer program stored in the storage unit.

[0030] In a third aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it implements the urban pipeline defect identification method based on deep learning.

[0031] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0032] In the pipeline defect recognition model constructed by the present application, the depthwise separable convolution of the first kernel size of the multi-branch feature extraction block captures the subtle features of the defect, and the depthwise separable convolution of the second kernel size captures the overall features of the detected defect. The parallel maximum pooling layer helps the model maintain the spatial hierarchy and enhances its robustness to changes in defect locations. The design of the semantic enhancement block introduces texture, line, and boundary features in the filter feature map on the basis of the pipeline image feature map to enhance the semantic representation of defects and improve the defect recognition ability of the pipeline defect recognition model. On the other hand, it guides the pipeline defect recognition model to pay more attention to defects through defect attention, which can speed up the convergence of the pipeline defect recognition model. The present application adopts more computationally efficient depthwise separable convolution on a large scale. The significant reduction in the number of parameters reduces the computational complexity and supports application to edge devices or mobile platforms with limited computing and storage capabilities.

[0033] This application configures weights based on the impact of defects on pipelines. During training, various defects are proportionally selected from the training set based on the assigned weights to train the pipeline defect recognition model. Weighted intersection-over-union loss and weighted classification cross entropy loss are introduced into the loss function. Weights are taken into account during the adjustment of model parameters to ensure that the pipeline defect recognition model has a stronger ability to identify defects that may have a greater impact on the pipeline. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 A flowchart of a method for identifying urban pipeline defects based on deep learning provided by an embodiment of the present invention;

[0037] Figure 2 This is an architectural diagram of a pipeline defect recognition model provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a multi-branch feature extraction block provided in an embodiment of the present invention;

[0039] Figure 4 A flowchart of the training provided by an embodiment of the present invention;

[0040] Figure 5Schematic diagram of a deep learning-based urban pipeline defect identification device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0042] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0043] Example 1

[0044] like Figure 1 As shown, the present invention provides a method for identifying urban pipeline defects based on deep learning, comprising:

[0045] S100: Obtain a data set for training a pipeline defect recognition model, including:

[0046] Pipeline images of various materials, shapes, sizes, and imaging environments are collected and annotated using defect semantic segmentation masks. The defect semantic segmentation masks contain the outline information of the defects contained in the pipeline images.

[0047] For the collected pipeline image, vertical and horizontal edge detection filters, vertical and horizontal line detection filters and diagonal line detection filters are used to perform filtering processing on the collected pipeline image, and multiple groups of corresponding filtering feature maps are obtained from the pipeline image.

[0048] The edge detection filter consists of two rectangular filter areas of the same size and closely adjacent to each other, and the two rectangular filter areas are assigned opposite weights respectively; in the vertical edge detection filter, the rectangular filter areas are arranged horizontally and adjacent to each other on the left and right; in the horizontal edge detection filter, the rectangular filter areas are arranged vertically and adjacent to each other on the top and bottom; when there is an edge in the area covered by the edge detection filter, the pixel values ​​of the rectangular filter areas on both sides are significantly different, which will produce larger positive and negative eigenvalues. The greater the difference, the more obvious the edge; if the pixels in the area are uniform and there is no edge, the eigenvalue is close to 0.

[0049] The line detection filter consists of three rectangular filter areas of the same size and closely adjacent to each other. The middle rectangular filter area and the rectangular filter areas on both sides are assigned opposite weights. In the vertical line detection filter, the rectangular filter areas are arranged horizontally and adjacent to each other on the left and right. In the horizontal line detection filter, the rectangular filter areas are arranged vertically and adjacent to each other on the top and bottom. When the line detection filter covers a line, the pixel value difference between the middle rectangle and the two side rectangles is significant, and the eigenvalue is large. If the line detection filter covers a uniform area or a wide edge, the difference between the middle and the two sides will be diluted, and the eigenvalue is small.

[0050] The diagonal detection filter consists of four rectangular filter areas. In the quadrants corresponding to the four rectangular filter areas, the two rectangular filter areas in one diagonal and the two rectangular filter areas in the other diagonal are assigned opposite weights. When there are diagonal edges in the image, the pixel values ​​in the diagonal areas are significantly different, and the diagonal detection filter will produce a larger eigenvalue. If there is no diagonal structure in the area, the pixel difference in the diagonal areas is small and the eigenvalue is close to 0.

[0051] Defective areas in pipelines, such as dents, misaligned pipes, and cracks, have obvious edges. Vertical and horizontal edge detection filters, vertical and horizontal line detection filters, and diagonal line detection filters are used to extract filtered feature maps that highlight edge and texture features from pipeline images to enhance the representation of defective areas.

[0052] Mark the types of defects contained in pipeline images;

[0053] Through the above process, a dataset containing the original pipeline image, the filtered feature map of the pipeline image, the defect semantic segmentation mask and the defect type label was obtained. The dataset was divided into training set, test set and validation set at 70%, 15% and 15% respectively.

[0054] S200: Construct pipeline defect recognition model. In the specific implementation process, Figure 2 As shown, the pipeline defect recognition model includes:

[0055] Five layers of multi-branch feature extraction blocks; the first four layers of multi-branch feature extraction blocks are followed by a maximum pooling layer for downsampling; the output of the maximum pooling layer is transmitted to the next set of multi-branch feature extraction blocks directly or after being enhanced by the semantic enhancement block.

[0056] The last four layers of multi-branch feature extraction blocks are followed by convolution blocks. The final convolution block and multiple upsampling steps upsample the outputs of the convolution blocks connected to the last three layers of multi-branch feature extraction blocks and combine them with the outputs of the convolution blocks in the next layer to generate multiple sets of target feature maps. These multiple sets of target feature maps are processed and upsampled using depthwise separable convolutions before being added together. The summed results are then mapped using depthwise separable convolutions to generate the predicted semantic segmentation mask for the defect.

[0057] A semantic enhancement block is set between at least one adjacent maximum pooling layer and multi-branch feature extraction block. The semantic enhancement block combines the maximum pooling layer output and the filtered feature map or combines the maximum pooling layer output and the down-sampled filtered feature map to obtain defect attention, and uses the defect attention to weightedly process the maximum pooling layer output to obtain a pipeline image feature map with defect semantic enhancement.

[0058] During the specific implementation process, the semantic enhancement block uses the depthwise separable convolution to process the received maximum pooling layer output so that the maximum pooling layer output is aligned with the filter feature map or the downsampled filter feature map in the channel dimension. The output of the depthwise separable convolution is added to the filter feature map or the downsampled filter feature map, and then the addition result is processed by point convolution and activation function to obtain the defect attention. The defect attention is multiplied point by point with the maximum pooling layer output to obtain the pipeline image feature map with defect semantic enhancement.

[0059] like Figure 2 As shown, in Figure 2 In the given example, one semantic enhancement block receives the filtered feature map and the output of the second maximum pooling layer. The semantic enhancement block combines the output of the second maximum pooling layer and the filtered feature map to obtain a first defect attention, and uses the first defect attention to weight the output of the second maximum pooling layer to obtain a pipeline image feature map with defect semantic enhancement. The filtered feature map is sampled, and another semantic enhancement block receives the downsampled filtered feature map and the output of the third maximum pooling layer. The semantic enhancement block combines the output of the third maximum pooling layer and the downsampled filtered feature map to obtain a second defect attention, and uses the second defect attention to weight the output of the third maximum pooling layer to obtain a pipeline image feature map with defect semantic enhancement.

[0060] In another example, compared to Figure 2 In the given example, the semantic enhancement module between the third-layer maximum pooling layer and the fourth-layer multi-branch feature extraction block can be removed.

[0061] On the one hand, the semantic enhancement block can enhance the semantic representation of defects in the pipeline image feature map and improve the defect recognition ability of the pipeline defect recognition model. On the other hand, it can guide the pipeline defect recognition model to pay attention to defects faster through defect attention, thereby accelerating the convergence speed of the pipeline defect recognition model.

[0062] The specific implementation of using the last convolution block and multiple upsampling operations to upsample the outputs of the convolution blocks connected to the last three layers of multi-branch feature extraction blocks and then combine them with the outputs of the convolution blocks of the next layer to obtain multiple sets of target feature maps is as follows: the output of the fifth layer of multi-branch feature extraction blocks is processed by a convolution block to reduce the channel dimension to obtain the fourth target feature map, the output of the fourth layer of multi-branch feature extraction blocks is processed by a convolution block and then combined with the upsampling result of the fourth target feature map to obtain the third target feature map, the output of the third layer of multi-branch feature extraction blocks is processed by a convolution block and then combined with the upsampling result of the third target feature map to obtain the second target feature map, and the output of the second layer of multi-branch feature extraction blocks is processed by a convolution block and then combined with the upsampling result of the second target feature map to obtain the first target feature map. High-resolution target feature maps are generated layer by layer through upsampling operations and feature fusion. A refined feature map is combined with a feature map with rich contextual information but low resolution to achieve seamless integration, thereby obtaining a target feature map with richer defect information, which is helpful for subsequent defect segmentation.

[0063] Furthermore, the first, second, third, and fourth target feature maps are each subjected to depthwise separable convolution, upsampled to the pipeline image size, combined, and processed again through depthwise separable convolution to produce the predicted defect semantic segmentation mask. The depthwise separable convolution results for the first, second, third, and fourth target feature maps are upsampled by a factor of 2, 4, 8, and 16, respectively.

[0064] In order to better identify the details of the defect, such as Figure 3 As shown, the multi-branch feature extraction block of the present application includes the following three parallel branches:

[0065] The first depthwise separable convolution branch consists of two depthwise separable convolutions with a first kernel size, which maintain the feature map size unchanged. The second depthwise separable convolution branch consists of two depthwise separable convolutions with a second kernel size, larger than the first. Compared to the first, the second depthwise separable convolution branch captures a wider range of contextual information while also maintaining the feature map size unchanged. The pooling and pointwise convolution branch includes max pooling with a third kernel size and a stride of 1, also maintaining the feature map size unchanged. Pointwise convolution is then used to adjust the number of channels in the feature map after max pooling. The multi-branch feature extraction block concatenates the outputs of the three branches in a fixed order along the channel dimension to produce the final output. The two depthwise separable convolution sizes capture a wide range of spatial information for objects of different scales. Specifically, the depthwise separable convolution with the first kernel size captures subtle features of defects, while the depthwise separable convolution with the second kernel size captures the overall characteristics of the detected defects. The parallel max pooling layers help the model maintain spatial hierarchy and enhance its robustness to variations in defect location.

[0066] Depthwise separable convolution decomposes the standard convolution into depthwise convolution and point-wise convolution, which greatly reduces the number of model parameters and computational complexity.

[0067] The convolution block includes depthwise separable convolution, batch normalization, and activation functions. The activation function uses the ReLU activation function by default to introduce nonlinear characteristics, enabling the model to learn complex patterns.

[0068] S300, the process of training the pipeline defect recognition model using the data set, such as Figure 4 Shown, including:

[0069] Because different defects have varying degrees of impact on pipeline safety and availability, during the training process, defects are weighted based on their impact. Defects with greater impact receive a greater weight. For example, a pipe break defect has a greater impact on pipeline safety and availability than a pipe dent defect, so the weight assigned to the pipe break defect is greater than that to the pipe dent defect.

[0070] Various defects are proportionally selected from the training set according to the weights assigned to the defects to train the pipeline defect recognition model, so that the pipeline defect recognition model can better identify major defects that affect pipeline safety and availability.

[0071] During the training of the pipeline defect recognition model, the learning rate is set, the Adam optimizer is used to adjust the model parameters, and the weighted sum of the weighted intersection-over-union entropy loss and the weighted classification cross entropy loss between the predicted defect semantic segmentation mask and the true defect semantic segmentation mask is used as the loss function. The parameters of the pipeline defect recognition model are adjusted with the goal of minimizing the loss function.

[0072] The formula for the weighted intersection-over-union entropy loss is:

[0073] ,

[0074] in, is the predicted defect semantic segmentation mask for the c-th defect, is the true defect semantic segmentation mask of the c-th defect, is the weight corresponding to the c-th type of defect, and C is the total number of defect types;

[0075] The formula for the weighted classification cross entropy loss is:

[0076] ;

[0077] in, is the predicted probability of the c-th type defect, is the true type label of the c-th defect.

[0078] The loss function is: ,in, is the weight adjustment coefficient of the two losses.

[0079] S400, when applied, collects a pipeline image through a preset imaging device, filters the collected pipeline image to obtain a filtered feature map, and submits the pipeline image and the filtered feature map to a pre-trained pipeline defect recognition model for defect recognition; when necessary, the pipeline defect recognition model downsamples the filtered feature map and uses the downsampled filtered feature map to enhance the semantics of the pipeline image feature map.

[0080] S500: Issue an alarm based on the defect identification result.

[0081] This application performs well in defect semantic segmentation, and the design of the multi-branch feature extraction block supports capturing defect details in pipeline images. The design of the semantic enhancement block introduces texture, line, and boundary features in the filter feature map based on the pipeline image feature map to enhance the defect semantics. The intersection over union (IoU) of the model in defect semantic segmentation is 6% higher than that of the baseline model. This application adopts depthwise separable convolution with higher computational efficiency on a large scale, and the significant reduction in the number of parameters reduces computational complexity, supporting application to edge devices or mobile platforms with limited computing and storage capabilities. The introduction of the semantic enhancement module also speeds up the convergence of the model, making training more efficient.

[0082] Ablation experiments show that incorporating features extracted from filtered feature maps can significantly improve the performance of the pipeline defect recognition model, which is manifested in: improving the pipeline defect recognition model's ability to capture fine details and complex structures of defects, improving the accuracy of defect semantic segmentation, and accelerating the convergence of the pipeline defect recognition model and shortening the training time.

[0083] Example 2

[0084] See Figure 5 As shown, an embodiment of the present invention provides a deep learning-based urban pipeline defect identification device, comprising: at least one processing unit, the processing unit connected to a storage unit via a bus unit. The storage unit, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as the software programs, computer executable programs, and modules corresponding to a deep learning-based urban pipeline defect identification method in an embodiment of the present invention. The processing unit implements the deep learning-based urban pipeline defect identification method by running the software programs, computer executable programs, and modules stored in the storage unit, including:

[0085] Obtaining a dataset for training a pipeline defect recognition model, the dataset comprising a pipeline image, a filtered feature map of the pipeline image, a defect semantic segmentation mask, and a defect type label;

[0086] Construct and use the data set to train a pipeline defect recognition model; wherein, the pipeline defect recognition model includes: a five-layer multi-branch feature extraction block; the first four layers of multi-branch feature extraction blocks are respectively followed by a maximum pooling layer for downsampling; the last four layers of multi-branch feature extraction blocks are followed by a convolution block; using the last convolution block and multiple upsampling to upsample the outputs of the convolution blocks connected to the last three layers of multi-branch feature extraction blocks, and then combine them with the outputs of the convolution blocks of the next layer to obtain multiple groups of target feature maps, the multiple groups of target feature maps are respectively processed by depthwise separable convolution and upsampled and then added, the addition results are mapped by depthwise separable convolution to obtain predicted defect semantic segmentation masks, and the addition results are processed by a fully connected layer to obtain predicted defect type probability; a semantic enhancement block is set between at least one adjacent maximum pooling layer and multi-branch feature extraction block, the semantic enhancement block combines the output of the maximum pooling layer and the filtered feature map or combines the output of the maximum pooling layer and the downsampled filtered feature map to obtain defect attention, and uses the defect attention to weight the output of the maximum pooling layer to obtain a pipeline image feature map with defect semantic enhancement;

[0087] When applied, pipeline images are collected, the collected pipeline images are filtered to obtain filtered feature maps, the pipeline images and filtered feature maps are submitted to a pre-trained pipeline defect recognition model for defect recognition, and an alarm is issued based on the defect recognition results.

[0088] Of course, the computer program stored in the storage unit of the urban pipeline defect identification system based on deep learning provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the urban pipeline defect identification method based on deep learning provided by any embodiment of the present invention.

[0089] Example 3

[0090] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed, the method for identifying urban pipeline defects based on deep learning is implemented, including:

[0091] Obtaining a dataset for training a pipeline defect recognition model, the dataset comprising a pipeline image, a filtered feature map of the pipeline image, a defect semantic segmentation mask, and a defect type label;

[0092] Construct and use the data set to train a pipeline defect recognition model; wherein, the pipeline defect recognition model includes: a five-layer multi-branch feature extraction block; the first four layers of multi-branch feature extraction blocks are respectively followed by a maximum pooling layer for downsampling; the last four layers of multi-branch feature extraction blocks are followed by a convolution block; using the last convolution block and multiple upsampling to upsample the outputs of the convolution blocks connected to the last three layers of multi-branch feature extraction blocks, and then combine them with the outputs of the convolution blocks of the next layer to obtain multiple groups of target feature maps, the multiple groups of target feature maps are respectively processed by depthwise separable convolution and upsampled and then added, the addition results are mapped by depthwise separable convolution to obtain predicted defect semantic segmentation masks, and the addition results are processed by a fully connected layer to obtain predicted defect type probability; a semantic enhancement block is set between at least one adjacent maximum pooling layer and multi-branch feature extraction block, the semantic enhancement block combines the output of the maximum pooling layer and the filtered feature map or combines the output of the maximum pooling layer and the downsampled filtered feature map to obtain defect attention, and uses the defect attention to weight the output of the maximum pooling layer to obtain a pipeline image feature map with defect semantic enhancement;

[0093] When applied, pipeline images are collected, the collected pipeline images are filtered to obtain filtered feature maps, the pipeline images and filtered feature maps are submitted to a pre-trained pipeline defect recognition model for defect recognition, and an alarm is issued based on the defect recognition results.

[0094] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, structure or unit, which can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for identifying urban pipeline defects based on deep learning, characterized in that: include: Obtaining a dataset for training a pipeline defect recognition model, the dataset comprising a pipeline image, a filtered feature map of the pipeline image, a defect semantic segmentation mask, and a defect type label; Construct and use the data set to train a pipeline defect recognition model; wherein, the pipeline defect recognition model includes: a five-layer multi-branch feature extraction block; the first four layers of multi-branch feature extraction blocks are respectively followed by a maximum pooling layer for downsampling; the last four layers of multi-branch feature extraction blocks are followed by a convolution block; using the last convolution block and multiple upsampling to upsample the outputs of the convolution blocks connected to the last three layers of multi-branch feature extraction blocks, and then combine them with the outputs of the convolution blocks of the next layer to obtain multiple groups of target feature maps, the multiple groups of target feature maps are respectively processed by depthwise separable convolution and upsampled and then added, the addition results are mapped by depthwise separable convolution to obtain predicted defect semantic segmentation masks, and the addition results are processed by a fully connected layer to obtain predicted defect type probability; a semantic enhancement block is set between at least one adjacent maximum pooling layer and multi-branch feature extraction block, the semantic enhancement block combines the output of the maximum pooling layer and the filtered feature map or combines the output of the maximum pooling layer and the downsampled filtered feature map to obtain defect attention, and uses the defect attention to weight the output of the maximum pooling layer to obtain a pipeline image feature map with defect semantic enhancement; The process of training the pipeline defect recognition model includes: Defects are assigned weights based on their impact on pipeline safety and availability, with the greater the impact, the greater the weight assigned. Various defects are proportionally selected from the training set based on the weights assigned to the defects to train the pipeline defect recognition model. During the training of the pipeline defect recognition model, a learning rate is set, the Adam optimizer is used to adjust the model parameters, and the weighted sum of the weighted intersection-over-entropy loss and the weighted classification cross-entropy loss between the predicted defect semantic segmentation mask and the true defect semantic segmentation mask is used as the loss function. The parameters of the pipeline defect recognition model are adjusted with the goal of minimizing the loss function. The formula for the weighted intersection-over-entropy loss is: ， in, is the predicted defect semantic segmentation mask for the c-th defect, is the true defect semantic segmentation mask of the c-th defect, is the weight corresponding to the c-th type of defect, and C is the total number of defect types; The formula for the weighted classification cross entropy loss is: ； in, is the predicted probability of the c-th type defect, is the true type label of the c-th defect; When applied, pipeline images are collected, the collected pipeline images are filtered to obtain filtered feature maps, the pipeline images and filtered feature maps are submitted to a pre-trained pipeline defect recognition model for defect recognition, and an alarm is issued based on the defect recognition results.

2. The urban pipeline defect identification method based on deep learning according to claim 1 is characterized in that: The datasets used to train the pipeline defect recognition model include: Collect pipeline images of various materials, shapes, sizes, and imaging environments, and annotate the pipeline images using defect semantic segmentation masks; For the collected pipeline image, vertical and horizontal edge detection filters, vertical and horizontal line detection filters and diagonal line detection filters are used to perform filtering processing on the collected pipeline image, and multiple groups of corresponding filtering feature maps are obtained from the pipeline image.

3. The urban pipeline defect identification method based on deep learning according to claim 2 is characterized in that: The edge detection filter consists of two rectangular filter areas of the same size and closely adjacent to each other. The two rectangular filter areas are assigned opposite weights. In the vertical edge detection filter, the rectangular filter areas are arranged horizontally and adjacent to each other on the left and right. In the horizontal edge detection filter, the rectangular filter areas are arranged vertically and adjacent to each other on the top and bottom. The line detection filter consists of three rectangular filter areas of equal size and closely adjacent to each other. The middle rectangular filter area and the rectangular filter areas on both sides are assigned opposite weights. In the vertical line detection filter, the rectangular filter areas are arranged horizontally and adjacent to each other on the left and right. In the horizontal line detection filter, the rectangular filter areas are arranged vertically and adjacent to each other on the top and bottom. The diagonal line detection filter is composed of four rectangular filter areas. In the quadrants corresponding to the four rectangular filter areas, two rectangular filter areas at one diagonal corner and two rectangular filter areas at another diagonal corner are assigned opposite weights.

4. The urban pipeline defect identification method based on deep learning according to claim 1 is characterized in that: The multi-branch feature extraction block contains the following three parallel branches: The first depth-wise separable convolution branch includes: two depth-wise separable convolutions with a first kernel size, and the depth-wise separable convolution keeps the feature map size unchanged; the second depth-wise separable convolution branch includes: two depth-wise separable convolutions with a second kernel size, the second kernel size is larger than the first kernel size, and the second depth-wise separable convolution branch captures a wider range of contextual information than the first depth-wise separable convolution branch, while also keeping the feature map size unchanged; the pooling and point convolution branch includes maximum pooling with a third kernel size and a step size of 1, which also keeps the feature map size unchanged, and then uses point convolution to adjust the number of channels of the feature map after maximum pooling; the multi-branch feature extraction block connects the outputs of the three branches in a fixed order in the channel dimension to obtain the final output.

5. The urban pipeline defect identification method based on deep learning according to claim 1 is characterized in that: The semantic enhancement block uses depthwise separable convolution to process the received maximum pooling layer output so that the maximum pooling layer output is aligned with the filtered feature map or the downsampled filtered feature map in the channel dimension. The output of the depthwise separable convolution is added to the filtered feature map or the downsampled filtered feature map, and then the addition result is processed by point convolution and activation function to obtain the defect attention. The defect attention is multiplied point by point with the maximum pooling layer output to obtain the pipeline image feature map with defect semantic enhancement.

6. The urban pipeline defect identification method based on deep learning according to claim 1 is characterized in that: The semantic enhancement module is arranged between the second-layer maximum pooling layer and the third-layer multi-branch feature extraction block, or the semantic enhancement module is arranged both between the second-layer maximum pooling layer and the third-layer multi-branch feature extraction block and between the third-layer maximum pooling layer and the fourth-layer multi-branch feature extraction block.

7. A deep learning-based urban pipeline defect identification device, comprising: At least one processing unit, the processing unit being connected to a storage unit via a bus unit, characterized in that the storage unit stores a computer program, and the processing unit implements the urban pipeline defect identification method based on deep learning according to any one of claims 1 to 6 by running the computer program stored in the storage unit.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the urban pipeline defect identification method based on deep learning according to any one of claims 1 to 6 is implemented.

Images