A method and device for detecting defects in underground drainage pipes based on fine-grained features

Abstract

The present application relates to the technical field of fine-grained image recognition, and particularly relates to a fine-grained feature-based underground drainage pipeline defect detection method, which comprises the following steps: obtaining a target recognition model and a to-be-recognized picture of an underground drainage pipeline, the target recognition model comprising a feature extraction network module, a feature fusion network module and an output head module, the output result of the output head module comprising a specified small-scale feature map, and the specified small-scale feature map being formed by an extended feature fusion layer output by the feature fusion network module; and performing detection and recognition processing on the to-be-recognized picture through the target recognition model to obtain a detection and recognition result of the recognized picture, wherein the detection and recognition result comprises a defect position, a defect category, a defect level and a confidence of a label frame corresponding to the defect of the recognized picture. The method can accurately recognize defects in the underground drainage pipeline picture, accurately divide easily-confused defect categories and more easily-confused defect levels, and improve defect detection performance.

Description

Technical Field

[0001] This invention relates to the field of fine-grained image recognition technology, and in particular to a method and apparatus for detecting defects in underground drainage pipes based on fine-grained features. Background Technology

[0002] Currently, defect detection technology is widely used in industrial fields such as defect analysis of urban underground drainage pipes, insulator feature analysis, and surface crack defect analysis of metals and magnetic tiles, showing very broad application prospects and huge potential economic value. With the rapid development of cities, problems such as blockage and bursting of urban underground drainage pipes are emerging frequently. How to accurately and quickly detect defects in captured images or video sequences of urban underground drainage pipes has always been one of the hot application problems in industrial defect detection.

[0003] With the continuous development of deep learning, underground drainage pipe defect detection technology increasingly relies on high-performance target detectors. High-performance target detectors heavily depend on the feature extraction of the backbone network in the early stage and the feature fusion of the neck network in the middle stage. Depending on whether the detector generates region proposals, detection algorithms are divided into one-stage and two-stage methods. Two-stage methods, such as the R-CNN series detectors, SPPNet, and RFCN, offer higher detection accuracy. While two-stage methods achieve higher accuracy in determining the target location and classification, they are limited by the large number of regions to be calculated, heavily reliant on computational resources, and have slow inference speeds, leading to their gradual decline in the industrial defect detection field. One-stage methods, such as SSD, RetinaNet, and YOLO series detectors, offer higher real-time performance. One-stage methods directly regress the target's location, category, and confidence score without excessive computation, but they are limited by the matching degree between the pre-defined prior bounding box information and the overall target bounding box information. A low matching degree results in low accuracy, but this method is currently widely used in the defect detection field.

[0004] Based on the level of detail in visual image analysis, defect detection technology can be divided into coarse-grained and fine-grained defect detection methods. Most existing methods for detecting defects in underground drainage pipes employ coarse-grained approaches, classifying easily processed categories within the image. Identifying categories with minimal differences achieves good results without considering the severity or grade of each defect. However, defects in underground drainage pipes can be easily confused, often leading to false positives due to similar characteristics between certain categories. Adding a severity assessment further reduces the detector's accuracy. Therefore, existing defect detection methods for underground drainage pipes are insufficient to accurately distinguish between easily confused defect categories and even more easily confused defect grades. Summary of the Invention

[0005] This application provides a method and apparatus for detecting defects in underground drainage pipes based on fine-grained features. This solves the technical problem of low detection accuracy in the prior art for some defects of easily confused defect categories and defects of difficult-to-distinguish levels in underground drainage pipe images. It achieves the technical effect of not only accurately identifying defects in underground drainage pipe images, but also accurately classifying easily confused defect categories and more easily confused defect levels, thereby improving the overall defect detection performance.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting defects in underground drainage pipes based on fine-grained features, comprising:

[0007] The target recognition model and the image of the underground drainage pipe to be recognized are obtained. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output result of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module.

[0008] The target recognition model is used to detect and recognize the image to be recognized, and the detection and recognition result of the recognized image is obtained. The detection and recognition result includes the defect location, defect category, defect level and confidence level of the label box corresponding to the defect in the recognized image.

[0009] Preferably, the acquisition of the target recognition model includes:

[0010] Obtain an image training set of the underground drainage pipe, wherein each image in the image training set carries a manually labeled bounding box;

[0011] The initial recognition model is iteratively trained using the image training set until the current loss function value obtained by the total loss function corresponding to the iterative recognition model is not greater than the maximum value of the total loss function. At this point, the current recognition model is taken as the target recognition model. The total loss function is as follows:

[0012] Loss = Loss obj +λ1Loss reg +Loss cls +Loss level +λ2Loss cls-mc +λ3Loss level-mc ;

[0013] Where Loss represents the total loss function value, Loss obj Loss represents the confidence loss. regLoss represents the location coordinate loss. cls Loss represents the defect category classification loss. level Loss represents the defect level classification loss. cls-mc Loss represents the cross-channel loss value. level-mc λ1, λ2, and λ3 represent the loss values ​​of the hierarchical inter-channel, and λ1, λ2, and λ3 all represent the hyperparameters of the total loss function.

[0014] Preferably, the iterative training of the initial recognition model using the image training set includes:

[0015] During the backpropagation process of each iteration training process, the predicted feature layer set corresponding to each image output by the recognition model of the iteration is first obtained, and then the predicted feature layer set corresponding to each image is concatenated to obtain the prediction result set corresponding to the image.

[0016] For each prediction result in the prediction result set corresponding to each image, the category cost of the prediction result is obtained based on the category soft label of the prediction result, the level cost of the prediction result is obtained based on the level soft label of the prediction result, the position cost of the prediction result is obtained based on the intersection-union ratio of the manually labeled bounding box of the image and the predicted bounding box of the prediction result, and the cost of the feature points of the prediction result in the ground truth bounding box is obtained based on the mask matrix of the feature points of the image in the manually labeled bounding box of the image.

[0017] Based on the category cost, rank cost, location cost, and the cost of the feature point within the ground truth bounding box of the prediction result, the total cost of the prediction result is obtained, and the formula for the total cost is as follows:

[0018] Cost = Cost cls +Cost level +τ1×Cost reg +τ2×Cost in_box ;

[0019] Cost cls Cost represents the category cost. level Cost represents the level cost. reg Cost represents the location cost. in_box The cost of the feature point within the ground truth bounding box is represented by τ1, where τ1 is the location cost weight and τ2 is the cost weight of the feature point within the ground truth bounding box.

[0020] After performing the above operation on each prediction result in the prediction result set corresponding to each image, the total cost of each prediction result in the prediction result set corresponding to each image and multiple total costs of the prediction result set corresponding to each image are obtained.

[0021] For each image, the total costs of the prediction result set are sorted in ascending order, and the prediction results corresponding to the first N total costs are taken to obtain the N positive samples corresponding to the image, where N is an integer greater than 2.

[0022] After performing the above operation on the prediction result set corresponding to each image, N positive samples corresponding to each image are obtained.

[0023] Preferably, after obtaining N positive samples corresponding to each image, the method further includes:

[0024] For each image corresponding to the prediction feature layer set, each prediction feature layer in the prediction feature layer set corresponding to the image is dimensionality reduced to obtain each dimensionality-reduced prediction feature layer. Each dimensionality-reduced prediction feature layer is then copied to obtain N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer. The dimensionality of each dimensionality-reduced prediction feature layer is consistent. The prediction feature layer set corresponding to the image includes a category prediction feature layer and a level prediction feature layer.

[0025] The N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer are grouped according to the number of defect categories to obtain multiple category feature layer sets. Similarly, the N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer are grouped according to the number of defect level categories to obtain multiple level feature layer sets. Each category feature layer set has the same number of feature channels, and each category feature layer set corresponds to one defect category of the initial recognition model. The same applies to each level feature layer set; each level feature layer set corresponds to one defect level of the initial recognition model.

[0026] Based on the multiple category feature sets and the N positive samples corresponding to the image, the category discriminative loss of the image is obtained, and based on the multiple level feature sets and the N positive samples corresponding to the image, the level discriminative loss of the image is obtained.

[0027] After performing the above operations on the prediction feature layer set corresponding to each image, the category discriminant loss and the class discriminant loss for each image are obtained.

[0028] Preferably, for the prediction feature layer set corresponding to each image, after obtaining the multiple grouped category prediction feature layers and the multiple grouped level prediction feature layers, the method further includes:

[0029] Based on the multiple grouped category prediction feature layers, the category diversity loss of the image is obtained;

[0030] The level diversity loss of the image is obtained based on the multiple grouped level prediction feature layers.

[0031] After performing the above operations on the prediction feature layer set corresponding to each image, the category diversity loss and level diversity loss of each image are obtained.

[0032] After obtaining the category diversity loss and the level diversity loss for each image, for each image, the category interchannel loss of the image is obtained based on the category discriminative loss and the category diversity loss of the image, and the level interchannel loss of the image is obtained based on the level discriminative loss and the level diversity loss of the image.

[0033] After performing the above operations on each image, the category interchannel loss and the rank interchannel loss of each image are obtained.

[0034] Preferably, the iterative training of the initial recognition model using the image training set includes:

[0035] For each image, during the forward propagation process of each iteration of training, the image training set is input into the feature extraction network module of the initial recognition model to obtain the feature extraction layer set of the feature extraction network module of the initial recognition model. The feature extraction layer set includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer arranged in sequence. The size of the first feature extraction layer is smaller than the size of the second feature extraction layer, and the size of the second feature extraction layer is smaller than the size of the third feature extraction layer.

[0036] The feature extraction layer set is input into the feature fusion network module of the initial recognition model to obtain a feature fusion layer set, which includes the extended feature fusion layer, and the extended feature fusion layer is obtained based on the first feature extraction layer.

[0037] The feature fusion layer set is input into the output header of the initial recognition model to obtain the predicted feature layer set corresponding to the image;

[0038] Perform the above operation on each image to obtain the prediction feature layer set corresponding to each image.

[0039] Preferably, the step of inputting the feature extraction layer set into the feature fusion network module of the initial recognition model to obtain the feature fusion layer set includes:

[0040] The first feature extraction layer is convolved to obtain the intermediate feature layer of the first feature extraction layer.

[0041] The intermediate feature layer of the first feature extraction layer is upsampled and processed by spatial channel attention mechanism to obtain the upsampled layer of the intermediate feature layer of the first feature extraction layer. Then, the upsampled layer of the intermediate feature layer of the first feature extraction layer and the second feature extraction layer are convolved and connected to obtain the intermediate feature layer of the second feature extraction layer.

[0042] The intermediate feature layer of the second feature extraction layer is upsampled and processed by spatial channel attention mechanism to obtain the upsampled layer of the intermediate feature layer of the second feature extraction layer. Then, the upsampled layer of the intermediate feature layer of the second feature extraction layer and the third feature extraction layer are convolved and connected to obtain the intermediate feature layer of the third feature extraction layer.

[0043] The intermediate feature layer of the third feature extraction layer is used as the first feature fusion layer. The first feature fusion layer is then downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the first feature fusion layer. The downsampled layer of the first feature fusion layer and the intermediate feature layer of the second feature extraction layer are then convolved and connected to obtain the second feature fusion layer.

[0044] The second feature fusion layer is downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the second feature fusion layer. Then, the downsampled layer of the second feature fusion layer and the intermediate feature layer of the first feature extraction layer are convolved and connected to obtain the third feature fusion layer.

[0045] The third feature fusion layer is downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the third feature fusion layer. At the same time, the intermediate feature layer of the first feature extraction layer is downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the intermediate feature layer of the first feature extraction layer. Then, the downsampled layer of the third feature fusion layer and the downsampled layer of the intermediate feature layer of the first feature extraction layer are convolved and connected to obtain the extended feature fusion layer.

[0046] The feature fusion layer set includes: a first feature fusion layer, a second feature fusion layer, and a third feature fusion layer. The size of the first feature fusion layer is larger than the size of the second feature fusion layer, the size of the second feature fusion layer is larger than the size of the third feature fusion layer, and the size of the third feature fusion layer is larger than the size of the extended feature fusion layer.

[0047] Based on the same inventive concept, in a second aspect, the present invention also provides a defect detection device for underground drainage pipes based on fine-grained features, comprising:

[0048] The acquisition module is used to acquire the target recognition model and the image to be recognized of the underground drainage pipe. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output result of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module.

[0049] The detection module is used to perform detection and recognition processing on the image to be recognized through the target recognition model to obtain the detection and recognition result of the recognized image. The detection and recognition result includes the defect location, defect category, defect level and confidence level of the label box corresponding to the defect in the recognized image.

[0050] Based on the same inventive concept, in a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a method for detecting defects in underground drainage pipes based on fine-grained features.

[0051] Based on the same inventive concept, in a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a method for detecting defects in underground drainage pipes based on fine-grained features.

[0052] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages:

[0053] In this embodiment of the invention, firstly, an image of the target recognition model and the underground drainage pipe to be recognized is acquired. The target recognition model includes a feature extraction network module, a feature fusion network module, and an output head module. The output of the output head module includes a specified small-scale feature map, which is formed by an extended feature fusion layer output by the feature fusion network module. Here, the acquired target recognition model adds an extended feature fusion layer O6 to obtain a finer-scale feature layer, which facilitates the target recognition model's ability to extract fine-grained information and improves the recognition accuracy of the target recognition model.

[0054] Next, the target recognition model is used to detect and identify the image to be identified, resulting in the detection and recognition results. These results include the location, category, and level of defects in the identified image, as well as the confidence level of the corresponding bounding box. This approach not only accurately identifies defects in underground drainage pipe images but also precisely classifies easily confused defect categories and even more easily confused defect levels, thus improving overall defect detection performance. Attached Figure Description

[0055] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0056] Figure 1 A schematic flowchart of the steps of the underground drainage pipeline defect detection method based on fine-grained features in an embodiment of the present invention is shown.

[0057] Figure 2 A schematic diagram of the target recognition model in an embodiment of the present invention is shown;

[0058] Figure 3 A schematic diagram of the forward propagation process of the target recognition model in an embodiment of the present invention is shown;

[0059] Figure 4 A chart comparing the results of data experiments between the fine-grained feature-based underground drainage pipe defect detection method and the baseline method in this embodiment of the invention is shown.

[0060] Figure 5 The illustration shows a visual comparison of the underground drainage pipe defect detection method based on fine-grained features and the baseline method in terms of defect “cracks” in an embodiment of the present invention.

[0061] Figure 6 A schematic diagram of a module for a defect detection device for underground drainage pipes based on fine-grained features, according to an embodiment of the present invention, is shown. Detailed Implementation

[0062] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0063] Example 1

[0064] The first embodiment of the present invention provides a method for detecting defects in underground drainage pipes based on fine-grained features, such as... Figure 1 As shown, it includes:

[0065] S101, acquire the target recognition model and the image to be recognized of the underground drainage pipe. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module.

[0066] S102, the target recognition model is used to detect and recognize the image to be recognized, and the detection and recognition results of the recognized image are obtained. The detection and recognition results include the defect location, defect category, defect level and confidence of the corresponding bounding box of the recognized image.

[0067] Below, in conjunction with Figure 1 This embodiment details the specific implementation steps of the underground drainage pipeline defect detection method based on fine-grained features provided in this example:

[0068] First, step S101 is executed to obtain the target recognition model and the image to be recognized of the underground drainage pipe. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module.

[0069] Specifically, the method for obtaining images of underground drainage pipes to be identified is to take pictures of the underground drainage pipes using devices with camera functions such as mobile phones, tablets, or cameras, so that the images of the underground drainage pipes can be detected and identified by the target recognition model of this embodiment.

[0070] like Figure 2 As shown, the target recognition model includes a feature extraction network module, a feature fusion network module, and an output head module connected in sequence. The feature extraction network module extracts multi-scale features from the image to be recognized. The feature fusion network module fuses the extracted multi-scale features and outputs the fused multi-scale features. The output head module uses the fused multi-scale features to obtain the prediction result of the image to be recognized. The prediction result is a three-dimensional array at different scales, such as... Figure 2The diagram shows three-dimensional arrays with scales of 80*80, 40*40, 20*20, and 10*10. The prediction results are then concatenated and processed using NMS (Non-Maximum Suppression) to obtain the detection and recognition results.

[0071] Preferably, the feature extraction network module is CSPDarkNet53 (Cross Stage Paritial Network), the feature fusion network module is PAFPN (Path Aggregation Network with Feature Pyramid Network), and the output head module is a YoloX output head, such as... Figure 2 The YOLO Head shown. The feature extraction network module, feature fusion network module, and output head module can all be selected according to actual needs, choosing appropriate neural network modules. Figure 2 middle,

[0072] The process of obtaining the target recognition model is as follows: First, a training set of images of underground drainage pipes is obtained, in which each image in the training set carries a manually labeled bounding box. Then, the initial recognition model is iteratively trained using the image training set until the current loss function value obtained by the total loss function corresponding to the iterative recognition model is not greater than the maximum value of the total loss function. At this point, the current recognition model is taken as the target recognition model, where the total loss function is as shown in formula (1):

[0073] Loss = Loss obj +λ1Loss reg +Loss cls +Loss level +λ2Loss cls-mc +λ3Loss level-mc (1);

[0074] Where Loss represents the total loss function value, Loss obj Loss represents the confidence loss. reg Loss represents the location coordinate loss. cls Loss represents the defect category classification loss. level Loss represents the defect level classification loss. cls-mc Loss represents the cross-channel loss value. level-mc λ1, λ2, and λ3 represent the inter-channel loss values, and are hyperparameters of the total loss function. λ1, λ2, and λ3 can be set according to actual needs; preferably, λ1 = 5.0, λ2 = 3.0, and λ3 = 8.0.

[0075] Alternatively, after obtaining a training set of images of underground drainage pipes, the initial recognition model can be iteratively trained using the image training set until the number of iterations meets the iteration threshold. The current recognition model that meets the iteration threshold is then determined as the target recognition model.

[0076] It should be noted that the initial recognition model, the iterative recognition model, or the target recognition model all include a feature extraction network module, a feature fusion network module, and an output head module. The values ​​of parameters such as gradients, hyperparameters, and the weights of certain parameters differ in different recognition models.

[0077] The process of iteratively training the initial recognition model using the image training set consists of two processes: the forward propagation process and the backward propagation process for each iteration.

[0078] For each image, such as Figure 3 As shown, during the forward propagation process of each iteration of training, the image training set is first input into the feature extraction network module of the initial recognition model, resulting in the feature extraction layer set of the feature extraction network module of the initial recognition model. This feature extraction layer set includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer arranged sequentially. The size of the first feature extraction layer is smaller than the size of the second feature extraction layer, and the size of the second feature extraction layer is smaller than the size of the third feature extraction layer. Figure 3 In the model, the size of the first feature extraction layer is 20*20*512, the size of the second feature extraction layer is 40*40*256, and the size of the third feature extraction layer is 80*80*128.

[0079] The feature extraction layer set is then input into the feature fusion network module of the initial recognition model to obtain the feature fusion layer set, which includes a first feature fusion layer, a second feature fusion layer, a third fused feature layer, and an extended feature fusion layer. The extended feature fusion layer is obtained based on the first feature extraction layer.

[0080] Specifically, such as Figure 3 As shown, the first feature extraction layer undergoes convolution processing to obtain the intermediate feature layer P5 of the first feature extraction layer, and the size of P5 is 20*20*256.

[0081] The intermediate feature layer P5 of the first feature extraction layer is upsampled and processed using a spatial channel attention mechanism to obtain an upsampled layer of the intermediate feature layer P5 of the first feature extraction layer. Then, the upsampled layer of the intermediate feature layer P5 of the first feature extraction layer is convolved and connected with the second feature extraction layer to obtain the intermediate feature layer P4 of the second feature extraction layer. The size of P4 is 40*40*128.

[0082] The intermediate feature layer P4 of the second feature extraction layer is upsampled and processed using a spatial channel attention mechanism to obtain an upsampled layer of the intermediate feature layer P4 of the second feature extraction layer. Then, the upsampled layer of the intermediate feature layer P4 of the second feature extraction layer and the third feature extraction layer are convolved and connected to obtain the intermediate feature layer P3 of the third feature extraction layer. The size of P3 is 80*80*128.

[0083] The intermediate feature layer P3 of the third feature extraction layer is used as the first feature fusion layer O3. The first feature fusion layer O3 is then downsampled and processed using a spatial channel attention mechanism to obtain a downsampled layer of the first feature fusion layer O3. The downsampled layer of the first feature fusion layer O3 is then convolved and connected with the intermediate feature layer P4 of the second feature extraction layer to obtain the second feature fusion layer O4. The size of O3 is 80*80*128, and the size of O4 is 40*40*256.

[0084] The second feature fusion layer O4 is downsampled and processed using a spatial channel attention mechanism to obtain a downsampled layer of the second feature fusion layer O4. Then, the downsampled layer of the second feature fusion layer O4 and the intermediate feature layer P5 of the first feature extraction layer are convolved and connected to obtain the third feature fusion layer O5. The size of O5 is 20*20*512.

[0085] The third feature fusion layer O5 is downsampled and processed using a spatial channel attention mechanism to obtain a downsampled layer of the third feature fusion layer O5. Simultaneously, the intermediate feature layer P5 of the first feature extraction layer is downsampled and processed using a spatial channel attention mechanism to obtain a downsampled layer P6 of the intermediate feature layer P5 of the first feature extraction layer. Then, the downsampled layer of the third feature fusion layer O5 and the downsampled layer P6 of the intermediate feature layer of the first feature extraction layer are convolved and connected to obtain the extended feature fusion layer O6. The size of P6 is 10*10*512, and the size of O6 is 10*10*1024.

[0086] It should be noted that the size of the first feature fusion layer is larger than the size of the second feature fusion layer, the size of the second feature fusion layer is larger than the size of the third feature fusion layer, and the size of the third feature fusion layer is larger than the size of the extended feature fusion layer.

[0087] In the feature fusion network module, a spatial channel attention mechanism (Convolutional Block Attention Module, CBAM) is incorporated during the upsampling process of the intermediate feature layers corresponding to each feature extraction layer, and during the downsampling process of the feature fusion layer corresponding to each feature extraction layer. This allows the feature fusion network module to pre-focus on useful feature information at both the channel and spatial levels. Furthermore, an extended feature fusion layer O6 is added based on the intermediate feature layers corresponding to the first feature extraction layer and the third feature fusion layer to obtain a finer-scale feature layer. This enhances the target recognition model's ability to extract fine-grained information and improves its recognition accuracy.

[0088] Then, the feature fusion layer set is input into the output head of the initial recognition model to obtain the predicted feature layer set corresponding to the image.

[0089] Specifically, the feature fusion layer set is input into the YOLO output head. The YOLO output head performs convolution processing on the feature fusion layer set and outputs a predicted feature layer set. The predicted feature layer set includes a first predicted feature layer, a second predicted feature layer, a third predicted feature layer, and a fourth predicted feature layer. The size of the first predicted feature layer is larger than that of the second predicted feature layer, the second predicted feature layer is larger than that of the third predicted feature layer, and the third predicted feature layer is larger than that of the fourth predicted feature layer. Each predicted feature layer is actually a three-dimensional array. The dimensions of the first predicted feature layer are 80*80*25, the second predicted feature layer is 40*40*25, the third predicted feature layer is 20*20*25, and the fourth predicted feature layer is 10*10*25. The final combined output can also be regarded as a prediction result set of (80*80+40*40+20*20+10*10)*25=8500*25, that is, 8500 prediction results.

[0090] by Figure 3Taking the feature fusion layer set output by the feature fusion network module as an example, the first feature fusion layer yields the first predicted feature layer, with a size of 80*80*128 and a prediction result of 80*80*25 = 6400*25. The second feature fusion layer yields the second predicted feature layer, with a size of 40*40*256 and a prediction result of 40*40*25 = 1600*25. The third feature fusion layer yields the third predicted feature layer, with a size of 20*20*512 and a prediction result of 20*20*25 = 400*25. The extended feature fusion layer yields the fourth predicted feature layer, with a size of 10*10*1024 and a prediction result of 10*10*25 = 100*25. It should be explained that the 25 here represents 25 output parameters, namely 16 defect categories, 4 defect levels, 4 coordinates of the predicted bounding box (two coordinates at the top left and two at the bottom right), and 1 confidence score of the predicted bounding box. In the following text, defect categories will be referred to as categories, and defect levels as levels. The division of defect categories and defect levels can be set according to actual needs, and therefore the number of defect categories and defect levels can also be set according to actual needs. In this embodiment, it is preferable to have 16 defect categories and 4 defect levels.

[0091] Finally, perform the above operation on each image to obtain the predicted feature layer set corresponding to each image.

[0092] After obtaining the predicted feature set for each image, the backpropagation process for each iteration of training needs to be executed based on the predicted feature set for each image. During the backpropagation process of each iteration, the predicted feature set for each image output by the recognition model is first obtained. Then, the predicted feature sets for each image are concatenated, meaning the first, second, third, and fourth predicted feature layers are concatenated to obtain the prediction result set for the image. The number of prediction result sets is (10*10 + 20*20 + 40*40 + 80*80) = 8500*25.

[0093] Then, the simOTA algorithm is used to calculate the prediction result set for each image, resulting in N positive samples for each image. The total cost matrix of simOTA is branched into four parts, namely the classification cost. cls Location Cost reg Level Cost level Cost of determining whether a feature point is within the ground truth bounding box in_box The formula for the total cost is shown in formula (2).

[0094] Cost = Cost cls +Cost level +τ1×Cost reg +τ2×Cost in_box (2);

[0095] Cost cls Cost represents the category cost. level Cost represents the cost of a level. reg Cost represents the location cost. in _box τ1 represents the cost of a feature point within the ground truth bounding box, and τ2 represents the cost weight of the feature point within the ground truth bounding box. Both τ1 and τ2 can be set according to actual needs. The ground truth bounding box is a manually labeled bounding box.

[0096] Specifically, the process of obtaining N positive samples corresponding to each image is as follows: for each prediction result in the prediction result set corresponding to each image, the category cost of the prediction result is obtained according to the category soft label of the prediction result, as shown in formulas (3)-(5).

[0097]

[0098]

[0099]

[0100] Where i represents the index of the manually labeled bounding box in the image, n represents the total number of manually labeled bounding boxes in the image, j represents the index of the predicted bounding box in the predicted feature layer set of the image, and m represents the total number of predicted bounding boxes in the predicted feature layer set of the image, i.e., m = 8500. This represents the probability that both the i-th manual bounding box and the j-th predicted bounding box belong to a certain category within the defect category. This represents the category hard label of the i-th manual bounding box in the image. This represents the soft label for the category of the i-th predicted bounding box in the image. i,j This represents the intersection-over-union ratio (IOU) between the i-th manual bounding box and the j-th predicted bounding box, i.e., the IOU between the i-th manual bounding box and the j-th predicted bounding box. The category difference factor represents the i-th manual bounding box in the image, which is the difference between the soft label of the i-th manual bounding box and the category probability of each predicted bounding box.

[0101] And based on the level soft label of the prediction result, the level cost of the prediction result is obtained, as shown in formulas (6)-(8).

[0102]

[0103]

[0104]

[0105] in, This represents the probability that both the i-th manual annotation box and the j-th predicted annotation box belong to a certain level of the defect level. This represents the hard label indicating the level of the i-th manual bounding box in the image. This represents the soft label of the i-th predicted bounding box in the image. The rank difference factor represents the rank difference factor of the i-th manual bounding box in the image, which is the difference between the soft label of the i-th manual bounding box and the rank probability of each predicted bounding box.

[0106] The location cost of the prediction result is obtained based on the intersection-union ratio of the manually labeled bounding box of the image and the predicted bounding box of the prediction result, as shown in formula (9).

[0107] Cost reg =-logiou i,j (9).

[0108] And based on the mask matrix of the feature points in the image within the manually labeled bounding box, the cost of the predicted feature points within the actual labeled bounding box is obtained; as shown in formula (10).

[0109] Cost in _ box =~(M) in_box ∩M in_center (10);

[0110] Among them, M in_box and M in_center The intersection of these features represents the mask matrix of the feature points within the manually labeled bounding boxes. First, the entire 640*640 image is divided into 8500 feature points, M... in_box M represents the mask matrix containing feature points within the manually labeled bounding box. in_center This represents the mask matrix of the feature points within a 3-pixel region of the center point of the manually labeled bounding box.

[0111] Next, based on the category cost, level cost, location cost, and cost of feature points within the ground truth bounding box of the prediction results, the total cost of the prediction results is obtained through formula (2).

[0112] After performing the above operation on each prediction result in the prediction result set corresponding to each image, the total cost of each prediction result in the prediction result set corresponding to each image is obtained. That is, one prediction result corresponds to one total cost. There are 8500 prediction results, so the number of total costs for each prediction result is also 8500. In other words, multiple total costs are obtained for the prediction result set corresponding to each image.

[0113] For each image's prediction result set, the multiple total costs are sorted in ascending order. The prediction results corresponding to the top N total costs are then selected to obtain N positive samples for that image, where N is an integer greater than 2. Specifically, since each image corresponds to one prediction result set, there are 8500 prediction results, resulting in 8500 total costs. These 8500 total costs are sorted in ascending order, and the top N smallest total costs are selected. These represent the prediction results corresponding to each of the top N total costs, resulting in N prediction results, which are then N positive samples for that image. After performing the above operation on the prediction result set for each image, N positive samples are obtained for each image.

[0114] In this embodiment, to address the issue of inconsistent defect categories and defect levels in images, simOTA employs a strategy of replacing hard labels with soft labels and adopts a soft label allocation mechanism to enhance the noise resistance of the neural network during training, thereby improving the recognition accuracy and defect detection performance of the target recognition model.

[0115] After obtaining N positive samples for each image, for the prediction feature layer set corresponding to each image, the dimensionality of each prediction feature layer in the prediction feature layer set corresponding to the image is first reduced to obtain each dimensionality-reduced prediction feature layer. Then, each dimensionality-reduced prediction feature layer is copied to obtain N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer. The dimensionality of each dimensionality-reduced prediction feature layer is consistent. The prediction feature layer set corresponding to the image includes a category prediction feature layer and a level prediction feature layer.

[0116] Specifically, each of the four prediction feature layers in the prediction feature layer set undergoes dimensionality reduction, resulting in four dimensionality-reduced prediction feature layers with consistent dimensions. Then, each of these four dimensionality-reduced prediction feature layers is copied N times, resulting in N dimensionality-reduced prediction feature layers for each layer. The purpose of this copying process is to ensure that the number of each dimensionality-reduced prediction feature layer matches the number of N positive samples, facilitating the calculation of the loss function.

[0117] Taking a predictive feature layer with a scale of 80*80 as an example, the specific process of dimensionality reduction is as follows: before dimensionality reduction, the dimension of the predictive feature layer is F. 1×128×W×H First, the dimensionality of the prediction feature layer is reduced from 128 channels to C channels using a 1×1 convolution kernel. C can be set according to actual needs, and C satisfies formula (11). After dimensionality reduction, the dimension of the prediction feature layer is F. 1×C×W×H .

[0118] C = c × ξ (11);

[0119] Where c represents the number of categories, such as 16 defect categories and 4 defect level categories in this embodiment, and ξ represents the number of channels allocated to each category. Based on experience, it is generally set to be greater than 2, but it can also be set according to actual needs.

[0120] The dimensionality-reduced prediction feature layer F 1×C×W×H By replicating N times, N predicted feature layers after dimensionality reduction are obtained, i.e., F. N ×C×W×H It should be noted that, in order to form dimensionality-consistent predicted feature layers with different scales, the convolution kernels used in the dimensionality reduction process for predicted feature layers of different scales can be set according to actual needs. Dimensionality consistency of each dimensionality-reduced predicted feature layer means that the number of C channels in each dimensionality-reduced predicted feature layer is consistent.

[0121] After obtaining N dimensionality-reduced predictive feature layers corresponding to each dimensionality-reduced predictive feature layer, the N dimensionality-reduced predictive feature layers corresponding to each dimensionality-reduced predictive feature layer are grouped according to the number of defect categories to obtain multiple category feature layer sets. Similarly, the N dimensionality-reduced predictive feature layers corresponding to each dimensionality-reduced predictive feature layer are grouped according to the number of defect level categories to obtain multiple level feature layer sets. Each category feature layer set has the same number of feature channels and corresponds to one defect category in the initial recognition model. Each level feature layer set has the same number of feature channels and corresponds to one defect level in the initial recognition model.

[0122] Specifically, the predicted feature layer F with N dimensions reduced to 80*80*C is used. N×C×W×H For example, assuming C is 64, the number of defect categories is c1 = 16, and the number of defect levels is c2 = 4. Following formula (1), the number of channels for each defect category is ξ1 = 4, and the number of channels for each defect level is ξ2 = 16. These N dimensionality-reduced prediction feature layers are grouped according to the number of defect categories to obtain multiple category feature layer sets F. a , 'a' is the index of the category feature layer set. The specific grouping is shown in formula (12). A category feature layer set includes N 80*80*ξ1 dimensionality-reduced prediction feature layers.

[0123]

[0124] in, This is the predicted feature layer after dimensionality reduction. The subscript indicates only the channel number.

[0125] like And so on. In this way, each set of feature layers corresponds to a defect category.

[0126] The grouping principle based on the number of defect level categories is the same as the grouping principle based on the number of defect level categories. The N dimensionality-reduced predicted feature layers are grouped according to the number of defect level categories to obtain multiple feature layer sets F. b , b is the index of the hierarchical feature layer set. The specific grouping is shown in formula (13). A hierarchical feature layer set includes N dimensionality-reduced prediction feature layers of 80*80*ξ2.

[0127]

[0128] in, This is the predicted feature layer after dimensionality reduction. The subscript indicates only the channel number.

[0129] like And so on. In this way, each level of feature layer corresponds to a defect level.

[0130] After obtaining multiple class feature sets and multiple level feature sets, the class discriminative loss of the image is obtained based on the multiple class feature sets and N positive samples corresponding to the image, and the level discriminative loss of the image is obtained based on the multiple level feature sets and N positive samples corresponding to the image.

[0131] Specifically, the class discriminative loss of the image is obtained by calculating the N positive samples corresponding to the multiple class feature layers and the image using formulas (14)-(15).

[0132]

[0133]

[0134] in, F is the class label of the r-th positive sample. p It is the Pth category feature layer, W S H SIt is the product of the length and width of a certain feature layer, where c1 is the number of defect categories, in F P,Q,k In this context, P is the index of the category feature layer set, Q is the index of the number of channels assigned to a category, and k is the number of segments to divide the feature map of each dimensionality-reduced predictive feature layer into W. S H S The index of one of the feature points. A feature point is actually a pixel.

[0135] M P =diag(Mask) P Mask P ∈R ξ ξ∈{ξ1,ξ2} is a random 0-1 mask matrix, with ξ / 2 zero matrices and ξ / 2 one matrices randomly set, M P M is the diagonal matrix of the mask matrix. P ·F P,Q,k This represents a dropout operation along the channel dimension, randomly discarding half of the channels to make the remaining channels more focused, thereby improving the fine-grained capability of each channel. The feature map dimensions before and after this operation are c1×ξ×W×H and c1×ξ / 2×W×H, respectively.

[0136] This indicates a cross-channel max pooling operation, which places the maximum response values ​​of all channels onto a single feature map. Average pooling is not used because it suppresses peak information in the feature channels, hindering the concentration of attention in local regions. Max pooling, on the other hand, preserves this attention, which is beneficial for fine-grained classification. The feature map dimensions before and after this operation are c1×ξ / 2×W×H and c1×W×H, respectively.

[0137] This represents a global average pooling operation, equivalent to spatial dimensionality reduction, used to calculate the average response for each feature channel. The feature map dimensions before and after this operation are c1×W×H and c1×1, respectively.

[0138] This represents the normalized exponential function softmax operation, which yields c1 probability scores for defect categories. These probability scores are then used to calculate the cross-entropy loss for each positive target in the image, and the results are summed.

[0139] The image's level discriminative loss is obtained by calculating multiple level feature sets and N positive samples corresponding to the image using formulas (16)-(17).

[0140]

[0141]

[0142] in, F is the rank label of the r-th positive sample. A It is the Ath level feature layer, W S H S It is the product of the length and width of a certain feature layer, and c2 is the number of defect level categories. A,B,k In this context, A is the index of the grade feature layer set, and B is the index of the number of channels assigned to a grade.

[0143] M A =diag(Mask) A Mask A ∈R ξ ξ∈{ξ1,ξ2} is a random 0-1 mask matrix, with ξ / 2 zero matrices and ξ / 2 one matrices randomly set, M A M is the diagonal matrix of the mask matrix. A ·F A,B,k This represents a dropout operation along the channel dimension, randomly discarding half of the channels to make the remaining channels more focused, thereby improving the fine-grained capability of each channel. The feature map dimensions before and after this operation are c2×ξ×W×H and c2×ξ / 2×W×H, respectively.

[0144] This indicates a cross-channel max pooling operation, which places the maximum response values ​​of all channels onto a single feature map. Average pooling is not used because it suppresses peak information in the feature channels, hindering the concentration of attention in local regions. Max pooling, on the other hand, preserves this attention, which is beneficial for fine-grained classification. The feature map dimensions before and after this operation are c2×ξ / 2×W×H and c2×W×H, respectively.

[0145] This represents a global average pooling operation, equivalent to spatial dimensionality reduction, used to calculate the average response for each feature channel. The feature map dimensions before and after this operation are c2×W×H and c2×1, respectively.

[0146] This represents a softmax operation, yielding probability scores for c2 defect level categories. These probability scores are then used to calculate the cross-entropy loss for each target in the image, and the results are summed.

[0147] Then, after performing the above operation on the predicted feature layer set corresponding to each image, the class discriminative loss and the class discriminative loss for each image are obtained.

[0148] It should be noted that both the image category discriminative loss and the image grade discriminative loss are essentially calculating discriminative loss, so their formulas are based on the same principle. Since one is a category discriminative loss concerning defect categories and the other is a grade discriminative loss concerning defect grades, different letters are used in the formulas to represent different parameters.

[0149] In this embodiment, discriminative loss is used to set the channels in the feature map to a category alignment mode, that is, to distinguish the category from other categories by using channels corresponding to different categories, so that the trained target recognition model can accurately identify defects in underground drainage pipe images, and can also accurately classify easily confused defect categories and more easily confused defect levels.

[0150] For each image, after obtaining multiple grouped category prediction feature layers and multiple grouped level prediction feature layers, the category diversity loss of the image is obtained based on the multiple grouped category prediction feature layers, as shown in formulas (18)-(19); the level diversity loss of the image is obtained based on the multiple grouped level prediction feature layers, as shown in formulas (20)-(21).

[0151]

[0152]

[0153]

[0154]

[0155] Where, k and The definitions are consistent, except that the same letter cannot be used to represent them in advanced mathematical expressions. and This represents the normalization operation for spatial dimensions. and This represents the cross-channel max pooling operation, which has the same effect as in discriminative loss. h(F) P ) and h(F A The summation () represents the cross-channel max pooling result for each category. Each target is summed and averaged relative to each category, and the results for each target in the image are summed to obtain the classification diversity loss for that image. and loss of hierarchical diversity and Both are better the larger they are; a larger value indicates that the feature maps of the ξ channels in each category focus on more dispersed regions. This loss is equivalent to cross-channel decorrelation, making the model more effective at capturing fine-grained details by starting from different salient regions.

[0156] After performing the above operations on the predicted feature layer set corresponding to each image, we obtain the category diversity loss and level diversity loss for each image.

[0157] In this embodiment, the diversity loss is used to calculate the total similarity of all channels, so that different feature channels of each category focus on different regions of the image, rather than having all channels focus on the most discriminative region, thereby diversifying the feature channels of each group and reducing redundant information.

[0158] After obtaining the category diversity loss and the level diversity loss for each image, for each image, the category cross-channel loss of the image is obtained based on the image's category discriminative loss and category diversity loss, as shown in formula (22), and the level cross-channel loss of the image is obtained based on the image's level discriminative loss and level diversity loss, as shown in formula (23).

[0159]

[0160]

[0161] η1 and η2 are preset hyperparameters, with default values ​​of 1.5 and 20.0, respectively, but can be set according to actual needs. This represents the averaging operation across the scale dimensions. After obtaining the cross-channel losses for feature maps at scales of 80×80, 40×40, 20×20, and 10×10, these losses are summed and averaged to obtain the final class cross-channel loss (Loss). cls-mc Inter-level channel loss level-mc .

[0162] After performing the above operations on each image, the category interchannel loss and the ranking interchannel loss for each image are obtained. Then, based on the category interchannel loss and the ranking interchannel loss for each image, the corresponding loss function value for each image is obtained.

[0163] In this embodiment, the category interchannel loss of each image is obtained by using the category discriminative loss and category diversity loss of each image, and the level interchannel loss of each image is obtained by using the level discriminative loss and level diversity loss of each image. This facilitates the accurate identification of defects in underground drainage pipe images, and can also accurately classify easily confused defect categories and more easily confused defect levels, thereby improving the overall defect detection performance and enhancing the accuracy of defect detection and identification results.

[0164] After iteratively training the initial recognition model using an image training set until a target recognition model is obtained, the model needs to be validated. This embodiment provides a novel loss function, Multi-scale MC Loss, which is applied to the convolutional feature maps before the YoloX category and rank output heads. During loss backpropagation, the model can focus on all local discriminative regions different for each category, while the simultaneous use of cross-entropy loss allows the model to also focus on global discriminative features. The organic fusion of these two loss methods can significantly enhance the model's attention to fine-grained features of pipeline defects.

[0165] Ablation experiments were conducted to assess the effectiveness of each module of the target recognition model. For example... Figure 4 As shown, the Baseline is the basic model used for comparison with the target recognition model in this embodiment, and can also be called the baseline method. This basic model includes a feature extraction module, a feature fusion module, and an output head. The feature fusion module of the basic module does not have the extended feature fusion layer O6 of this target recognition model, but instead has O3-O5. This basic model does not use soft labels.

[0166] exist Figure 4 The values ​​in the Baseline row represent the Baseline model's representation of the defect presence / absence index (mEF1), defect category index (mCF1), and defect level index (mLF1), respectively. Rows three through seven represent the indexes using only one of these factors in the Baseline model. For example, row three, PAFPN one more layer, indicates that the Baseline model only uses the factor of adding an extended feature fusion layer O6 to the feature fusion module of the base model. Row eight, Result, represents the final result of combining the three factors from rows three, four, and seven in the Baseline model.

[0167] Therefore, in the third row, which represents the factor of adding an additional downsampling layer to the feature extraction network in the baseline model, improvements of 0.1%, 0.5%, and 0.4% were achieved in the defect presence / absence mEF1, defect category mCF1, and defect level mLF1 indicators, respectively. In the fourth row, changing the label assignment strategy from SimOTA to Soft SimOTA improved the three indicators by 0.7%, 0.6%, and 0.2%, respectively. In the fifth row, it shows that after adopting the category cross-channel loss factor, i.e., adding Multi-scale MC Loss before the classification output head, the first two indicators improved by 0.3% and 0.6%, respectively. In the sixth row, it shows that after adopting the level cross-channel loss and adding Multi-scale MC Loss before the level output head, the three indicators improved by 0.3%, 0.1%, and 0.2%, respectively. In the seventh row, it is shown that after incorporating category cross-channel loss and rank cross-channel loss factors, i.e., after adding Multi-scale MC Loss to both the classification output head and the rank output head, the three metrics improved by 0.3%, 1.0%, and 0.6%, respectively. In the eighth row, the object recognition model improved by 0.7%, 1.2%, and 0.9% on the three metrics, respectively.

[0168] To achieve a more intuitive representation, the prediction maps and heatmaps obtained from the Baseline model were visually compared with those obtained from this target identification model (which incorporates fine-grained methods). For the more difficult-to-distinguish defect category, a visual comparison of the "fracture" defect is shown below. Figure 5 .exist Figure 5 In the visualization comparison of the defect "cracks", (a), (b), and (c) are three defect examples. The first row of images shows the labeled bounding box of the defect example. The second and fourth rows of images show the prediction results and prediction heatmaps of the baseline method for the defect example. The third and fifth rows of images show the prediction results and prediction heatmaps of the defect example after adding the fine-grained method.

[0169] exist Figure 5 In the image, the first row represents the Ground Truth (GT) bounding boxes. The second and third rows represent the predicted images after the baseline method and the addition of the fine-grained method, respectively. It's clear that previously inconspicuous fracture defects are now largely noticed by the model after the addition of the fine-grained method. The fourth and fifth rows represent the heatmaps of the predictions after the baseline method and the addition of the fine-grained method, respectively. Larger grayscale values ​​indicate a higher probability of defect prediction, and the visualization also shows that the model's ability to focus on "fracture" defects has indeed been improved to some extent.

[0170] Next, step S102 is executed, in which the target recognition model is used to detect and recognize the image to be recognized, and the detection and recognition results of the recognized image are obtained. The detection and recognition results include the defect location, defect category, defect level and confidence of the corresponding bounding box of the recognized image.

[0171] Specifically, the detection and recognition process is the process by which the target recognition model detects and recognizes the image to be recognized; essentially, it is the forward propagation process of the target recognition model. The recognized image is the image to be recognized that has already passed the detection and recognition by the target recognition model.

[0172] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages:

[0173] In this embodiment of the invention, firstly, an image of the target recognition model and the underground drainage pipe to be recognized is acquired. The target recognition model includes a feature extraction network module, a feature fusion network module, and an output head module. The output of the output head module includes a specified small-scale feature map, which is formed by an extended feature fusion layer output by the feature fusion network module. Here, the acquired target recognition model adds an extended feature fusion layer O6 to obtain a finer-scale feature layer, which facilitates the target recognition model's ability to extract fine-grained information and improves the recognition accuracy of the target recognition model.

[0174] Next, the target recognition model is used to detect and identify the image to be identified, resulting in the detection and recognition results. These results include the location, category, and level of defects in the identified image, as well as the confidence level of the corresponding bounding box. This approach not only accurately identifies defects in underground drainage pipe images but also precisely classifies easily confused defect categories and even more easily confused defect levels, thus improving overall defect detection performance.

[0175] Example 2

[0176] Based on the same inventive concept, the second embodiment of the present invention also provides a defect detection device for underground drainage pipes based on fine-grained features, such as... Figure 6 As shown, it includes:

[0177] The acquisition module 201 is used to acquire the target recognition model and the image to be recognized of the underground drainage pipe. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output result of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module.

[0178] The detection module 202 is used to perform detection and recognition processing on the image to be recognized through the target recognition model to obtain the detection and recognition result of the recognized image. The detection and recognition result includes the defect location, defect category, defect level and confidence level of the label box corresponding to the defect in the recognized image.

[0179] As an optional embodiment, obtaining the target recognition model includes:

[0180] Obtain an image training set of the underground drainage pipe, wherein each image in the image training set carries a manually labeled bounding box;

[0181] The initial recognition model is iteratively trained using the image training set until the current loss function value obtained by the total loss function corresponding to the iterative recognition model is not greater than the maximum value of the total loss function. At this point, the current recognition model is taken as the target recognition model. The total loss function is as follows:

[0182] Loss = Loss obj +λ1Loss reg +Loss cls +Loss level +λ2Loss cls-mc +λ3Loss level-mc ;

[0183] Where Loss represents the total loss function value, Loss obj Loss represents the confidence loss. reg Loss represents the location coordinate loss. cls Loss represents the defect category classification loss. level Loss represents the defect level classification loss. cls-mc Loss represents the cross-channel loss value. level-mc λ1, λ2, and λ3 represent the loss values ​​of the hierarchical inter-channel, and λ1, λ2, and λ3 all represent the hyperparameters of the total loss function.

[0184] Since the fine-grained feature-based underground drainage pipeline defect detection device described in this embodiment is the same device used to implement the fine-grained feature-based underground drainage pipeline defect detection method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation and various variations of the fine-grained feature-based underground drainage pipeline defect detection device in this embodiment based on the fine-grained feature-based underground drainage pipeline defect detection method described in Embodiment 1 of this application. Therefore, how this fine-grained feature-based underground drainage pipeline defect detection device implements the method in Embodiment 1 of this application will not be described in detail here. Any device used by those skilled in the art to implement the fine-grained feature-based underground drainage pipeline defect detection method in Embodiment 1 of this application falls within the scope of protection of this application.

[0185] Example 3

[0186] Based on the same inventive concept, the third embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above-described methods for detecting defects in underground drainage pipes based on fine-grained features.

[0187] Example 4

[0188] Based on the same inventive concept, the fourth embodiment of the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods of the underground drainage pipeline defect detection method based on fine-grained features described in the first embodiment above.

[0189] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0191] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for detecting defects in underground drainage pipes based on fine-grained features, characterized in that, include: The target recognition model and the image of the underground drainage pipe to be recognized are obtained. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output result of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module. The target recognition model is used to detect and recognize the image to be recognized to obtain the detection and recognition result of the recognized image. The detection and recognition result includes the defect location, defect category, defect level and confidence level of the label box corresponding to the defect in the recognized image. The acquisition of the target recognition model includes: Obtain an image training set of the underground drainage pipe, wherein each image in the image training set carries a manually labeled bounding box; The initial recognition model is iteratively trained using the image training set until the current loss function value obtained by the total loss function corresponding to the iterative recognition model is not greater than the threshold of the total loss function. Then, the current recognition model is used as the target recognition model. The iterative training of the initial recognition model using the image training set includes: For each image, during the forward propagation process of each iteration of training, the image training set is input into the feature extraction network module of the initial recognition model to obtain the feature extraction layer set of the feature extraction network module of the initial recognition model. The feature extraction layer set includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer arranged in sequence. The size of the first feature extraction layer is smaller than the size of the second feature extraction layer, and the size of the second feature extraction layer is smaller than the size of the third feature extraction layer. The feature extraction layer set is input into the feature fusion network module of the initial recognition model to obtain a feature fusion layer set, which includes the extended feature fusion layer, and the extended feature fusion layer is obtained based on the first feature extraction layer. The feature fusion layer set is input into the output header of the initial recognition model to obtain the predicted feature layer set corresponding to the image; Perform the above operation on each image to obtain the prediction feature layer set corresponding to each image.

2. The method as described in claim 1, characterized in that, The total loss function is as follows: ； Where Loss represents the total loss function value. Indicates confidence loss. Indicates the loss of position coordinates. This represents the loss for defect category classification. Indicates the loss due to defect level classification. This represents the cross-channel loss value. This represents the inter-channel loss value. , and Both represent the hyperparameters of the total loss function.

3. The method as described in claim 2, characterized in that, The iterative training of the initial recognition model using the image training set includes: During the backpropagation process of each iteration training process, the predicted feature layer set corresponding to each image output by the recognition model of the iteration is first obtained, and then the predicted feature layer set corresponding to each image is concatenated to obtain the prediction result set corresponding to the image. For each prediction result in the prediction result set corresponding to each image, the category cost of the prediction result is obtained based on the category soft label of the prediction result, the level cost of the prediction result is obtained based on the level soft label of the prediction result, the position cost of the prediction result is obtained based on the intersection-union ratio of the manually labeled bounding box of the image and the predicted bounding box of the prediction result, and the cost of the feature points of the prediction result in the ground truth bounding box is obtained based on the mask matrix of the feature points of the image in the manually labeled bounding box of the image. Based on the category cost, rank cost, location cost, and the cost of the feature point within the ground truth bounding box of the prediction result, the total cost of the prediction result is obtained, and the formula for the total cost is as follows: ； in, Indicates the cost of the category, This indicates the cost of the stated level. This represents the location cost. This represents the cost of the feature point being within the ground truth bounding box. It is the position cost weight. It is the cost weight of the feature point within the ground truth bounding box; After performing the above operation on each prediction result in the prediction result set corresponding to each image, the total cost of each prediction result in the prediction result set corresponding to each image and multiple total costs of the prediction result set corresponding to each image are obtained. For each image, the total costs of the prediction result set are sorted in ascending order, and the prediction results corresponding to the first N total costs are taken to obtain the N positive samples corresponding to the image, where N is an integer greater than 2. After performing the above operation on the prediction result set corresponding to each image, N positive samples corresponding to each image are obtained.

4. The method as described in claim 3, characterized in that, After obtaining N positive samples corresponding to each image, the process also includes: For each image corresponding to the prediction feature layer set, each prediction feature layer in the prediction feature layer set corresponding to the image is dimensionality reduced to obtain each dimensionality-reduced prediction feature layer. Each dimensionality-reduced prediction feature layer is then copied to obtain N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer. The dimensionality of each dimensionality-reduced prediction feature layer is consistent. The prediction feature layer set corresponding to the image includes a category prediction feature layer and a level prediction feature layer. The N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer are grouped according to the number of defect categories to obtain multiple category feature layer sets. Similarly, the N dimensionality-reduced prediction feature layers corresponding to each dimensionality-reduced prediction feature layer are grouped according to the number of defect level categories to obtain multiple level feature layer sets. Each category feature layer set has the same number of feature channels, and each category feature layer set corresponds to a defect category in the initial recognition model. The same applies to each level feature layer set; each level feature layer set corresponds to a defect level in the initial recognition model. Based on the multiple category feature layers and the N positive samples corresponding to the image, the category discriminative loss of the image is obtained, and based on the multiple level feature layers and the N positive samples corresponding to the image, the level discriminative loss of the image is obtained. After performing the above operations on the prediction feature layer set corresponding to each image, the category discriminant loss and the class discriminant loss for each image are obtained.

5. The method as described in claim 4, characterized in that, For each image corresponding to the prediction feature layer set, after obtaining the multiple grouped category prediction feature layers and the multiple grouped level prediction feature layers, the method further includes: Based on the multiple grouped category prediction feature layers, the category diversity loss of the image is obtained; The level diversity loss of the image is obtained based on the multiple grouped level prediction feature layers. After performing the above operations on the prediction feature layer set corresponding to each image, the category diversity loss and level diversity loss of each image are obtained. After obtaining the category diversity loss and the level diversity loss for each image, for each image, the category interchannel loss of the image is obtained based on the category discriminative loss and the category diversity loss of the image, and the level interchannel loss of the image is obtained based on the level discriminative loss and the level diversity loss of the image. After performing the above operations on each image, the category interchannel loss and the rank interchannel loss of each image are obtained.

6. The method as described in claim 1, characterized in that, The step of inputting the feature extraction layer set into the feature fusion network module of the initial recognition model to obtain the feature fusion layer set includes: The first feature extraction layer is convolved to obtain the intermediate feature layer of the first feature extraction layer. The intermediate feature layer of the first feature extraction layer is upsampled and processed by spatial channel attention mechanism to obtain the upsampled layer of the intermediate feature layer of the first feature extraction layer. Then, the upsampled layer of the intermediate feature layer of the first feature extraction layer and the second feature extraction layer are convolved and connected to obtain the intermediate feature layer of the second feature extraction layer. The intermediate feature layer of the second feature extraction layer is upsampled and processed by spatial channel attention mechanism to obtain the upsampled layer of the intermediate feature layer of the second feature extraction layer. Then, the upsampled layer of the intermediate feature layer of the second feature extraction layer and the third feature extraction layer are convolved and connected to obtain the intermediate feature layer of the third feature extraction layer. The intermediate feature layer of the third feature extraction layer is used as the first feature fusion layer. The first feature fusion layer is then downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the first feature fusion layer. The downsampled layer of the first feature fusion layer and the intermediate feature layer of the second feature extraction layer are then convolved and connected to obtain the second feature fusion layer. The second feature fusion layer is downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the second feature fusion layer. Then, the downsampled layer of the second feature fusion layer and the intermediate feature layer of the first feature extraction layer are convolved and connected to obtain the third feature fusion layer. The third feature fusion layer is downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the third feature fusion layer. At the same time, the intermediate feature layer of the first feature extraction layer is downsampled and processed by spatial channel attention mechanism to obtain the downsampled layer of the intermediate feature layer of the first feature extraction layer. Then, the downsampled layer of the third feature fusion layer and the downsampled layer of the intermediate feature layer of the first feature extraction layer are convolved and connected to obtain the extended feature fusion layer. The feature fusion layer set includes: a first feature fusion layer, a second feature fusion layer, and a third feature fusion layer. The size of the first feature fusion layer is larger than the size of the second feature fusion layer, the size of the second feature fusion layer is larger than the size of the third feature fusion layer, and the size of the third feature fusion layer is larger than the size of the extended feature fusion layer.

7. A defect detection device for underground drainage pipes based on fine-grained features, characterized in that, include: The acquisition module is used to acquire the target recognition model and the image to be recognized of the underground drainage pipe. The target recognition model includes a feature extraction network module, a feature fusion network module and an output head module. The output result of the output head module includes a specified small-scale feature map, which is formed by the extended feature fusion layer output by the feature fusion network module. The detection module is used to perform detection and recognition processing on the image to be recognized through the target recognition model to obtain the detection and recognition result of the recognized image, wherein the detection and recognition result includes the defect location, defect category, defect level and confidence level of the label box corresponding to the defect in the recognized image; The acquisition of the target recognition model includes: Obtain an image training set of the underground drainage pipe, wherein each image in the image training set carries a manually labeled bounding box; The initial recognition model is iteratively trained using the image training set until the current loss function value obtained by the total loss function corresponding to the iterative recognition model is not greater than the threshold of the total loss function. Then, the current recognition model is used as the target recognition model. The iterative training of the initial recognition model using the image training set includes: For each image, during the forward propagation process of each iteration of training, the image training set is input into the feature extraction network module of the initial recognition model to obtain the feature extraction layer set of the feature extraction network module of the initial recognition model. The feature extraction layer set includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer arranged in sequence. The size of the first feature extraction layer is smaller than the size of the second feature extraction layer, and the size of the second feature extraction layer is smaller than the size of the third feature extraction layer. The feature extraction layer set is input into the feature fusion network module of the initial recognition model to obtain a feature fusion layer set, which includes the extended feature fusion layer, and the extended feature fusion layer is obtained based on the first feature extraction layer. The feature fusion layer set is input into the output header of the initial recognition model to obtain the predicted feature layer set corresponding to the image; Perform the above operation on each image to obtain the prediction feature layer set corresponding to each image.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-6.

Images