A sewer network defect detection method based on multi-label zero-shot learning

Abstract

The application discloses a sewer network defect detection method based on multi-label zero-shot learning, and the method comprises the following steps: generating a defect description corresponding to each pipeline defect category through a large language model; using a representation guide module to perform feature extraction and field adaptation on a pipeline inner wall image and the defect description, so as to obtain global and local image features and defect detailed description text features; comprehensively considering the global and local image features of the image, respectively calculating global and local matching scores of the image and the detailed description text features, and fusing the global and local matching scores to obtain an initial defect prediction score; constructing a semantic relation adjacency matrix for displaying the relation between different defect categories; and using the semantic relation adjacency matrix between the categories to correct the initial prediction score, so as to obtain a final prediction score of each defect category. The method disclosed by the application realizes knowledge transfer from known defects to unknown defects by constructing a guide-fusion-correction network, and effectively solves the problem of recognizing unknown defect types.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method for detecting defects in sewer networks based on multi-label zero-shot learning. Background Technology

[0002] Accurate detection and identification of pipeline defects are fundamental to pipeline health assessment and repair decisions. However, current supervised learning-based detection methods suffer from two major bottlenecks: first, the models are limited by the training dataset and have poor ability to identify the constantly emerging "unknown" defect types in reality, posing a risk of missed detections; second, model updates and maintenance are costly, requiring repeated data collection, labeling, and training to cover new defects, making it difficult to adapt to dynamically changing operational needs.

[0003] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for detecting defects in sewage pipe networks based on multi-label zero-shot learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for detecting defects in sewer networks based on multi-label zero-shot learning includes the following steps:

[0007] Step S1: Generate a defect description corresponding to each pipeline defect category using a large language model;

[0008] Step S2: Use the representation guidance module to extract features and adapt the pipeline inner wall image and defect description to obtain global and local image features and detailed defect description text features;

[0009] Step S3: Combine the global and local image features of the image, calculate the global and local matching scores with the features of the detailed description text, and fuse them to obtain the initial defect prediction score;

[0010] Step S4: Construct a semantic relation adjacency matrix using the generated defect brief phrases to display the relationships between different defect categories;

[0011] Step S5: The initial prediction scores are corrected using the semantic relationship adjacency matrix between categories to obtain the final prediction score for each defect category.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] Step S11: Generate a detailed description of the defect corresponding to each pipeline defect category. and brief phrases ;

[0014] Step S12, use a text encoder to process the brief phrases. Encode the data to obtain brief phrase text features for all defect categories. .

[0015] Furthermore, step S2 specifically includes the following steps:

[0016] Step S21, obtain the image of the inner wall of the pipe. and detailed description of defects in pipeline defect description Input the image encoder and text encoder respectively to obtain the original image features and the original text features respectively;

[0017] Step S22: Use the representation guidance module to perform sewer pipe domain adaptation on the original image features and original text features to obtain the adapted global image features. Local image features and detailed description of defects and text features .

[0018] Furthermore, in step S22, the guiding module modifies the features output by the Transformer encoder through linear transformation, GeLU activation function, Dropout operation, and residual connection, as calculated below:

[0019]

[0020]

[0021] In the formula, Features obtained through representation guidance; The raw image features obtained after the encoder, Given the input features; To adapt to the image and text features of pipeline scenarios; and It is a linear transformation layer; It is a non-linear activation function; This is a discard operation.

[0022] Furthermore, step S3 specifically includes the following steps:

[0023] Step S31: Calculate the global matching score using global image features and detailed defect description text features. ;

[0024]

[0025] In the formula, For global image features; Provide a detailed description of the textual features of the defect; It is the transpose operator;

[0026] Step S32: Calculate the affinity weights between local image features and the features of the detailed description of defects in the text, and aggregate them through a score aggregation network to obtain local matching scores. ;

[0027]

[0028]

[0029] In the formula, Weighted by affinity; This is the normalization function; For local image features; For a multilayer perceptron; T is the transpose operator;

[0030] Step S33: Aggregate the global matching score and the local matching score to obtain the initial prediction score;

[0031]

[0032] In the formula, The initial predicted score.

[0033] Furthermore, step S4 specifically includes the following steps:

[0034] Step S41, use the brief phrase text features in the pipeline defect description. Calculate the cosine similarity matrix between defect categories and filter them using a preset threshold to construct an initial semantic relationship adjacency matrix. ;

[0035]

[0036]

[0037] In the formula, This is the cosine similarity matrix; For the text features of brief phrases beg Norm; T is the transpose operator; Cosine similarity matrix The Middle line, number Column similarity values; The preset similarity threshold; The initial semantic relation adjacency matrix The Middle line, number The value of the column; , , The total number of defect categories;

[0038] Step S42: Process the initial semantic relation adjacency matrix and calculate the final semantic relation adjacency matrix. ;

[0039]

[0040]

[0041] In the formula, It is the identity matrix; This is an adjacency matrix with self-connected semantic relationships; Let be a diagonal matrix whose diagonal elements satisfy... .

[0042] Furthermore, step S5 specifically includes the following steps:

[0043] Step S51: Based on the initial predicted score and semantic relation adjacency matrix, use a graph convolutional network to predict the corrected score. ;

[0044]

[0045] In the formula, This is a semantic relation adjacency matrix; It is a non-linear activation function; The initial predicted score; and These are learnable weights;

[0046] Step S52: Aggregate the initial predicted scores and the corrected scores to obtain the final predicted score for each defect category;

[0047]

[0048] In the formula, This is the final predicted score.

[0049] Furthermore, during model training, a loss function combining contrastive learning and ranking constraints is used for model optimization, including the following steps:

[0050] Step S61: Calculate the multimodal contrast loss function. ;

[0051]

[0052] In the formula, This is the set of indices for all positive sample pairs within the same batch. ; This represents the total number of all positive sample pairs. The sample index for positive sample pairs. ; The number of samples; The class index for positive sample pairs. ; Number of categories; Used to specify the Does the nth sample contain the nth sample? One defect category; It is an exponential function; It is the natural logarithm; Indicates the first The sample at the th Final predicted scores for each defect category; Temperature scaling factor to control the intensity of contrastive learning; Indicates the first The sample at the th Final predicted scores for each defect category; To iterate through all samples, the sample index is... ; To iterate through the category indices of all categories, ;

[0053] Step S62, calculate the sorting loss function. ;

[0054]

[0055] In the formula, This is the set of indices for all positive sample pairs within the same batch; This represents the total number of all positive sample pairs. For the sample In its positive label The predicted score; For the sample The highest predicted score across all negative labels; The minimum distance between the positive label and the negative label with the highest predicted score; Calculate the maximum value; For sample index, ; It is a positive category index;

[0056] Step S63, apply the multimodal contrast loss function and sorting loss function The weighted combination constitutes the total loss function for model training. And optimize all learnable parameters using the gradient descent algorithm;

[0057]

[0058] In the formula, This is the loss balance factor.

[0059] The beneficial effects of this invention are:

[0060] 1. This invention introduces a multi-label zero-shot learning paradigm to construct an intelligent solution for sewer defect detection tasks. By introducing visual-semantic alignment into the field of pipeline defect recognition, the model can effectively identify defect types that have never appeared in the training data through the mapping relationship between learned image features and text semantics, breaking through the closed-set limitation of traditional detection models. While improving the model's generalization ability, it significantly reduces system maintenance costs. When it is necessary to expand the detection range, only the semantic description needs to be updated to identify new defect types, without repeating large-scale data collection and model training. This characteristic enables the detection system to have sustainable evolution capabilities, adapting to the actual needs of dynamic changes in pipeline defects. By combining feature guidance, multi-granularity evidence fusion, and score correction mechanisms, it can perform fine-grained analysis of complex defects, effectively distinguishing easily confused defect categories with similar image features, further improving detection accuracy.

[0061] 2. Unlike simply applying general multi-label zero-shot learning methods, this invention designs a professional feature extraction and fusion system tailored to the unique environmental characteristics and defect features of sewer pipes, ensuring the practicality and accuracy of the method in complex pipe environments. Attached Figure Description

[0062] Figure 1 This is a flowchart of a sewage pipe network defect detection method based on multi-label zero-shot learning in this embodiment. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example: A defect detection method for sewer networks based on multi-label zero-shot learning. This method achieves knowledge transfer from known defects to unknown defects by constructing a guide-fusion-correction network, effectively solving the problem of identifying unseen defect types. While ensuring high accuracy, it significantly reduces the cost of continuous model maintenance, providing key technical support for building an intelligent detection system with sustainable evolution capabilities.

[0065] Specifically, such as Figure 1 As shown, it includes the following steps:

[0066] Step S1: Generate a defect description for each pipeline defect category using a large language model.

[0067] Specifically, step S1 includes the following steps:

[0068] Step S11: Generate a detailed description of the defect corresponding to each pipeline defect category. and brief phrases ;

[0069] Step S12, use a text encoder to process the brief phrases. Encode the data to obtain brief phrase text features for all defect categories. .

[0070] Step S2: Use the representation guidance module to extract features and adapt the pipeline inner wall image and defect description to obtain global and local image features and detailed defect description text features.

[0071] This step aims to obtain images of the inner wall of the pipe. and detailed description of defects Deep feature extraction is performed, and a representation guidance module is used to make the feature representation more suitable for the unique scenario of sewer pipe defect detection; finally, global image features are obtained from the image. and local image features Determine the defects from the text and describe the text features in detail. .

[0072] Specifically, it includes the following steps:

[0073] Step S21, obtain the image of the inner wall of the pipe. and detailed description of defects Input the image encoder and text encoder separately to obtain the original image features and original text features respectively; the image encoder and text encoder are frozen;

[0074] Step S22: Use the representation guidance module to perform sewer pipe domain adaptation on the original image features and original text features to obtain the adapted global image features. Local image features and detailed description of defects and text features The instruction here indicates that the guidance module is learnable.

[0075] In step S22, the guiding module modifies the features output by the Transformer encoder through linear transformation, GeLU activation function, Dropout operation, and residual connection, as calculated below:

[0076]

[0077]

[0078] In the formula, Features obtained through representation guidance; The raw image features obtained after the encoder, Given the input features; To adapt to the image and text features of pipeline scenarios; and This is a linear transformation layer used to perform feature dimension mapping; It is a non-linear activation function used to enhance the non-linear expressive power of the network; This is a dropout operation used for model regularization to prevent overfitting.

[0079] Step S3: Combine the global and local image features of the image, calculate the global and local matching scores with the features of the detailed description text, and fuse them to obtain the initial defect prediction score.

[0080] Step S3 specifically includes the following steps:

[0081] Step S31: After normalization, the global image features and the detailed defect description text features are used to calculate the global matching score using the multi-granularity evidence fusion module. ;

[0082]

[0083] In the formula, For global image features; Provide a detailed description of the textual features of the defect; It is the transpose operator;

[0084] Step S32: The local matching score is calculated using a multi-granularity evidence fusion module. The local matching score is aggregated based on the importance of different image regions. Specifically, after normalization, the affinity weights of local image features and detailed description text features of defects are calculated, and then aggregated through a score aggregation network to obtain the local matching score. ;

[0085]

[0086]

[0087] In the formula, Weighted by affinity; This is a normalization function used to normalize local matching scores; For local image features; It is a multilayer perceptron with linear mapping capability; T is the transpose operator;

[0088] Step S33: Aggregate the global matching score and the local matching score to obtain the initial prediction score;

[0089]

[0090] In the formula, The initial predicted score.

[0091] Step S4: Construct a semantic relation adjacency matrix using the generated defect brief phrases to display the relationships between different defect categories.

[0092] This step utilizes the semantic similarity of the brief descriptions of each defect category to construct a matrix that explicitly models the relationships between different defect categories.

[0093] Specifically, it includes the following steps:

[0094] Step S41, use the brief phrase text features in the pipeline defect description. Calculate the cosine similarity matrix between defect categories and filter them using a preset threshold to construct an initial semantic relationship adjacency matrix. ;

[0095]

[0096]

[0097] In the formula, This is the cosine similarity matrix; For the text features of brief phrases beg Norm; T is the transpose operator; Cosine similarity matrix The Middle line, number Column similarity values; The preset similarity threshold is used to filter the similarity of defect categories, and in this embodiment it is set to 0.85; The initial semantic relation adjacency matrix The Middle line, number The value of the column; , , The total number of defect categories;

[0098] Step S42: Process the initial semantic relation adjacency matrix and calculate the final semantic relation adjacency matrix. ;

[0099]

[0100]

[0101] In the formula, It is the identity matrix; This is an adjacency matrix with self-connected semantic relationships; Let be a diagonal matrix whose diagonal elements satisfy... .

[0102] Step S5: The initial prediction scores are corrected using the semantic relationships between categories to obtain the final prediction score for each defect category.

[0103] This step inputs the initial predicted score and the semantic association adjacency matrix into the graph convolutional network, and uses the semantic association between categories to reason and correct the initial predicted score, resulting in a more accurate predicted score.

[0104] Specifically, it includes the following steps:

[0105] Step S51: Based on the initial predicted score and the semantic relation adjacency matrix, a graph convolutional network is used in the relation score correction module to predict the corrected score. ;

[0106]

[0107] In the formula, This is a semantic relation adjacency matrix; It is a non-linear activation function; The initial predicted score; and These are learnable weights;

[0108] Step S52: Aggregate the initial predicted scores and the corrected scores to obtain the final predicted score for each defect category;

[0109]

[0110] In the formula, This is the final predicted score.

[0111] Furthermore, during model training, a loss function combining contrastive learning and ranking constraints is used. Optimize the model to ensure it can accurately distinguish between known and unknown defect types.

[0112] Specifically, it includes the following steps:

[0113] Step S61: Calculate the multimodal contrast loss function. The aim is to narrow the semantic distance between an image and its actual corresponding defect category, and to widen the distance between it and other categories;

[0114]

[0115] In the formula, This is the set of indices for all positive sample pairs within the same batch. ; This represents the total number of all positive sample pairs. The sample index for positive sample pairs. ; The number of samples; The class index for positive sample pairs. ; Number of categories; Used to specify the Does the nth sample contain the nth sample? One defect category; It is an exponential function; It is the natural logarithm; Indicates the first The sample at the th Final predicted scores for each defect category; To control the temperature scaling factor of the contrast learning intensity, it is set to 0.1 in this embodiment; Indicates the first The sample at the th Final predicted scores for each defect category; To iterate through all samples, the sample index is... ; To iterate through the category indices of all categories, ;

[0116] Step S62, calculate the sorting loss function. This ensures that each sample scores higher on the positive label than on the negative label, while maintaining a minimum margin. ;

[0117]

[0118] In the formula, This is the set of indices for all positive sample pairs within the same batch; This represents the total number of all positive sample pairs. For the sample In its positive label The predicted score; For the sample The highest predicted score across all negative labels; The minimum spacing between the positive label and the negative label with the highest predicted score is set to 0.2 in this embodiment; Calculate the maximum value; For sample index, ; It is a positive category index;

[0119] Step S63, apply the multimodal contrast loss function and sorting loss function The weighted combination constitutes the total loss function for model training. And optimize all learnable parameters using the gradient descent algorithm;

[0120]

[0121] In the formula, The loss balance factor is set to 10 in this embodiment.

[0122] The defect detection method proposed in this embodiment introduces a multi-label zero-shot learning paradigm to construct an intelligent solution for sewer defect detection tasks. By introducing visual-semantic alignment into the field of pipeline defect recognition, the model can effectively identify defect types that have never appeared in the training data through the mapping relationship between learned image features and text semantics, breaking through the closed-set limitation of traditional detection models. While improving the model's generalization ability, it significantly reduces system maintenance costs. When it is necessary to expand the detection range, only the semantic description needs to be updated to identify new defect types, without repeating large-scale data collection and model training. This characteristic enables the detection system to have sustainable evolution capabilities, adapting to the actual needs of dynamic changes in pipeline defects. By combining feature guidance, multi-granularity evidence fusion, and score correction mechanisms, it can perform fine-grained analysis of complex defects, effectively distinguishing easily confused defect categories with similar image features, further improving detection accuracy. Unlike simply applying general multi-label zero-shot learning methods, the defect detection method proposed in this embodiment designs a professional feature extraction and fusion system for the unique environmental characteristics and defect features of sewer pipelines, ensuring the practicality and accuracy of the method in complex pipeline environments.

[0123] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting defects in sewer pipe networks based on multi-label zero-shot learning, characterized in that, Includes the following steps: Step S1: Generate a defect description corresponding to each pipeline defect category using a large language model; Step S2: Use the representation guidance module to extract features and adapt the pipeline inner wall image and defect description to obtain global and local image features and detailed defect description text features; Step S3: Combine the global and local image features of the image, calculate the global and local matching scores with the features of the detailed description text, and fuse them to obtain the initial defect prediction score; Step S4: Construct a semantic relation adjacency matrix using the generated defect brief phrases to display the relationships between different defect categories; Step S5: The initial prediction scores are corrected using the semantic relationship adjacency matrix between categories to obtain the final prediction score for each defect category; Step S3 specifically includes the following steps: Step S31: Calculate the global matching score using global image features and detailed defect description text features. ; In the formula, For global image features; Provide a detailed description of the textual features of the defect; It is the transpose operator; Step S32: Calculate the affinity weights between local image features and the features of the detailed description of defects in the text, and aggregate them through a score aggregation network to obtain local matching scores. ; In the formula, Weighted by affinity; This is the normalization function; For local image features; For a multilayer perceptron; T is the transpose operator; Step S33: Aggregate the global matching score and the local matching score to obtain the initial prediction score; In the formula, The initial predicted score.

2. The method for detecting defects in sewer networks based on multi-label zero-shot learning according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S11: Generate a detailed description of the defect corresponding to each pipeline defect category. and brief phrases ; Step S12, use a text encoder to process the brief phrases. Encode the data to obtain brief phrase text features for all defect categories. .

3. The method for detecting defects in sewer networks based on multi-label zero-shot learning according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S21, obtain the image of the inner wall of the pipe. and detailed description of defects in pipeline defect description Input the image encoder and text encoder respectively to obtain the original image features and the original text features respectively; Step S22: Use the representation guidance module to perform sewer pipe domain adaptation on the original image features and original text features to obtain the adapted global image features. Local image features and detailed description of defects and text features .

4. The method for detecting defects in sewer networks based on multi-label zero-shot learning according to claim 3, characterized in that, In step S22, the guiding module modifies the features output by the Transformer encoder through linear transformation, GeLU activation function, Dropout operation, and residual connection, as calculated below: In the formula, Features obtained through representation guidance; The raw image features obtained after the encoder, Given the input features; To adapt to the image and text features of pipeline scenarios; and It is a linear transformation layer; It is a non-linear activation function; This is a discard operation.

5. The method for detecting defects in sewer networks based on multi-label zero-shot learning according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S41, use the brief phrase text features in the pipeline defect description. Calculate the cosine similarity matrix between defect categories and filter them using a preset threshold to construct an initial semantic relationship adjacency matrix. ; In the formula, This is the cosine similarity matrix; For the text features of brief phrases beg Norm; T is the transpose operator; Cosine similarity matrix The Middle line, number Column similarity values; The preset similarity threshold; The initial semantic relation adjacency matrix The Middle line, number The value of the column; , , The total number of defect categories; Step S42: Process the initial semantic relation adjacency matrix and calculate the final semantic relation adjacency matrix. ; In the formula, It is the identity matrix; This is an adjacency matrix with self-connected semantic relationships. Let be a diagonal matrix whose diagonal elements satisfy... .

6. The method for detecting defects in a sewer network based on multi-label zero-shot learning according to claim 1, characterized in that, Step S5 specifically includes the following steps: Step S51: Based on the initial predicted score and semantic relation adjacency matrix, use a graph convolutional network to predict the corrected score. ; In the formula, This is a semantic relation adjacency matrix; It is a non-linear activation function; The initial predicted score; and These are learnable weights; Step S52: Aggregate the initial predicted scores and the corrected scores to obtain the final predicted score for each defect category; In the formula, This is the final predicted score.

7. The method for detecting defects in sewer networks based on multi-label zero-shot learning according to claim 1, characterized in that, During model training, a loss function combining contrastive learning and ranking constraints is used for model optimization, including the following steps: Step S61: Calculate the multimodal contrast loss function. ; In the formula, This is the set of indices for all positive sample pairs within the same batch. ; This represents the total number of all positive sample pairs. The sample index for positive sample pairs. ; The number of samples; The class index for positive sample pairs. ; Number of categories; Used to specify the Does the nth sample contain the nth sample? One defect category; It is an exponential function; It is the natural logarithm; Indicates the first The sample at the th Final predicted scores for each defect category; Temperature scaling factor to control the intensity of contrastive learning; Indicates the first The sample at the th Final predicted scores for each defect category; To iterate through all samples, the sample index, ; To iterate through the category indices of all categories, ; Step S62, calculate the sorting loss function. ; In the formula, This is the set of indices for all positive sample pairs within the same batch; This represents the total number of all positive sample pairs. For the sample In its positive label The predicted score; For the sample The highest predicted score across all negative labels; The minimum distance between the positive label and the negative label with the highest predicted score; Calculate the maximum value; For sample index, ; It is a positive category index; Step S63, apply the multimodal contrast loss function and sorting loss function The weighted combination constitutes the total loss function for model training. And optimize all learnable parameters using the gradient descent algorithm; In the formula, This is the loss balance factor.

Images