Label correlation guided multi-modal sewer symbiotic disease identification method

Abstract

The invention belongs to the technical field of image recognition, and discloses a label correlation guided multi-modal sewer symbiotic disease recognition method, which comprises the following steps: firstly, constructing a sewer symbiotic disease recognition system, and then inputting a to-be-processed image into the constructed sewer symbiotic disease recognition system to obtain a sewer symbiotic disease recognition result; the sewer symbiotic disease identification system comprises a student model and a classifier. A teacher model is introduced to guide learning of the student model; and meanwhile, discrete image-text interaction based on label guidance is introduced to guide a student model to capture complex correlation among different disease defects. According to the invention, the efficiency and accuracy of sewer disease detection can be significantly improved, new technical support is provided for efficient maintenance and management of an urban sewage system, and the cost of manual inspection and potential safety risks are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and relates to sewer disease detection technology, especially to a multi-modal sewer co-disease recognition method guided by label correlation. Background Art

[0002] With the continuous acceleration of the urbanization process and the aging of urban infrastructure, the maintenance and management of urban sewage systems have increasingly become an important issue. As the core part of the urban drainage system, the operation status of sewer pipe networks directly affects the flood control ability, environmental hygiene of the city, and the quality of life of residents. However, with the increase in the service life of pipe networks, various diseases have emerged continuously. As Figure 1 shown, these diseases include cracks, settlements, root intrusion, etc. If not repaired in time, it may lead to serious consequences such as sewage leakage, road surface collapse, and even urban waterlogging. It can also be seen from Figure 1 that in the real scene, sewer data generally shows a long-tail distribution, and diseases often occur symbiotically.

[0003] Therefore, developing an efficient and automatic sewer disease recognition method based on modern technology has become a necessary means to improve the management level of urban sewage systems. In recent years, significant progress has been made in sewer disease detection technology based on machine vision and deep learning. Through computer vision algorithms, especially the application of convolutional neural networks (CNNs), sewer pipe images can be efficiently processed and analyzed to automatically identify diseases such as cracks and settlements. The rise of these methods has significantly improved the efficiency of automatic sewer inspection, reducing the high labor intensity and high risk in traditional manual inspections.

[0004] In terms of specific implementation, existing disease recognition models mainly rely on deep learning models to learn and identify features in sewer images. For example, Kumar et al. proposed a sewer disease recognition framework based on a deep convolutional neural network, which improved the accuracy and robustness of disease recognition through automatic feature extraction (see S.S. Kumar, D.M. Abraham, M.R. Jahanshahi, T. Iseley, and J. Starr, “Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks,” Autom. Construction, vol. 91, pp. 273–283, Jul. 2018). In addition, Haurum et al. proposed the CT-GNN model, which further enhanced the semantic relevance between labels by using cross-task graphs and class node embeddings, thus improving the performance of the model in dealing with multi-label classification problems (see Haurum, Joakim Bruslund et al. “Multi-Task Classification of Sewer Pipe Defects and Properties using a Cross-Task Graph Neural Network Decoder.” 2022 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV) (2021): 1441-1452).

[0005] However, despite the many successes of deep learning techniques in the field of sewer disease detection, existing models still show obvious deficiencies when facing some specific technical challenges. These deficiencies are mainly reflected in the following two aspects:

[0006] (1) Poor recognition performance for rare but high-risk disease categories: In the actual sewer environment, the distribution of disease categories often exhibits a long-tail characteristic, that is, the occurrence frequency of a few high-risk category diseases is extremely low. For example, the number of diseases such as ruptures and remaining walls is significantly less than that of common diseases such as cracks; this long-tail distribution makes it difficult for existing models to effectively learn the features of these rare diseases from limited data, thus reducing the detection sensitivity and increasing the risk of missed detection. And the method of simply increasing the tail data often ignores the potential correlation between disease types;

[0007] (2) The complex correlation among co-existing diseases makes it difficult for the model to extract discriminative features: In the actual sewer environment, disease types usually do not exist independently. Instead, multiple diseases often coexist. For example, it is very common for cracks to be accompanied by leakage or settlement. The complex co-symbiotic relationship among these diseases requires the model to simultaneously handle the mutual influence among multiple labels. However, existing models are weak in multi-label learning and cannot fully capture this complex co-symbiotic relationship of diseases, resulting in a decrease in recognition accuracy. Especially in complex scenarios where diseases coexist, it is difficult to ensure the robustness and accuracy of the recognition results.

[0008] In summary, although existing sewer disease recognition technologies have made certain progress in terms of automation and recognition accuracy, they still face huge challenges in dealing with the difficult recognition of rare high-risk diseases and making full use of the complex co-symbiotic relationship among diseases. Summary of the Invention

[0009] The purpose of the present invention is to address the challenges of difficult recognition of low-frequency high-value diseases and make full use of the correlation of co-existing diseases, and provide a multi-modal sewer co-existing disease recognition method guided by label correlation, introducing knowledge distillation, multi-modal data fusion, and more efficient correlation modeling among labels to improve the overall performance and robustness of the model.

[0010] To achieve the above object, the present invention adopts the following technical solutions.

[0011] The present invention provides a multi-modal sewer co-existing disease recognition method guided by label correlation, which includes the following steps:

[0012] S1 Construct a sewer co-existing disease recognition system, where the sewer co-existing disease recognition system includes a student model and a classifier; introduce a teacher model to guide the learning of the student model; at the same time, introduce label-guided discrete graphic-text interaction to guide the student model to capture the complex correlation among different disease defects; Step S1 includes the following sub-steps:

[0013] S11 Input the training data into the teacher model and the student model respectively, extract the corresponding feature vectors; and determine the distillation loss L based on the extracted feature vectors KD ;

[0014] S12 Construct a text prompt sentence with labels and obtain the corresponding text embedding representation through a text editor;

[0015] S13 Construct a forward transfer probability matrix and a reverse transfer probability matrix for quantifying the interaction intensity between the image and the text embedding representation based on the feature vectors extracted by the student model and the text embedding representation; and obtain the two-way discrete graphic-text interaction cost L based on the forward transfer probability matrix and the reverse transfer probability matrix cost;

[0016] The feature vector extracted by S14 according to the student model is used to obtain the classification probability through the classifier; the asymmetric loss L is determined according to the classification probability pre ;

[0017] S15 According to the distillation loss L KD , the two-way discrete graphic-text interaction cost L cost , the asymmetric loss L pre Construct the overall loss L;

[0018] S16 Optimize the parameters of the student model and the classifier according to the overall loss;

[0019] Repeat steps S11 - S16 until the student model converges, and use the trained student model and classifier as the sewer co-disease identification system;

[0020] S2 Input the image to be processed into the constructed sewer co-disease identification system. First, extract the feature vector through the student model, and then obtain the classification probability through the classifier to get the sewer co-disease identification result.

[0021] In the above step S1, the knowledge distillation technology is adopted, and the pre-trained CLIP model is used as the teacher model to guide the learning of the student model, so as to enhance the learning ability of the student model for rare disease features and alleviate the problem of insufficient attention of the model to low-frequency high-value disease categories; in addition, the present invention also designs a label-guided discrete graphic-text interaction, which can deeply fuse image and text information, help the model better understand and capture the complex correlations between different diseases, and more effectively utilize the symbiotic relationship between diseases, so as to realize the inspection of various diseases.

[0022] In an implementable manner, the teacher model used is the pre-trained CLIP, ResNet, DenseNet or YOLO, etc. In order to enable the model to effectively identify rare but key sewer defect features, the present invention also uses the tail data to fine-tune the pre-trained CLIP model. This method solves the long-tail distribution problem in the sewer defect dataset and guides the attention of the model to uncommon but important labels.

[0023] According to the constructed tail data sample pair (where x i represents the i-th defect image, represents the m-th label of the i-th defect image, and M represents the number of labels), the fine-tuning process is implemented using the cross-modal contrast loss, and the cross-modal contrast loss is represented as L contrastive is:

[0024]

[0025] Among them, represents the i-th sample x i The image embedding (i.e., feature vector) obtained through the CLIP model, represents the number of samples used for fine-tuning the CLIP model; represents the sample x i associated label y i The positive sample label embedding text obtained through the CLIP model, t j represents the sample x i non-associated label y j The negative sample label embedding text obtained through the CLIP model; sim(·,·) represents the cosine similarity function; M represents the total number of all labels; the log function has e as the base.

[0026] The objective function of the fine-tuning process is as follows:

[0027]

[0028] Among them, θ represents the CLIP model parameters.

[0029] In one implementable manner, the student model uses a vision encoder based on Transformer or / and CNN.

[0030] In the above step S11, for the given sewer disease image x i , according to the specified patch side length P, it is divided into N blocks, and feature extraction is performed through the student model, and the extracted feature vector E i = {e i′}, i′ = 1,…,n.

[0031] In the knowledge distillation process, learning is carried out by minimizing the difference between the feature vectors extracted by the student model and the feature vectors extracted by the teacher model. Usually, the L2 norm is used to measure this difference, that is, the distillation loss L KD is defined as:

[0032]

[0033] Among them, x i represents the i-th training sample, E i and respectively represent the feature vectors extracted by the student model and the teacher model from x i extracted.

[0034] In the above step S12, for each label, a text prompt containing the label is generated, and the corresponding text embedding representation is obtained through the text encoder j = 1,…,M, M represents the number of labels.

[0035] In the above step S13, considering the interference parts in the sewer image, for the input image x i introduce the label-guided sparse probability weights α i and β i for the image features and text embeddings in the embedding space, which are expressed as follows:

[0036]

[0037] β i = softmax(y i );

[0038] where, denotes the transpose of E i , y i represents the true label of the i-th image, s i represents label perception, and TopK(·, k) is a sparsity operation, indicating selecting the top k with higher weights to effectively suppress the influence of noise features.

[0039] Then, based on the label-guided sparse probability weights, establish the forward transmission probability matrix and the backward transmission probability matrix that quantify the interaction strength between the quantized image and the label embedding representation; the element in the forward transmission probability matrix and the element in the backward transmission probability matrix are defined as follows:

[0040]

[0041] where, e i′ represents the i'-th feature point in the image feature vector E i , α i,i′ represents the sparse probability weight of the i'-th feature point of the input image x i , t j represents the j-th text embedding point in the text embedding , β j represents the sparse probability weight of the j-th text embedding point, and φ(·, ·) represents the similarity between the image feature and the text embedding, which is used to measure the interaction strength between different modalities.

[0042] Then, define the bidirectional discrete image-text interaction cost as

[0043]

[0044] where, c i′j represents the interaction cost from image to text, and c ji′ represents the interaction cost from text to image, specifically:

[0045]

[0046] Through this two-way adjustment, the interaction between the graphics and text is continuously strengthened, forming a closed-loop optimization mechanism. By minimizing the interaction cost, not only the recognition ability of the model for a single disease is improved, but also in the complex environment of co-existing multiple diseases, the mutual relationship between diseases can be captured more accurately, significantly enhancing the recognition performance of the model.

[0047] In the above step S14, the classification probability obtained by the classifier is expressed as:

[0048]

[0049] where E i represents the feature vector extracted by the student model, σ(·) represents the sigmoid function, and MLP(·) represents the multi-layer perceptron.

[0050] To more effectively solve the problem of label imbalance, the present invention adopts the asymmetric loss (ASL). The asymmetric loss is a variant of the focal loss and is also a valuable choice in multi-label classification tasks. Here, the asymmetric loss L pre is specifically expressed as:

[0051]

[0052] where p mi represents the class probability of the i-th image x i in the m-th class obtained by the classifier, y mi = 1 indicates that the image x i has the m-th label, y mi = 0 indicates that the image x i does not have the m-th label, γ + and γ - represent the hyperparameters of the positive and negative labels, and the log function has e as the base.

[0053] In the above step S15, the overall loss L is expressed as:

[0054]

[0055] where λ p and λ c and λ d respectively represent the hyperparameters of L pre and L cost and L KD .

[0056] In the above step S16, based on minimizing the overall loss, the learnable parameters are optimized through backpropagation. In a specific implementation, according to the overall loss, the student model and classifier parameters are optimized through a gradient descent optimization algorithm.

[0057] Repeat the above steps S11 - S16. When the set upper limit of the number of iterations is reached, it can be considered that the student model converges. After the training is completed, the trained student model and classifier are used as the sewer co - morbidity identification system.

[0058] Compared with the prior art, the multi - modal sewer co - morbidity identification method guided by label correlation provided by the present invention has the following beneficial effects:

[0059] (1) In the process of constructing the sewer co - morbidity identification system, by introducing the knowledge distillation technology, the student model can effectively learn the high - level feature representations in the teacher model, thereby enhancing its attention to rare disease categories, reducing missed detection situations, improving the overall recognition accuracy, and also reducing the economic losses caused by low - frequency high - value diseases; through the label - correlation - guided multi - modal learning, the present invention has significant advantages in dealing with the long - tail distribution of disease data and the problem of label sparsity, thereby effectively reducing the risk of overfitting and enhancing the adaptability and generalization ability of the model in complex scenarios;

[0060] (2) The present invention uses the tail data in the long - tail distribution of the sewer defect dataset to fine - tune the teacher model to improve its ability to identify key defect features existing in the tail data;

[0061] (3) By introducing the non - symmetric loss, the present invention can more effectively solve the problem of label imbalance;

[0062] (4) The present invention can significantly improve the efficiency and accuracy of sewer disease detection, provide new technical support for the efficient maintenance and management of urban sewage systems, and reduce the cost of manual inspection and potential safety risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the sewer disease distribution;

[0064] Figure 2 It is a process view of the multi - modal sewer co - morbidity identification method guided by label correlation;

[0065] Figure 3 It is a schematic diagram of the construction principle of the sewer co - morbidity identification system;

[0066] Figure 4 It is a schematic diagram of the discrete graphic - text interaction process;

[0067] Figure 5 The test results of using the multi-modal sewer co-disease identification method to identify different sewer images; (a) and (e) are the original sewer images with defect labels of "hidden connection of branch pipes, corrosion, and misalignment" and "disconnection, corrosion, and scaling" respectively; (b)-(d) and (f)-(h) are the corresponding heatmaps predicted by the method of the present invention; these heatmaps intuitively show the attention areas of the model in the image through color gradients; the colors from blue to red represent the attention degree of the model to the defect areas, increasing from low to high in sequence. Specific implementation mode

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope protected by the present invention.

[0069] Embodiment 1

[0070] This embodiment provides a multi-modal sewer co-disease identification method guided by label correlation, as Figure 2 shown, which includes the following steps:

[0071] S1 Construct a sewer co-disease identification system;

[0072] S2 Input the image to be processed into the constructed sewer co-disease identification system. First, extract the feature vector through the student model, and then obtain the classification probability through the classifier to get the sewer co-disease identification result.

[0073] In this embodiment, by combining the information of two modalities of image and text, the disease characteristics are deeply mined, especially enhancing the understanding of diseases in the scenario of complex co-diseases. In addition, by introducing the knowledge distillation technology, the recognition ability of the model for tail diseases is strengthened, thereby effectively improving the performance of the model in unbalanced data and reducing the potential economic losses caused by ignoring rare diseases.

[0074] The construction principle of the sewer co-disease identification system provided in this embodiment is as Figure 3As shown, the sewer co-disease identification system includes a student model and a classifier; the student model (Vision Encoder Student) is mainly used to extract features from images, and can be a vision encoder based on Transformer (see Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. Advances in neural information processing systems, 2017, 30); the classifier (Classification Header) uses a multi-layer perceptron (MLP). The sewer co-disease identification system is trained mainly from three aspects: (1) By introducing a teacher model to guide the learning of the student model; (2) Designing label-guided discrete graphic-text interaction, which can deeply integrate image and text information to help the model better understand and capture the complex correlations between different diseases; (3) For the classifier, introducing an asymmetric loss to effectively solve the problem of label imbalance.

[0075] In the specific implementation manner, the above step S1 includes the following sub-steps:

[0076] S11 respectively input the training data into the teacher model and the student model, extract the corresponding feature vectors; and determine the distillation loss L based on the extracted feature vectors KD 。

[0077] In this embodiment, the teacher model (Vision Encoder Teacher) used is a pre-trained CLIP model, see Radford, Alec et al. “Learning Transferable Visual Models From Natural Language Supervision.” International Conference on Machine Learning (2021).

[0078] In order to enable the model to effectively identify rare but critical sewer defect features, this embodiment also fine-tunes the pre-trained CLIP model using tail data. This method solves the long-tail distribution problem in the sewer defect dataset and guides the model's attention to uncommon but important labels.

[0079] In this embodiment, a number of sample data located at the tail in the sewer data distribution (such as sediment defects, disconnection defects, residual wall defects, etc.) are used to construct positive and negative sample pairs, that is, tail data. For any sample within a training batch, its own corresponding label is designated as the positive sample label, and the labels of the remaining samples are used as negative sample labels; the resulting important end - defect image and label text pair is expressed as where x i represents the i - th defect image, represents the m - th label of the i - th defect image, and M represents the number of labels. This method is not only easy to implement but also beneficial for the model to quickly learn long - tail labels. Although this method may occasionally introduce false negatives, the overall data volume is limited, and the subsequent closed - loop graphic - text interaction module effectively reduces the noise, thereby improving the recognition accuracy.

[0080] Specifically, the fine - tuning process of the CLIP model can be expressed as:

[0081]

[0082] where CLIP fine-tuned represents the fine - tuned CLIP model, CLIP pretrained represents the CLIP model before fine - tuning, and D tail represents the tail data for training.

[0083] The fine - tuning process is implemented using a cross - modal contrast loss, and the cross - modal contrast loss is denoted as L contrastive as:

[0084]

[0085] where represents the image embedding (i.e., feature vector) of the i - th sample x i obtained through the CLIP model, represents the number of samples used for fine - tuning the CLIP model; represents the sample x i associated with the label y i the positive sample label embedding text obtained through the CLIP model, and t j represents the sample x i not associated with the label y j the negative sample label embedding text obtained through the CLIP model; sim(·,·) represents the cosine similarity function; M represents the total number of all labels; the log function has base e.

[0086] The cosine similarity function is expressed as follows:

[0087]

[0088] Among them, a and b respectively represent two vectors to be compared.

[0089] The objective function of the fine-tuning process is as follows:

[0090]

[0091] Among them, θ represents the CLIP model parameters.

[0092] In this embodiment, according to the calculated cross-modal contrast loss, the CLIP model parameters are optimized by the gradient descent optimization algorithm; after several iterations of the above training process, the cross-modal contrast loss is basically unchanged, and the fine-tuning of the CLIP model is completed.

[0093] Through this simple fine-tuning method, the identification ability of the model for long-tail defects can be significantly improved, the negative impact of the long-tail distribution on the model performance can be effectively reduced, and the simplicity and operability of the method are maintained at the same time.

[0094] Then, the training data samples are respectively input into the fine-tuned teacher model CLIP and student model to extract the corresponding feature vectors. In this embodiment, several types (including scum, leakage, rupture, residue, corrosion, deformation, misalignment, etc.) of sewer disease defect image samples collected are used as training data.

[0095] For the i-th sewer disease, it is denoted as x i , and the feature vector extracted by the teacher model from x i is denoted as:

[0096]

[0097] Among them, f CLIP (·) represents the feature extraction function of the teacher model; represents the feature vector extracted by the teacher model.

[0098] For the i-th sewer disease, it is denoted as x i , where H×W represents the size of the image, C represents the number of channels, and it is reshaped into a series of flat 2D patches where P represents the side length of each patch, and the total number of patches can be expressed as n = HW / P 2 , and feature extraction is performed through the student model, and the extracted feature vector is denoted as:

[0099] E i = f backbone (x i )(6);

[0100] Among them, f backbone(·) represents the feature extraction function of the student model; represents the feature vector extracted by the student model, E i ={e i′}, i′ = 1, …, n, where These vectors not only capture the local features of the image but also provide a basis for subsequent image-text fusion.

[0101] During the knowledge distillation process, learning is carried out by minimizing the difference between the feature vectors extracted by the student model and the feature vectors extracted by the teacher model. Usually, the L2 norm is used to measure this difference, that is, the distillation loss L is defined KD as:

[0102]

[0103] Here, L KD (x i ) measures the mean squared error between the feature vector extracted by the student model and the feature vector extracted by the teacher model. By minimizing this loss, the student model can be guided to learn a feature representation closer to that of the teacher model, greatly enhancing the recognition accuracy for those low-frequency and high-value diseases. This strategy effectively alleviates the long-tail problem and improves the generalization ability and accuracy of the model in practical applications.

[0104] S12 constructs text prompt sentences with labels and obtains the corresponding text embedding representations through a text editor.

[0105] During the prompt selection process, this embodiment considers the balance between the simplicity and expressiveness of the text. Since the defect categories of the sewer system usually have relatively consistent semantic features, this embodiment selects a unified template to describe these defects. A text prompt such as "This is a sewer disease system containing {label}" is also generated for each sewer disease image. Thus, M label text prompt sentences are obtained. Further, the text embedding representations of the labels are obtained through a text encoder j = 1, …, M, where M represents the number of labels.

[0106] S13 constructs a forward transfer probability matrix and a backward transfer probability matrix for quantifying the interaction strength between the image and the text embedding representation based on the feature vector extracted by the student model and the text embedding representation; and obtains the bidirectional discrete image-text interaction cost L cost .

[0107] In this embodiment, the image-text interaction process is as Figure 4As shown, the image embedding and text embedding achieve closed-loop alignment through bidirectional optimization to minimize the cost of cross-modal interaction. This mechanism effectively reduces noise and improves the recognition accuracy of the model in complex disease environments.

[0108] Considering the interference parts in the sewer images, for the input image x i introduce label-guided sparse probability weights α i and β i to the image features in the embedding space and text embeddings. This weight is calculated through the interaction between the label embedding and the image features, aiming to strengthen the recognition of key diseases. This step reduces the interference of noise and irrelevant information by selecting the image patches that are most beneficial to disease recognition.

[0109] α i and β i are expressed as follows:

[0110]

[0111] β i = softmax(y i )(9);

[0112] where, represents the transpose of E i , y i represents the true label of the i-th image, s i represents label perception, and TopK(·, k) is a sparsity operation, indicating to select the top k with higher weights, thus effectively suppressing the influence of noise features.

[0113] Then, according to the label-guided sparse probability weights, establish the forward transmission probability matrix and the reverse transmission probability matrix that quantify the interaction strength between the quantized image and the label embedding representation; the element in the forward transmission probability matrix and the element in the reverse transmission probability matrix are defined as follows:

[0114]

[0115] where, e i′ represents the i'-th feature point in the image feature vector E i , α i,i′ represents the sparse probability weight of the i'-th feature point of the input image x i , t j represents the j-th text embedding point in the text embedding , β jdenotes the sparse probability weight of the j-th text embedding point, and φ(·,·) represents the similarity between the image features and the text embedding, which is used to measure the interaction intensity between different modalities.

[0116] Then, define the bidirectional discrete image-text interaction cost as

[0117]

[0118] where c i′j denotes the interaction cost from image to text, and c ji′ denotes the interaction cost from text to image, specifically:

[0119]

[0120] Through this bidirectional adjustment, the interaction between images and texts is continuously strengthened, forming a closed-loop optimization mechanism. By minimizing the interaction cost, not only the model's recognition ability for a single disease is improved, but also in the complex environment of multiple co-existing diseases, the mutual relationship between diseases can be captured more accurately, significantly enhancing the model's recognition performance.

[0121] S14 obtains the classification probability through the classifier based on the feature vector extracted by the student model; determines the asymmetric loss L pre .

[0122] Obtaining the classification probability through the classifier is expressed as:

[0123]

[0124] where E i denotes the feature vector extracted by the student model, σ(·) represents the sigmoid function, and MLP(·) represents the multi-layer perceptron.

[0125] To more effectively solve the problem of label imbalance, the present invention adopts the asymmetric loss (ASL). The asymmetric loss is a variant of the focal loss and is also a valuable choice in multi-label classification tasks. Here, the asymmetric loss L pre is specifically expressed as:

[0126]

[0127] where p mi denotes the class probability of the i-th image in the m-th class obtained through the classifier, y mi =1 indicates that the image x i has the m-th label, y mi =0 indicates that the image x i does not have the m-th label, γ + and γ -Hyperparameters representing positive and negative labels, with the log function having base e.

[0128] S15 is based on the distillation loss L KD and the bidirectional discrete graphic-text interaction cost L cost and the asymmetric loss L pre to construct the overall loss L.

[0129] Here, the overall loss L is expressed as:

[0130]

[0131] where λ p and λ c and λ d respectively represent the hyperparameters of L pre and L cost and L KD respectively.

[0132] S16 optimizes the student model parameters and classifier parameters based on the overall loss.

[0133] Based on minimizing the overall loss, the learnable parameters are optimized through backpropagation. In the specific implementation, according to the overall loss, the student model and classifier parameters are optimized through the gradient descent optimization algorithm.

[0134] Repeat the above steps S11 - S16. When the set upper limit of the number of iterations is reached, the student model can be considered to have converged. Use the trained student model and classifier as the sewer co-disease identification system.

[0135] In the above step S2, the collected sewer images to be processed are input into the sewer co-disease identification system constructed in step S1. First, it is divided into several patches, feature vectors are extracted through the student model, and then classification probabilities are obtained through the classifier to get the sewer co-disease identification result.

[0136] To further verify the performance of the proposed model in the identification of co-occurring diseases, the present invention conducted a correlation analysis on the prediction results and true labels of the model. Through relevant experiments on the Sewer-ML public dataset (see Haurum, Joakim Bruslund and Thomas Baltzer Moeslund. “Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark.” 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 13456-13467), images containing typical co-occurring diseases were selected for testing. Figure 1 The specific prediction results of the model on these images are shown. Figure 5 It can be seen that the multi-modal sewer co-occurring disease identification method provided by the present invention can reasonably identify the spatial positions and ranges of multiple defects in the complex scenarios of co-occurring diseases. Figure 5 For example, the red area in (b) indicates the high attention of the model to the defect of concealed connection of branch pipes, while Figure 5 (h) shows the good capture effect of the present invention on the scaling disease. This further verifies the robustness and reliability of the present invention in dealing with the task of co-occurring disease identification.

[0137] Table 1: Comparison results of different models under the sewer-ML dataset

[0138]

[0139] In addition, on the SewerML disease test set, as shown in Table 1, it was observed that the method of the present invention is superior to all other methods in terms of F2 CIW and F1 Defects . Compared with DS-MSHViT using the ResNet-101 backbone, the F2 CIW score increased by 3.02%. Therefore, the knowledge distillation combined with the graphic and text dual-modal discrete transmission proposed by the present invention can effectively improve the performance of multi-label sewer disease identification. The multi-modal discrete transmission architecture allows the model to learn more complex relationships between diseases, and distillation learning reduces the impact brought by the long-tailed distribution of data, thereby improving the performance of the model on the Sewer-ML dataset.

[0140] Those of ordinary skill in the art will realize that the embodiments herein are provided to assist the reader in understanding the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on these technical revelations disclosed in the present invention, and these deformations and combinations are still within the scope of protection of the present invention.

Claims

1. A label correlation-guided multimodal sewer symbiotic disease identification method, characterized in that: The following steps are involved: S1 constructs a sewer symbiotic disease recognition system, which includes a student model and a classifier; introduces a teacher model to guide the learning of the student model; and introduces label-guided discrete image-text interaction to guide the student model to capture the complex correlation between different disease defects; Step S1 includes the following sub-steps: S11 inputs the training data into the teacher model and the student model respectively, extracts the corresponding feature vectors; and determines the distillation loss L based on the extracted feature vectors KD ; S12 constructs a text prompt sentence containing a label and obtains the corresponding text embedding representation through a text editor; S13 constructs the forward transmission probability matrix and the reverse transmission probability matrix for quantifying the interaction strength between the image and text embedding representation based on the feature vector extracted by the student model and the text embedding representation; and obtains the bidirectional discrete image-text interaction cost L based on the forward transmission probability matrix and the reverse transmission probability matrix cost ; S14 obtains the classification probability through the classifier based on the feature vector extracted by the student model; determines the asymmetric loss L based on the classification probability pre ; S15 based on distillation loss L KD , bidirectional discrete image-text interaction cost L cost , asymmetric loss L pre Construct the overall loss L; S16 optimizes the student model parameters and classifier parameters according to the overall loss; Repeat steps S11-S16 until the student model converges, and use the trained student model and classifier as a sewer symbiotic disease identification system; S2 inputs the image to be processed into the constructed sewer symbiotic disease recognition system, first extracts the feature vector through the student model, then obtains the classification probability through the classifier, and obtains the sewer symbiotic disease recognition result.

2. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 1 is characterized in that: In step S1, the teacher model used is pre-trained CLIP, ResNet, DenseNet or YOLO.

3. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 2 is characterized in that: Several sample data at the tail of the sewer data distribution are used to construct positive and negative sample pairs and tail data; then the pre-trained CLIP model is fine-tuned using the tail data.

4. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 3 is characterized in that: Based on the constructed tail data sample pair Among them, x i represents the i-th defect image, represents the mth label of the i-th defect image, M represents the number of labels, and the fine-tuning process is implemented using cross-modal contrast loss, which represents L contrastive for: in, represents the i-th sample x i The image embedding obtained by the CLIP model, that is, the feature vector, represents the number of samples used for CLIP model fine-tuning; t i + Represents sample x i Associated label y i The positive sample label obtained by the CLIP model is embedded in the text, t j Represents sample x i Non-associated label y j The negative sample labels obtained by the CLIP model are embedded in the text; sim(·,·) represents the cosine similarity function; M represents the total number of all labels; The objective function of the fine-tuning process is as follows: Here, θ represents the CLIP model parameters.

5. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 1, characterized in that: The student model uses a visual encoder based on Transformer and / or CNN.

6. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 1, characterized in that: In step S11, during the knowledge distillation process, the distillation loss L is defined KD for: Among them, x i represents the i-th training sample, E i and Respectively represent the student model and the teacher model from x i The extracted feature vector.

7. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 1, characterized in that: In step S12, for each label, a text prompt containing the label is generated, and the corresponding text embedding representation is obtained through the text encoder. M represents the number of labels.

8. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 7, characterized in that: In step S13, for a given sewer disease image x i , according to the specified patch side length P, it is divided into N blocks, and features are extracted through the student model. The extracted feature vector E i ={e i′ }, i′=1,…,n; For the input image x i The image features and text embeddings in the embedding space introduce label-guided sparse probability weights α i and β i , which is expressed as follows: β i =softmax(and i ); in, Indicates E i The transpose of y i represents the true label of the i-th image, s i Indicates label awareness; Then, based on the label-guided sparse probability weights, the forward transmission probability matrix and the backward transmission probability matrix are established to quantify the interaction strength between the image and the label embedding representation; the elements in the forward transmission probability matrix are and the elements in the reverse transmission probability matrix The definition is as follows: Among them, e i′ Represents the image feature vector E i The i′th feature point in t j Representing text embedding The jth text embedding point in , φ(·,·) represents the similarity between the image feature and the text embedding; Then, define the bidirectional discrete image-text interaction cost as Among them, ci′j represents the interaction cost from image to text, c ji′ Represents the interaction cost from text to image, specifically:

9. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 1, characterized in that: In step S14, the classification probability obtained by the classifier is expressed as: p i =σ(MLP(E i ))); Among them, E i represents the feature vector extracted by the student model, σ(·) represents the sigmoid function, and MLP(·) represents the multilayer perceptron; Not for this loss L pre Specifically expressed as: Among them, p mi represents the category probability of the i-th image in the m-th category obtained by the classifier, y mi =1 means image x i There is the mth label, y mi =0 means image x i There is no mth label, γ + , γ - A hyperparameter representing the positive and negative labels.

10. The label correlation-guided multimodal sewer symbiotic disease identification method according to claim 1, characterized in that: In step S15, the overall loss L is expressed as: Among them, λ p , c , d Respectively represent L pre , L cost , L KD Hyperparameters of .

Images