A method, device and storage device for identifying abnormal samples in underground pipeline images
By constructing the DeformMobileViT model, replacing the convolutional layers with variable convolutions and combining them with transfer learning, the problems of gradient vanishing and high resource requirements in the identification of abnormal samples of underground pipelines were solved, achieving higher accuracy and faster identification results.
Patent Information
- Application Number
- CN202310522204.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2043-05-05
AI Technical Summary
Existing convolutional neural network models suffer from gradient vanishing and gradient exploding problems in the identification of abnormal samples in underground pipelines. They also have high resource requirements and are not suitable for deployment on mobile devices. Furthermore, traditional models with fixed convolutional kernel sizes are difficult to extract high-level image features.
We construct the DeformMobileViT model by replacing the first and last convolutional layers of the MobileViT model with variable convolutional layers and combining transfer learning. We then train and validate the model using a public dataset of underground pipes to optimize it.
It improves the accuracy and speed of identifying abnormal samples in underground pipelines, reduces computational complexity and the number of parameters, and is suitable for deployment on mobile devices.
Smart Images
Figure CN116681650B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition, and in particular to a method, device, and storage device for identifying abnormal samples in underground pipeline images. Background Technology
[0002] Underground pipelines are an important urban infrastructure, and regular inspection and maintenance of underground pipelines is an important task for municipal management departments.
[0003] Currently, many scholars have conducted extensive research and achieved significant results in the field of deep learning-based defect detection for underground pipelines. For example, in 2019, Hassan et al. proposed a model based on transform learning of AlexNet, which can identify six main crack types. Xie et al. used a two-level hierarchical deep CNN model and achieved classification accuracy exceeding 94% on multiple benchmark datasets.
[0004] In 2021, Haurum JB, Moeslund TB, and others from Aalborg University (AAU) evaluated 12 models from the fields of multi-label classification and automatic underground pipeline detection based on the Sewer-ML dataset (CVPR 2021) in their research on underground pipeline defects. They used Xie's two-level model method to experiment on other models and obtained their own benchmark algorithm, achieving an F2CIW score of 55.11% and an F1normal score of 91.8%.
[0005] In current research, only Xie's study includes the identification of abnormal samples from underground pipelines, and it is used as the first-stage network. The parameters learned in the first stage are transferred to the multi-classification model by freezing the parameters of the convolutional layers and fine-tuning the fully connected layers.
[0006] In the research on abnormal defect detection of underground pipelines, traditional convolutional neural network (CNN) models are currently the primary method for extracting image features. However, the kernel size of CNNs is fixed, which means they can only extract image features locally. Multiple layers are needed to obtain higher-level feature information, which can easily lead to vanishing and exploding gradients. While networks optimized using residual connections can improve this problem to some extent, deeper networks require more resources, making them unsuitable for deployment on mobile devices. Summary of the Invention
[0007] To further improve the accuracy and speed of anomaly sample identification for underground pipelines, this invention provides anomaly sample identification for underground pipeline images based on DeformMobileViT. Specifically, it constructs a deep learning neural network model, DeformMobileViT, to distinguish between normal samples (samples without any pipeline defects) and abnormal samples (samples containing any one or more pipeline defects) of underground pipeline images acquired from the acquisition device, automatically identifying the abnormal images containing defects. This invention provides a method, device, and storage device for anomaly sample identification of underground pipeline images, wherein the method specifically includes the following steps:
[0008] S1. Construct the MobileViT model; the MobileViT model mainly includes the MV2 module and the MobileViT module;
[0009] S2. Replace the first and last convolutional layers in the MobileViT model with variable convolutional layers to obtain the DeformMobileViT model.
[0010] S3. Transfer the pre-trained parameters to the underground pipeline anomaly identification task to obtain the optimized DeformMobileViT model;
[0011] S4. Use the publicly available dataset of underground pipelines to test and optimize the DeformMobileViT model.
[0012] A storage device that stores instructions and data for implementing a method for identifying abnormal samples in underground pipeline images.
[0013] An anomaly sample discrimination device for underground pipeline images includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement an anomaly sample discrimination method for underground pipeline images.
[0014] The beneficial effects provided by this invention are: by combining MobileViT with deformable convolution technology, better performance can be achieved in the task of underground pipeline anomaly detection; the model trained by transfer learning can fit faster and achieve better results. Attached Figure Description
[0015] Figure 1 This is a flowchart of the method of the present invention;
[0016] Figure 2 This is the MobileViT model architecture diagram;
[0017] Figure 3 This is a schematic diagram of the MV2 module structure;
[0018] Figure 4This is a schematic diagram of the DeformMobileViT model;
[0019] Figure 5 This is a diagram of the hardware device used in this invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0021] Please refer to Figure 1 , Figure 1 This is a flowchart of the method of the present invention.
[0022] This invention provides a method, device, and storage device for identifying abnormal samples in underground pipeline images. The method specifically includes the following steps:
[0023] S1. Construct the MobileViT model; the MobileViT model mainly includes the MV2 module and the MobileViT module;
[0024] Please refer to Figure 2 , Figure 2 This is the architecture diagram of the MobileViT model.
[0025] It should be noted that the model architecture is built on the PyTorch deep learning framework and mainly includes two key modules: the MV2 module and the MobileViT module.
[0026] refer to Figure 2 On the left, the structure of the MobileViT model specifically includes the following components connected in sequence: a convolutional layer (conv 3×3), an MV2 module, an MV2 module, an MV2 module, an MV2 module, an MV2 module, a MobileViT module (Transformer layer, L=2), an MV2 module, a MobileViT module, an MV2 module, a MobileViT module, a convolutional layer (conv 1×1), a global pooling layer, and a fully connected layer (Linear).
[0027] The MobileViT module includes: a downsampling convolutional layer, a Transformer layer, and a feature fusion unit.
[0028] The MobileViT module processes data as follows:
[0029] S11. Downsample the input data using a downsampling layer;
[0030] S12. Divide the downsampled data into blocks and then input them into the Transformer layer for feature extraction to obtain global features;
[0031] S13. The feature fusion unit fuses global features and local features to obtain fused features.
[0032] It's important to note that the Transformer layer is used to extract global features from the image. By computing through multiple Transformer layers, long-range dependencies in the image can be captured. The feature fusion module combines local and global features to obtain a richer feature representation. In this way, MobileViT can significantly reduce computation and improve model speed while maintaining the quality of feature representation.
[0033] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the MV2 module structure.
[0034] The MV2 module includes an extension layer, a depthwise separable convolutional layer, and a linear projection layer. The core of the MV2 module borrows the idea of inverted residuals, aiming to improve the nonlinear expressive power of neural networks and achieve better performance by reducing computational cost and the number of parameters.
[0035] S2. Replace the first and last convolutional layers in the MobileViT model with variable convolutional layers to obtain the DeformMobileViT model.
[0036] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the DeformMobileViT model; Figure 4 In this paper, the first and last convolutional layers in the MobileViT model are replaced with variable convolutional layers, while the intermediate stages are retained as standard convolutional layers. This reduces the impact of the increased number of network parameters due to the introduction of variable convolutional layers on the network load. This design can further improve network performance while maintaining a lower computational cost.
[0037] It should be noted that deformable convolution means that the convolution position can be deformed. It can flexibly adjust the size and position of the convolution kernel according to different image shapes, so as to better adapt to various image processing tasks.
[0038] The variable convolutional layer adjusts the kernel size (n×n) according to different input images, and its output is as follows:
[0039]
[0040] Where R represents the size of the convolution kernel, n represents the side length of the convolution kernel, and P represents the side length of the convolution kernel.n Let P0 be any point in the convolution kernel, and let ΔP be the center point of the convolution kernel. n For a variable offset, w(P) n X(P0+P) represents the weight of any point in the convolution kernel; n +ΔP n ) indicates that the convolution kernel is centered at point P0, and the convolution kernel extends from point P to point P. n +ΔP n Output the offset points.
[0041] It should be noted that, taking a 3×3 convolution as an example, for each output y(P0), 9 positions are sampled from X. These 9 positions are all spread outwards from the center position X(P0), where (-1,-1) represents the upper left corner of X(P0) and (1,1) represents the lower right corner of X(P0). Therefore, the traditional convolution output is (where y is the n points in the grid).
[0042] R={(-1,-1),(-1,0),...,(0,1),(1,1)}
[0043]
[0044] Deformable convolution adds an offset to the traditional convolution operation. It is this offset that transforms the convolution into an irregular convolution. Note that this offset can be a decimal, so the eigenvalues in the following expression need to be calculated using bilinear interpolation:
[0045]
[0046] S3. Transfer the pre-trained parameters to the underground pipeline anomaly identification task to obtain the optimized DeformMobileViT model;
[0047] It should be noted that after constructing the DeformMobileViT model, the pre-trained parameters were transferred to the underground pipeline anomaly detection task. Because the tasks are different and the final outputs are different, all parameters except for the classification layer and the mismatched network layer were transferred.
[0048] First, use the load function in PyTorch to save the trained model and parameters of each layer to pretrained_dict;
[0049] Then, in the model constructed by torch.nn.Module, the state_dict function obtains the mapping relationship between each layer of the model and its corresponding parameters and saves it to model_dict. Because the final output is different, after removing the last fully connected layer, the corresponding values in pretrained_dict and model_dict are extracted, including the parameters of the convolutional layer and the fully connected layer, as well as the bias values, and saved to pretrained_dict.
[0050] Finally, update the model's weights. First, use update to update the pretrained key / value pairs dictionary, overwriting the original keys in model_dict. Then, load_state_dict loads model_dict into the model.
[0051] S4. Use a publicly available dataset of underground pipelines to test and optimize the DeformMobileViT model;
[0052] As one embodiment, in this invention, based on the internationally publicly available Sewer-ML dataset, 13004 samples are selected for sample anomaly detection, of which 7188 are normal samples and 5816 are defective samples. The model is trained and validated according to a ratio of approximately 7:1.5:1.5 between the training set, validation set, and test set.
[0053] The experimental results are shown in Table 1. Here, FLOPs is a metric for the computational complexity and cost of the model, param is the number of model parameters, ACC is the model's prediction accuracy, F1 score is the harmonic mean of the model's precision and recall, and AP is the area under the PR curve. Under the same hyperparameter settings, this method achieves higher accuracy than most CNN-based models. As shown in the table above, MobileViT is abbreviated as MViT, and DeformMobileViT is abbreviated as DMViT. In the Sewer-ML dataset, this method improves precision and F1 score by 2.7% and 3.57% respectively in the underground pipe image anomaly detection task compared to the original model. Furthermore, compared to traditional convolutional network models such as ResNet and DenseNet, this method has fewer parameters but achieves comparable accuracy.
[0054] Table 1 Experimental Results
[0055]
[0056] Please see Figure 5 , Figure 5This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: an underground pipeline image anomaly sample discrimination device 401, a processor 402, and a storage device 403.
[0057] An underground pipeline image anomaly sample discrimination device 401: The underground pipeline image anomaly sample discrimination device 401 implements the underground pipeline image anomaly sample discrimination method.
[0058] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the method for identifying abnormal samples in underground pipeline images.
[0059] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the method for identifying abnormal samples in underground pipeline images.
[0060] The beneficial effects of this invention are:
[0061] 1. The MobileViT model was applied to the anomaly detection of underground pipeline images. MobileViT is a lightweight and low-latency model that combines the advantages of CNN and ViT, and outperforms traditional convolutional network models in the anomaly detection task of underground pipeline images.
[0062] 2. Deformable convolution is introduced into the MobileViT model. Deformable convolutional layers can increase the receptive field by introducing offsets, and at the same time, make the receptive field adaptable to image features of different sizes and shapes, thereby improving classification performance.
[0063] 3. Transfer learning was used to train the model on the ImageNet dataset and then transferred to the underground pipeline anomaly sample discrimination model, which enabled the model to fit faster and achieve better classification results.
[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for identifying abnormal samples in underground pipeline images, characterized in that: Includes the following steps: S1. Construct the MobileViT model; the MobileViT model includes an MV2 module and a MobileViT module; S2. Replace the first and last convolutional layers in the MobileViT model with variable convolutional layers to obtain the DeformMobileViT model. S3. Transfer the pre-trained parameters to the underground pipeline anomaly identification task to obtain the optimized DeformMobileViT model; S4. Use a publicly available dataset of underground pipelines to test and optimize the DeformMobileViT model; The structure of the MobileViT model specifically includes the following components connected in sequence: convolutional layer, MV2 module, MV2 module, MV2 module, MV2 module, MobileViT module, MV2 module, MobileViT module, MV2 module, MobileViT module, convolutional layer, global pooling layer, and fully connected layer; The variable convolutional layer adjusts the kernel size according to different input images. n × n Its output is as follows: in, R Indicates the size of the convolution kernel. n Represents the side length of the convolution kernel, P n For any point in the convolution kernel, P 0 is the center point of the convolution kernel, Δ P n For variable offsets, w ( P n ) represents the weight of any point in the convolution kernel; X ( P 0+ P n +Δ P n ) indicates that the center point of the convolution kernel is used as the reference point. P Centered on 0, towards P n +Δ P n Output the offset points.
2. The method for identifying abnormal samples in underground pipeline images as described in claim 1, characterized in that: The MV2 module includes: an extension layer, a depthwise separable convolutional layer, and a linear projection layer.
3. The method for identifying abnormal samples in underground pipeline images as described in claim 1, characterized in that: The MobileViT module includes: a downsampling convolutional layer, a Transformer layer, and a feature fusion unit.
4. The method for identifying abnormal samples in underground pipeline images as described in claim 3, characterized in that: The MobileViT module processes data as follows: S11. Downsample the input data using a downsampling layer; S12. Divide the downsampled data into blocks and then input them into the Transformer layer for feature extraction to obtain global features; S13. The feature fusion unit fuses global features and local features to obtain fused features.
5. The method for identifying abnormal samples in underground pipeline images as described in claim 1, characterized in that: In step S3, when transferring the pre-trained parameters to the underground pipeline anomaly recognition task, all parameters except those of the classification layer and the mismatched network layer are transferred.
6. A storage device, characterized in that: The storage device stores instructions and data to implement the method for identifying abnormal samples in underground pipeline images as described in any one of claims 1 to 5.
7. A device for identifying abnormal samples in underground pipeline images, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the method for identifying abnormal samples in underground pipeline images as described in any one of claims 1 to 5.