Pipeline robot pipe network defect detection method and system based on deep learning

Abstract

The invention discloses a pipeline robot pipe network defect detection method and system based on deep learning, and relates to the technical field of image processing, and the method comprises the steps: collecting and preprocessing a pipeline inner wall image in real time, and constructing a high-quality pipeline inner wall image sample; constructing a shallow classification model to perform binary classification on the high-quality pipeline inner wall image samples, and marking the inner wall image samples containing defects as defect image samples; extracting local features and global features of defect image samples, fusing to obtain fine-grained features, obtaining weights of defects belonging to different defect labels, constructing label characterization, enabling the graph convolutional network to construct a multi-label classification model, adaptively modeling correlation information between the labels, and obtaining a defect label prediction result. According to the method, correlation information between the labels is modeled in a self-adaptive mode through the graph convolutional network, classification of various pipe network defect types is achieved, the efficiency and accuracy of pipe network defect detection are improved, and the actual application requirement is better met.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a pipeline robot network defect detection method and system based on deep learning. Background Art

[0002] The pipe network system is an important part of urban infrastructure and is widely used in fields such as water supply, drainage, gas, petroleum, and chemical industry. The health status of the pipe network is directly related to public safety, environmental protection, and resource utilization efficiency. However, due to the long-term burial of the pipe network underground, it is easily affected by factors such as environmental corrosion, mechanical wear, and geological changes, and defects such as cracks, corrosion, blockage, and leakage are likely to occur, resulting in the failure of the pipe network and even safety accidents. Traditional pipe network defect detection methods mainly rely on manual inspections and ground detection equipment, which have limitations such as low detection efficiency, inability to monitor in real time, and potential safety hazards.

[0003] With the progress of robot technology and artificial intelligence, pipeline robots have gradually become an important tool for pipe network defect detection. Pipeline robots can enter the interior of the pipeline and use a variety of sensors and detection devices to achieve precise detection and real-time monitoring of pipe network defects. Currently, most pipeline robots use image processing, machine learning, and deep learning algorithms to automatically identify and classify pipe network defects (such as cracks, corrosion, blockage). However, image acquisition is restricted by the internal environment of the pipeline, and the image quality is low, resulting in difficult recognition of defect features. Moreover, the current detection methods cannot effectively classify pipelines with a large number of defect categories. Therefore, how to enhance the ability to extract defect features and achieve multi-label classification of pipe network defects is an urgent problem to be solved currently. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a pipeline robot network defect detection method and system based on deep learning, which uses a graph convolutional network to adaptively model the correlation information between labels, realizes the classification of multi-label pipeline defects, and improves the efficiency and accuracy of pipe network defect detection.

[0005] The first aspect of the present invention provides a pipeline robot network defect detection method based on deep learning, including the following steps:

[0006] Collect pipeline inner wall images in real time, preprocess the pipeline inner wall images, and construct high-quality pipeline inner wall image samples through image enhancement;

[0007] Construct a shallow classification model based on data driving, use the shallow classification model to perform binary classification on the high-quality pipeline inner wall image samples, determine whether the image samples contain defects, and mark the inner wall image samples containing defects as defect image samples;

[0008] Extract local features and global features from the defective image samples through an improved YOLOv8 network, fuse them to obtain the fine-grained features of the defective image samples, and obtain the weights of different defect labels according to the fine-grained features;

[0009] Obtain label representations according to the weights of different defect labels, enable the graph convolutional network to construct a multi-label classification model, fuse the label representations with the fine-grained features, adaptively model the correlation information between labels, and obtain the defect label prediction results of the inner wall of the pipeline.

[0010] In this solution, the inner wall image of the pipeline is collected in real time, preprocessed, and a high-quality inner wall image sample of the pipeline is constructed through image enhancement. Specifically:

[0011] Collect the inner wall image of the pipeline through the binocular camera of the pipeline robot, transmit and store the inner wall image of the pipeline in real time through wireless communication, perform geometric correction on the stored inner wall image of the pipeline, perform preliminary denoising using a smoothing filter, and optimize uneven illumination using histogram equalization;

[0012] Use an improved generative adversarial network to perform image super-resolution reconstruction on the preprocessed inner wall image of the pipeline. Replace the generator module in the original generative adversarial network with Res2Net, and extract features of different scales from the input preprocessed inner wall image of the pipeline to obtain shallow image features and deep image features;

[0013] Introduce a channel attention mechanism to perform channel weighting on the obtained shallow image features and deep image features, realize the recalibration of the feature map, fuse the weighted shallow image features and deep image features, and use the upsampling operation to perform super-resolution reconstruction on the fused feature map;

[0014] Import the super-resolution image generated by the super-resolution reconstruction into the discriminator module. In the discriminator module, determine whether the imported image belongs to real data or generated data. Use the training data to alternately train the generator module and the discriminator module of the improved generative adversarial network, and update the parameters until the discriminator module cannot correctly classify the imported image, and then output the trained generative adversarial network;

[0015] Use the generator module in the trained generative adversarial network to obtain the super-resolution image corresponding to the preprocessed inner wall image of the pipeline, evaluate the image using the peak signal-to-noise ratio and the structural similarity. When the image evaluation result meets the preset standard, use the super-resolution image as the high-quality inner wall image sample of the pipeline.

[0016] In this solution, a shallow classification model is constructed based on data-driven. The high-quality inner wall image samples of the pipeline are classified binary using the shallow classification model to determine whether there are defects in the image samples. Specifically:

[0017] Obtain the pipeline type, usage scenario, and service life of the target pipe network to construct basic information. Use the basic information to construct a retrieval label. Through big data methods, obtain pipe network detection instances based on the retrieval label, and extract corresponding detection images from the pipe network detection instances;

[0018] Select defect-free detection images from the detection images according to the detection results of the pipe network detection instances to construct a normal detection image set. Use the normal detection image set to train a stacked autoencoder. Map the normal detection image samples to the hidden layer using the encoder, and use the decoder to reconstruct the hidden layer features to obtain the reconstruction error for layer-by-layer training;

[0019] Stack the pre-trained autoencoders, obtain the overall reconstruction error for fine-tuning, update the parameters of all layers using backpropagation, and output the trained stacked autoencoder as a shallow classification model;

[0020] Take the high-quality inner wall image samples of the pipeline as the input of the shallow classification model, obtain the output of the hidden layer of the last layer for binary classification, obtain the probability that the high-quality inner wall image samples of the pipeline belong to normal detection images, and preset a probability threshold. When the calculated probability is less than the probability threshold, it is determined that there are defects in the high-quality inner wall image samples of the pipeline, and they are marked as defective image samples.

[0021] In this solution, the defective image samples are used to extract local features and global features through an improved YOLOv8 network, and the fine-grained features of the defective image samples are obtained by fusion. Specifically:

[0022] Use the lightweight network MobileViTv3 as the backbone network of YOLOv8. Import the defective image samples into the backbone network, introduce translational variant convolutions to encode local spatial information, distinguish different local spatial information through affinity mapping, and capture the semantic relationships of different local spaces to obtain local features;

[0023] Generate corresponding flattened image patches according to the local features, perform spatial pooling operations on each flattened image patch to obtain the dependency relationships between different image patches, and associate and integrate different local spatial information to obtain global features;

[0024] Fuse the local features and global features using a concatenation operation to obtain a fused feature map. Perform one-dimensional convolution on the concatenated feature map for dimensionality reduction, and use residual connections to combine the dimensionality-reduced feature map with the local features to obtain the fine-grained features of the defective image samples.

[0025] In this solution, label representations are obtained according to the weights of different defect labels, specifically as follows:

[0026] Defect candidate regions are determined based on the fine-grained features of defect image samples to generate candidate boxes, and the candidate boxes that meet the requirements are selected as target anchor boxes. Based on the target anchor boxes, the corresponding target region feature maps are read.

[0027] The selected target region feature maps are regularized, and the weights of different defect labels are obtained using an SVM classifier in the fully connected layer. The weights are convolved with the feature map of the defect image sample after global pooling to obtain label representations including defect features.

[0028] In this solution, defect candidate regions are determined based on the fine-grained features of defect image samples to generate candidate boxes, and the candidate boxes that meet the requirements are selected as target anchor boxes, specifically as follows:

[0029] The center points of the defect target bounding boxes are obtained from the feature maps of different scales corresponding to the fine-grained features in the defect image samples. The regions of a preset size are selected using the center points as defect candidate regions, and the regression loss between the candidate boxes and the bounding boxes in the defect candidate regions is calculated.

[0030] The number of positive samples that the defect target needs to match is set according to the intersection over union (IoU) between the candidate boxes and the bounding boxes. Based on the number of positive samples, the corresponding number of candidate boxes are selected as positive samples according to the regression loss, and the remaining candidate boxes are used as negative samples. The improved YOLOv8 network is trained through the dynamic configuration of positive and negative samples, and the best target anchor box is determined according to the IoU between the defect target bounding box and the candidate box.

[0031] In this solution, a graph convolutional network is used to construct a multi-label classification model, and the label representations and the fine-grained features are feature fused to adaptively model the correlation information between the labels, specifically as follows:

[0032] Pipe network detection instances are retrieved according to the basic information of the target pipe network. The probabilities of different defect labels occurring simultaneously are calculated through the pipe network detection instances, and the relationships between different defect labels are obtained based on the calculated probabilities. The defect labels are used as nodes, and the edge structures between the nodes are constructed according to the relationships between different defect labels to realize the graph representation of different defect labels.

[0033] A graph convolutional network is used to construct a multi-label classification model. An adjacency matrix is constructed based on the graph representations of different defect labels, and the label representations are imported into the multi-label classification model to obtain the feature representations of each defect label node in combination with the adjacency matrix.

[0034] The fine-grained features of the defect image samples are concatenated with the feature representations of each defect label node to obtain the updated feature representations of each defect label node. An adjacency matrix adaptively updated based on the defect image samples is constructed through the updated feature representations of each defect node;

[0035] The multi-head attention mechanism is introduced, and the updated adjacency matrix is used for neighbor aggregation. The attention weights of each neighbor defect label node are generated through the attention heads, and the feature representations of the defect label nodes are weighted and aggregated through the attention weights to obtain the final feature representations of the defect label nodes;

[0036] The final feature representations of the defect label nodes are imported into the fully connected layer for binary classification to obtain the defect label prediction results of the defect image samples.

[0037] The second aspect of the present invention provides a pipeline robot pipe network defect detection system based on deep learning. The system includes: an image acquisition and preprocessing module, a shallow classification module, a label semantic representation construction module, a multi-label classification module, and a detection result output module;

[0038] The image acquisition and preprocessing module is responsible for real-time collecting the inner wall images of the pipeline for transmission and storage, preprocessing the inner wall images of the pipeline, and constructing high-quality inner wall image samples of the pipeline;

[0039] The shallow classification module is responsible for performing binary classification on the high-quality inner wall image samples of the pipeline to determine whether the image samples contain defects, and marking the inner wall image samples containing defects as defect image samples;

[0040] The label semantic representation construction module is responsible for extracting the local features and global features of the defect image samples through the improved YOLOv8 network, fusing and constructing fine-grained features for class awareness, obtaining the weights of different defect labels, and obtaining label representations according to the weights of different defect labels;

[0041] The multi-label classification module is responsible for enabling the graph convolutional network to fuse the label representations and the fine-grained features, adaptively modeling the correlation information between the labels, and obtaining the defect label prediction results of the inner wall of the pipeline;

[0042] The detection result output module is responsible for outputting the defect categories existing in the inner wall images of the pipeline, and visually annotating and displaying the corresponding defects and defect labels in a preset manner.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] The present invention significantly improves the extraction effect of defect features in low-quality images through image enhancement and the improved YOLOv8 network, and adapts to the complex and changeable pipe network environment. The graph convolutional network is used to adaptively model the correlation information between labels, realizing the classification of various pipe network defect types, improving the efficiency and accuracy of pipe network defect detection, and better meeting the actual application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments or the exemplary of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the exemplary description. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings shown.

[0046] Figure 1 The flowchart of the pipe network defect detection method of the pipeline robot based on deep learning is shown;

[0047] Figure 2 The flowchart of constructing a shallow classification model to determine whether the image sample contains defects is shown;

[0048] Figure 3 The flowchart of constructing a multi-label classification model for multi-label classification of pipe network defects is shown;

[0049] Figure 4 The block diagram of the pipe network defect detection system of the pipeline robot based on deep learning is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] In order to be able to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below in conjunction with the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0051] Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0052] Figure 1 The flowchart of the pipe network defect detection method of the pipeline robot based on deep learning is shown.

[0053] As Figure 1 shown, in the first embodiment of the present invention, a pipe network defect detection method of the pipeline robot based on deep learning is provided, including:

[0054] S102, Collect the inner wall image of the pipeline in real time, preprocess the inner wall image of the pipeline, and construct a high-quality inner wall image sample of the pipeline through image enhancement;

[0055] S104, Build a shallow classification model based on data driving, use the shallow classification model to perform binary classification on the high-quality inner wall image sample of the pipeline, determine whether the image sample contains defects, and mark the inner wall image sample containing defects as a defective image sample;

[0056] S106, Extract local features and global features of the defective image sample through the improved YOLOv8 network, fuse them to obtain the fine-grained features of the defective image sample, and obtain the weights of different defect labels according to the fine-grained features;

[0057] S108, Obtain label representations according to the weights of different defect labels, enable the graph convolutional network to build a multi-label classification model, fuse the label representations with the fine-grained features, adaptively model the correlation information between labels, and obtain the defect label prediction result of the inner wall of the pipeline.

[0058] It should be noted that during the movement of the pipeline robot, the inner wall image of the pipeline is collected by the binocular camera of the pipeline robot, and the inner wall image of the pipeline is transmitted in real time through wireless communication (such as 5G, Wi-Fi, LoRa), transmitted to the ground control center and stored. When transmitting over a long distance or under complex environmental conditions, relay devices or fiber optic sensing technology are used. The image data is stored locally or in the cloud for subsequent processing and analysis. The data is encrypted and backed up to ensure data security and integrity. After geometric correction of the stored inner wall image of the pipeline, preliminary denoising is performed using smoothing filters such as Gaussian filtering and median filtering, and histogram equalization is used to optimize uneven illumination, initially improving the image quality.

[0059] In order to further improve the image quality, an improved generative adversarial network is used to perform image super-resolution reconstruction on the preprocessed inner wall image of the pipeline. The Res2Net is used to replace the generator module in the original generative adversarial network. The skip connection of the residual network is introduced to further increase the depth of the generator module, and at the same time, the problem of gradient disappearance is solved. Res2Net performs feature extraction of different scales on the preprocessed inner wall image of the pipeline to obtain shallow image features and deep image features. The channel attention mechanism SENet is introduced in Res2Net to perform channel weighting on the obtained shallow image features and deep image features, realize the recalibration of the feature map, improve the sensitivity of the model to channel features, fuse the weighted shallow image features and deep image features, and use the upsampling operation to perform super-resolution reconstruction on the fused feature map; the super-resolution image generated by the super-resolution reconstruction is imported into the discriminator module. In the discriminator module, the cross-entropy between the output value of the discriminator module and the real value is used to determine whether the imported image belongs to real data or generated data. The generator module and the discriminator module of the improved generative adversarial network are alternately trained using the training data to update the parameters until the discriminator module cannot correctly classify the imported image, and then the trained generative adversarial network is output; the generator module in the trained generative adversarial network is used to obtain the super-resolution image corresponding to the preprocessed inner wall image of the pipeline, and the peak signal-to-noise ratio and structural similarity are used to evaluate the super-resolution image. When the image evaluation result meets the preset standard, the super-resolution image is used as a high-quality inner wall image sample of the pipeline. The peak signal-to-noise ratio takes the original image as a reference and calculates the difference in pixel values between the target image and the original image. The larger the value, the closer the target image is to the original image. The structural similarity obtains the structural similarity between the target image and the original image from three aspects: image brightness, contrast, and structure. Its index range is 0-1, and the larger the value, the closer the target image is to the original image.

[0060] Figure 2 The flowchart of constructing a shallow classification model to determine whether the image sample contains defects is shown.

[0061] According to an embodiment of the present invention, a shallow classification model is constructed based on data-driven, and the high-quality inner wall image sample of the pipeline is binary-classified using the shallow classification model to determine whether the image sample contains defects. Specifically:

[0062] S202, obtain the pipeline type, usage scenario, and service life of the target pipeline network to construct basic information, use the basic information to construct a retrieval label, obtain pipeline network detection instances based on the retrieval label through the big data method, and extract the corresponding detection images from the pipeline network detection instances;

[0063] S204. Select defect-free detection images from the detection images according to the detection results of the pipeline network detection instances to construct a normal detection image set. Use the normal detection image set to train a stacked autoencoder. Map the normal detection image samples to the hidden layer using the encoder, and use the decoder to reconstruct the hidden layer features to obtain the reconstruction error for layer-by-layer training.

[0064] S206. Stack the pre-trained autoencoders to obtain the overall reconstruction error for fine-tuning. Use backpropagation to update the parameters of all layers, and output the trained stacked autoencoder as a shallow classification model.

[0065] S208. Use the high-quality pipeline inner wall image samples as the input of the shallow classification model, obtain the output of the hidden layer of the last layer for binary classification, and obtain the probability that the high-quality pipeline inner wall image samples belong to the normal detection images. Preset a probability threshold. When the calculated probability is less than the probability threshold, it is determined that there are defects in the high-quality pipeline inner wall image samples, and they are marked as defective image samples.

[0066] It should be noted that historical pipeline network detection data similar to the pipeline network type, usage scenario, and service life of the target pipeline network is obtained through big data retrieval means. Defect-free pipeline network detection images are extracted from the pipeline network detection instances to construct a training sample to build a data-driven model. A stacked autoencoder is selected to establish a shallow classification model. Since the training samples are all detection images without defects under normal conditions, when the high-quality pipeline inner wall image samples are input into the shallow classification model, the shallow classification model is used to obtain the estimated image of the image samples under normal and defect-free conditions. If the input data deviates too much from the estimated image, it is determined that there are defects in the high-quality pipeline inner wall image samples. A binary classifier is added to the top layer of the trained stacked autoencoder, and the output of the hidden layer of the last layer is used as the input of the binary classifier to obtain the corresponding classification result. If there are defects, they are sent to the subsequent stage for multi-label classification of defects.

[0067] It should be noted that the lightweight network MobileViTv3 is used as the backbone network of YOLOv8. MobileViTv3 is mainly composed of MobileNetv2 blocks and MobileViT blocks. The MobileViT block is composed of a local representation module, a global representation module, and a feature fusion module. In the local representation module and the global representation module, the local features and global features of the input features are modeled through convolution and Linear Transformer respectively. The feature fusion module uses a concatenation operation to achieve the efficient fusion of local features and global features, and establishes the correlation between global information and local information.

[0068] Import the defective image sample into the backbone network. Introduce shift-variant convolution in the local feature module to replace the original convolution for encoding local spatial information. Distinguish different local spatial information through affinity mapping and capture the semantic relationships of different local spaces to obtain local features. The core of the shift-variant convolution is the convolution weight block specified by the affinity mapping. The affinity mapping implicitly captures the semantic relationships between different regions by learning to distinguish various local features on the feature map and maintains the feature weights at different positions in the image, which can improve the network's ability to recognize key part features. Expand and generate n non-overlapping flattened image patches according to the local features. Introduce the PoolFormer network to replace the Linear Transformer in the global feature module, perform spatial pooling operations on each flattened image patch, obtain the dependency relationships between different image patches, associate and integrate different local spatial information, obtain global features, significantly reduce the number of parameters and computational complexity, and improve the extraction efficiency of global context features. Use a concatenation operation to fuse the local features and global features through a feature fusion module to obtain a fused feature map. Perform one-dimensional convolution on the concatenated feature map for dimensionality reduction, and use residual connection to combine the dimensionality-reduced feature map with the local features to obtain the fine-grained features of the defective image sample.

[0069] To improve the detection performance of the YOLOv8 network and adapt to the multi-label detection task of pipeline defects, dynamic matching of positive and negative samples is carried out to obtain accurate defect anchor boxes in high-quality pipeline inner wall image samples. The dynamic label assignment strategy dynamically adjusts the assignment of positive and negative samples according to the matching degree between the model's prediction results and the ground truth boxes, which can better adapt to the shapes and sizes of different defect targets. Obtain the center points of the true bounding boxes of defect targets in the feature maps of different scales corresponding to the fine-grained features in the defective image sample. Use the center points to select a region of a preset size as the defect candidate region, and calculate the regression loss between the candidate box and the bounding box in the defect candidate region. Sum and round up according to the intersection over union (IoU) between the candidate box and the bounding box, set the number of positive samples that the defect target needs to match, select the corresponding number of candidate boxes as positive samples based on the regression loss according to the number of positive samples, and use the remaining candidate boxes as negative samples. Train the improved YOLOv8 network through the dynamic configuration of positive and negative samples, so that the improved YOLOv8 network can better adapt to the differences between different defect targets. Determine the best target anchor box according to the IoU between the defect target bounding box and the candidate box, and read the corresponding target region feature map based on the target anchor box. Regularize the selected target region feature map, and use an SVM classifier in the fully connected layer to obtain the weights of different defect labels. Convolve the weights with the feature map of the defective image sample after global pooling to obtain the label representation including defect features.

[0070] Figure 3The flowchart of constructing a multi-label classification model for multi-label classification of pipeline network defects is shown.

[0071] According to an embodiment of the present invention, a graph convolutional network is used to construct a multi-label classification model, and the label representation and the fine-grained features are fused in terms of features, and the correlation information between labels is adaptively modeled. Specifically:

[0072] S302, retrieve pipeline network detection instances according to the basic information of the target pipeline network, calculate the probability of different defect labels occurring simultaneously through the pipeline network detection instances, obtain the relationship between different defect labels according to the calculated probability, use the defect labels as nodes, and construct an edge structure between the nodes according to the relationship between different defect labels to realize the graph representation of different defect labels;

[0073] S304, use a graph convolutional network to construct a multi-label classification model, construct an adjacency matrix based on the graph representation of different defect labels, import the label representation into the multi-label classification model, and obtain the feature representation of each defect label node in combination with the adjacency matrix;

[0074] S306, splice the fine-grained features of the defect image samples with the feature representations of each defect label node to obtain the updated feature representations of each defect label node, and construct an adjacency matrix adaptively updated based on the defect image samples through the updated feature representations of each defect node;

[0075] S308, introduce a multi-head attention mechanism, use the updated adjacency matrix for neighbor aggregation, generate the attention weights of each neighbor defect label node through the attention heads, and perform weighted aggregation on the feature representations of the defect label nodes through the attention weights to obtain the final feature representation of the defect label nodes;

[0076] S310, import the final feature representation of the defect label nodes into a fully connected layer for binary classification to obtain the defect label prediction result of the defect image samples.

[0077] It should be noted that the graph convolution network is used to dynamically update the relevance of the defect labels, and the relationship between the defect labels is used to construct an adjacency matrix for representation learning, and the feature representation of the defect label node is obtained. The fine-grained features of the defect image sample are spliced ​​with the feature representation of each defect label node to obtain the updated feature representation of each defect label node. The weight is updated by updating the state after the convolution operation, and the adjacency matrix based on the defect image sample is adaptively updated through the Sigmoid activation function. The final feature representation obtained by the adaptively updated adjacency matrix is ​​accompanied by the defect classification features in the defect image sample and contains the relevance information of other defect labels. In the process of obtaining the final feature representation, the multi-head attention mechanism is introduced to dynamically aggregate the neighbor information of the defect label node, improve the expression ability of the model, and make the model pay more attention to the defect feature nodes with high weights, thereby improving the performance of the label multi-classification prediction task. The final feature representation of each defect label node is placed in the binary classifier to obtain the defect label prediction result of the defect image sample, and the multi-label classification model is optimized using the binary cross entropy. The model training parameters are preset. During the training process, the obtained defect label prediction result is used as one of the inputs of the loss function. The parameter values ​​of the multi-label classification model will be updated through the back propagation of the loss function until the number of training times reaches the preset maximum number of training batches. Finally, the output defect label prediction results are used to annotate the defect anchor boxes and visualize them in the preset way.

[0078] Figure 4 A block diagram of a pipeline robot network defect detection system based on deep learning is shown.

[0079] The second embodiment of the present invention provides a pipeline robot network defect detection system 4 based on deep learning, which includes: an image acquisition and preprocessing module 401, a shallow classification module 402, a label semantic representation construction module 403, a multi-label classification module 404 and a detection result output module 405;

[0080] The image acquisition and preprocessing module is responsible for real-time acquisition of pipeline inner wall images for transmission and storage, preprocessing of pipeline inner wall images, and construction of high-quality pipeline inner wall image samples;

[0081] The shallow classification module is responsible for performing binary classification on the high-quality pipeline inner wall image samples, determining whether the image samples contain defects, and marking the inner wall image samples containing defects as defective image samples;

[0082] The label semantic representation construction module is responsible for extracting local features and global features of defect image samples through the improved YOLOv8 network, fusing and constructing fine-grained features for category perception, obtaining weights of different defect labels, and obtaining label representations according to the weights of different defect labels;

[0083] The multi-label classification module is responsible for enabling the graph convolutional network to perform feature fusion on the label representation and the fine-grained features, adaptively modeling the correlation information between labels, and obtaining the prediction results of the defect labels on the inner wall of the pipeline;

[0084] The detection result output module is responsible for outputting the defect categories existing in the inner wall image of the pipeline, and visually annotating and displaying the corresponding defects and defect labels in a preset manner.

[0085] The third embodiment of the present invention provides a computer-readable storage medium, which includes a program for the method for detecting pipeline robot pipe network defects based on deep learning. When the program for the method for detecting pipeline robot pipe network defects based on deep learning is executed by a processor, the steps of the method for detecting pipeline robot pipe network defects based on deep learning are implemented.

[0086] In several embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0087] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: mobile storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks, or optical disks and other various media that can store program codes.

[0088] Alternatively, if the above-mentioned integrated module of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as removable storage devices, ROM, RAM, magnetic disks, or optical discs.

[0089] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A pipeline robot pipeline network defect detection method based on deep learning, characterized in that: The following steps are involved: Acquire pipeline inner wall images in real time, preprocess the pipeline inner wall images, and construct high-quality pipeline inner wall image samples through image enhancement; Building a shallow classification model based on data drive, using the shallow classification model to perform binary classification on the high-quality pipeline inner wall image samples, determining whether the image samples contain defects, and marking the inner wall image samples containing defects as defective image samples; Extract local features and global features from the defect image samples through an improved YOLOv8 network, fuse them to obtain fine-grained features of the defect image samples, and obtain weights of different defect labels according to the fine-grained features; The label representation is obtained according to the weights of different defect labels, and the graph convolutional network is used to build a multi-label classification model. The label representation is fused with the fine-grained features, and the correlation information between the labels is adaptively modeled to obtain the defect label prediction result of the inner wall of the pipeline.

2. The pipeline network defect detection method based on deep learning by pipeline robots according to claim 1 is characterized in that: The pipeline inner wall image is collected in real time, the pipeline inner wall image is preprocessed, and a high-quality pipeline inner wall image sample is constructed by image enhancement, specifically: The pipeline robot collects images of the inner wall of the pipeline through a binocular camera, transmits and stores the images of the inner wall of the pipeline in real time through wireless communication, performs geometric correction on the stored images of the inner wall of the pipeline, performs preliminary denoising using a smoothing filter, and uses histogram equalization to optimize uneven illumination; The improved generative adversarial network is used to perform image super-resolution reconstruction on the preprocessed pipeline inner wall image. Res2Net is used to replace the generator module in the original generative adversarial network, and features of different scales are extracted from the input preprocessed pipeline inner wall image to obtain shallow image features and deep image features. The channel attention mechanism is introduced to perform channel weighting on the acquired shallow image features and deep image features, so as to realize the recalibration of feature maps, fuse the weighted shallow image features and deep image features, and use upsampling operation to reconstruct the fused feature maps for super-resolution. Importing the super-resolution image generated by super-resolution reconstruction into the discriminator module, the discriminator module determines whether the imported image belongs to real data or generated data, and alternately trains the generator module and the discriminator module of the improved generative adversarial network using the training data to update the parameters until the discriminator module cannot correctly classify the imported image, and then outputs the trained generative adversarial network; The generator module in the trained generative adversarial network is used to obtain a super-resolution image corresponding to the preprocessed pipeline inner wall image, and the peak signal-to-noise ratio and structural similarity are used to perform image evaluation on the super-resolution image. When the image evaluation result meets the preset standard, the super-resolution image is used as a high-quality pipeline inner wall image sample.

3. The pipeline network defect detection method based on deep learning by pipeline robots according to claim 1 is characterized in that: A shallow classification model is constructed based on data-driven, and the shallow classification model is used to perform binary classification on the high-quality pipeline inner wall image samples to determine whether the image samples contain defects, specifically: Obtain basic information of the target pipe network, including pipe network type, usage scenario, and service life, use the basic information to construct search tags, obtain pipe network detection instances based on the search tags through a big data method, and extract corresponding detection images from the pipe network detection instances; According to the inspection results of the pipe network inspection instance, defect-free inspection images are selected from the inspection images to construct a normal inspection image set, the normal inspection image set is used to train a stacked autoencoder, the normal inspection image samples are mapped to a hidden layer using an encoder, the hidden layer features are reconstructed using a decoder, and the reconstruction error is obtained for layer-by-layer training; The pre-trained autoencoders are stacked to obtain the overall reconstruction error for fine-tuning, and the parameters of all layers are updated using back propagation. The trained stacked autoencoders are output as shallow classification models. The high-quality pipeline inner wall image sample is used as the input of the shallow classification model, and the output of the last hidden layer is obtained for binary classification to obtain the probability that the high-quality pipeline inner wall image sample belongs to a normal detection image. A probability threshold is preset. When the calculated probability is less than the probability threshold, it is determined that there is a defect in the high-quality pipeline inner wall image sample and it is marked as a defective image sample.

4. The pipeline network defect detection method based on deep learning by pipeline robot according to claim 1 is characterized in that: The defect image sample is subjected to the improved YOLOv8 network to extract local features and global features, and the fine-grained features of the defect image sample are obtained by fusion, specifically: A lightweight network MobileViTv3 is used as the backbone network of YOLOv8, the defect image samples are imported into the backbone network, and the translation variant convolution is introduced to encode the local spatial information. Different local spatial information is distinguished through affinity mapping and the semantic relationship of different local spaces is captured to obtain local features. Generate corresponding flattened image blocks according to the local features, perform spatial pooling operations on each flattened image block, obtain dependency relationships between different image blocks, associate and integrate different local spatial information, and obtain global features; The local features are fused with the global features using a splicing operation to obtain a fused feature map, the spliced ​​feature map is subjected to one-dimensional convolution dimensionality reduction, and the reduced-dimensional feature map is combined with the local features using a residual connection to obtain fine-grained features of the defect image sample.

5. The pipeline network defect detection method based on deep learning by pipeline robot according to claim 1 is characterized in that: The label representation is obtained according to the weights of different defect labels, specifically: Determine the defect candidate area to generate a candidate frame according to the fine-grained features of the defect image sample, select the candidate frame that meets the requirements as the target anchor frame, and read the corresponding target area feature map based on the target anchor frame; The selected target area feature map is regularized, and the weights of different defect labels are obtained using the SVM classifier in the fully connected layer. The weights are convolved with the feature map of the defect image sample after global pooling to obtain a label representation including the defect features.

6. The pipeline network defect detection method based on deep learning by pipeline robots according to claim 5 is characterized in that: Determine the defect candidate area and generate a candidate frame based on the fine-grained features of the defect image sample, and select the candidate frame that meets the requirements as the target anchor frame, specifically: Obtaining the center point of the defect target bounding box in the feature maps of different scales corresponding to the fine-grained features in the defect image sample, selecting an area of ​​a preset size as the defect candidate area using the center point, and calculating the regression loss of the candidate box and the bounding box in the defect candidate area; The number of positive samples that need to be matched with the defect target is set according to the intersection-and-union ratio of the candidate box and the bounding box. Based on the number of positive samples, a corresponding number of candidate boxes are selected as positive samples according to the regression loss, and the remaining candidate boxes are used as negative samples. The improved YOLOv8 network is trained through dynamic configuration of positive and negative samples, and the optimal target anchor box is determined according to the intersection-and-union ratio of the defect target bounding box and the candidate box.

7. The pipeline network defect detection method based on deep learning by pipeline robot according to claim 1 is characterized in that: The graph convolutional network is used to construct a multi-label classification model, the label representation is fused with the fine-grained features, and the correlation information between the labels is adaptively modeled, specifically: Retrieve the basic information of the target pipe network to obtain a pipe network detection instance, calculate the probability of different defect labels appearing at the same time through the pipe network detection instance, obtain the relationship between different defect labels according to the calculated probability, take the defect label as a node, and construct the edge structure between the nodes according to the relationship between different defect labels, so as to realize the graph representation of different defect labels; A multi-label classification model is constructed using a graph convolutional network, an adjacency matrix is ​​constructed based on the graph representations of different defect labels, the label representations are imported into the multi-label classification model, and the feature representation of each defect label node is obtained in combination with the adjacency matrix; The fine-grained features of the defect image samples are concatenated with the feature representations of each defect label node to obtain the updated feature representations of each defect label node, and an adjacency matrix based on the adaptive update of the defect image samples is constructed through the updated feature representations of each defect node. A multi-head attention mechanism is introduced, and the updated adjacency matrix is ​​used to perform neighbor aggregation. The attention weight of each neighbor defect label node is generated through the attention head. The feature representation of the defect label node is weightedly aggregated through the attention weight to obtain the final feature representation of the defect label node. The final feature representation of the defect label node is imported into the fully connected layer for binary classification to obtain the defect label prediction results of the defect image samples.

8. A pipeline robot network defect detection system based on deep learning, characterized in that: Used to implement the pipeline robot pipeline network defect detection method based on deep learning as described in any one of claims 1 to 7, the system includes: an image acquisition and preprocessing module, a shallow classification module, a label semantic representation construction module, a multi-label classification module and a detection result output module; The image acquisition and preprocessing module is responsible for real-time acquisition of pipeline inner wall images for transmission and storage, preprocessing of pipeline inner wall images, and construction of high-quality pipeline inner wall image samples; The shallow classification module is responsible for performing binary classification on the high-quality pipeline inner wall image samples, determining whether the image samples contain defects, and marking the inner wall image samples containing defects as defective image samples; The label semantic representation construction module is responsible for extracting local features and global features of defect image samples through the improved YOLOv8 network, fusing and constructing fine-grained features for category perception, obtaining weights of different defect labels, and obtaining label representations according to the weights of different defect labels; The multi-label classification module is responsible for enabling the graph convolutional network to perform feature fusion on the label representation and the fine-grained features, adaptively modeling the correlation information between the labels, and obtaining the defect label prediction result of the inner wall of the pipeline; The detection result output module is responsible for outputting the defect categories present in the pipeline inner wall image, and visually annotating and displaying the corresponding defects and defect labels in a preset manner.

Images