Pipeline leakage state recognition system and method based on deep learning

Abstract

The invention discloses a deep learning-based pipeline leakage state recognition system and method, and the method comprises the steps: firstly collecting a pipeline image from a measurement point through an inspection robot, carrying out the preprocessing of the image, and transmitting the image into a neural network for processing; the neural network comprises three parts, namely a backbone network, a segmentation branch and a detection branch. A backbone network extracts basic features of a picture and then enters a segmentation branch, and the segmentation branch performs fine segmentation on a pipeline region in an input image to generate a corresponding segmentation mask. And multiplying the segmented mask domain by the feature map extracted by the backbone network, and only retaining the feature value of the pipeline region. And then the characteristic value is sent to a detection branch, and finally whether a leakage target exists in the pipeline or not is checked through detection. Compared with manual inspection, the detection efficiency is greatly improved, and the method has important significance for ensuring the leakproofness of the pipeline and the safety of equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline inspection, and in particular to a pipeline leakage status recognition system and method based on deep learning. Background Art

[0002] In order to help the power industry inspection robots automatically identify whether there are water or oil leaks in the pipelines in power sites during the inspection process, the robots need to use their own cameras to take pictures of the pipelines, and then analyze these pictures to identify whether there are leaks in the pipelines.

[0003] In power plants, the safety of water or oil pipelines is crucial to the normal operation of equipment. Power equipment usually needs to operate continuously, and these pipeline systems are responsible for providing critical cooling water or lubricating oil. Once leakage occurs, insufficient cooling water will cause the equipment to overheat and shut down, affecting equipment operation; and lubricating oil leakage may cause increased wear of mechanical equipment and cause failures. The shape of the pipeline is relatively irregular and can easily be confused with background information. Traditional manual inspection methods require inspectors to carefully check every detail of the pipeline, which can easily lead to omissions. Therefore, it is of great significance to develop an efficient, real-time, and automated detection method to ensure pipeline tightness and equipment safety. Summary of the invention

[0004] In order to solve the problems of the prior art, the present invention provides a pipeline leakage status recognition system and method based on deep learning. A multi-task network model system is designed by adopting deep learning technology. The model analyzes the collected photos to determine whether there is leakage in the pipeline. Compared with manual inspection, the detection efficiency is greatly improved, which is of great significance for ensuring the airtightness of the pipeline and the safety of equipment.

[0005] The invention provides a pipeline dripping state recognition system based on deep learning, which is characterized by comprising a shooting module, a picture preprocessing module, a basic feature extraction module, a pipeline segmentation module and a dripping state recognition module; the shooting module shoots a pipeline image; the picture preprocessing module preprocesses the shot pipeline image; the basic feature extraction module is a U-shaped network, the U-shaped network is divided into two parts, a coding module and a decoding module, the coding module is responsible for extracting the features of the preprocessed image, gradually reducing the input image and extracting high-level features; the decoding part gradually enlarges the features to the size of the original image through upsampling and skip links; the pipeline segmentation module segments the pipeline through the segmentation branch of the network, obtains the mask of the area where the pipeline is located, multiplies the obtained mask with the feature map obtained by the basic feature extraction module, obtains the pipeline area feature, and shields the related background area feature; the dripping state recognition module processes the feature map, outputs the inference result, and then performs a maximum value suppression operation on the inference result to finally obtain the pipeline dripping recognition result.

[0006] The present invention also provides a pipeline leakage state recognition method based on deep learning, comprising the following steps:

[0007] Step 1) The shooting module of the inspection robot shoots the pipeline image and sends it to the image preprocessing module to preprocess the image and obtain the input data of the image;

[0008] Step 2) Input the input data preprocessed in step 1) into the backbone network of the basic feature extraction module. The backbone network is a U-shaped network, including an encoding module and a decoding module. The encoding module is responsible for extracting features, gradually reducing the input image and extracting high-level features; the decoding part gradually enlarges the features to the size of the original image through upsampling and skip links to obtain a feature map;

[0009] Step 3) The pipeline segmentation module segments the pipeline through the segmentation branch of the network, obtains the mask of the area where the pipeline is located, and then multiplies the obtained mask with the feature map obtained in step 2) to obtain the pipeline area feature map while shielding the relevant background area features;

[0010] Step 4) The pipeline area feature map obtained in step 3) is processed by the dripping state recognition module. After passing through several layers of convolution and detection heads, the inference result is output, and then the maximum value suppression operation is performed on the inference result to finally obtain the pipeline dripping recognition result.

[0011] Further improvement, in the preprocessing process of step 1), the image is first resized and padded to ensure that the input data meets the neural network input requirements, and then the adjusted image is normalized, the mean is subtracted and divided by the variance to obtain the input data of the image, specifically:

[0012] 1.1) Adjust the image size, adjust the long side of the image to 512 pixels, and adjust the short side to the corresponding size proportionally;

[0013] 1.2) Fill the short board to ensure that the size of the image sent to the network is 512*512;

[0014] 1.3) Divide each pixel value of the three channels by 255 to make the pixel value range between 0 and 1. Then subtract the mean 0.485, 0.456, and 0.406 from the R channel, G channel, and B channel respectively, and then divide them by the standard deviation 0.229, 0.224, and 0.225 to obtain the final input of the network.

[0015] Further improvement, the backbone network of the basic feature extraction module in step 2) is a U-shaped network, which includes an encoding part and a decoding part. The specific extraction process is as follows:

[0016] Step 2.1) The encoding part includes 3 stages. In each stage, the size of the feature map gradually decreases, but the number of channels gradually increases. The resolution changes from 512*512 to 64*64, and the number of channels finally becomes 512.

[0017] Step 2.11) Perform convolution operation on the input 3-channel image of size 512*512 to change the number of channels to 64;

[0018] Step 2.12) The special diagnosis is further extracted through the Bottleneck module. Using the residual structure, the Bottleneck module first uses 1x1 convolution for dimensionality increase, then uses depthwise separable convolution for convolution operation, then uses 1x1 convolution for dimensionality reduction, and finally performs a short-circuit operation; the activation function of Bottleneck is FRelu, and its expression is as follows:

[0019] y=FRelu(x)=max(x,f(x))

[0020] Where f(x) is a convolution based on a local neighborhood, expressed as follows:

[0021] f(x) = W*x+b;

[0022] Step 2.13) performs downsampling operation. Based on the Bottleneck module in step 2.12), the step size is set to 2 during convolution to halve the size of the feature map to better extract global features.

[0023] Step 2.2) The decoding part includes 3 stages. In each stage, the size of the feature map gradually decreases, but the number of channels gradually decreases. The feature map resolution becomes 512*512 and the number of channels becomes 192. After the convolution operation, the backbone network finally outputs a feature map of 512*512*64.

[0024] Step 2.21) Perform bilinear interpolation upsampling on the 64*64 feature map obtained from the encoding part to increase the resolution to 128*128;

[0025] Step 2.22) performs concat operation with the feature map of the same resolution corresponding to the encoded part, and the number of channels of the feature map becomes 768. After the convolution operation, the number of channels becomes 256.

[0026] Step 2.23) Further extract features through the Bottleneck module to make the feature map resolution become 512*512 and the number of channels become 192. After the convolution operation, the backbone network finally outputs a feature map of 512*512*64.

[0027] Further improvement, step 3) the pipeline segmentation module segments the pipeline through the segmentation branch of the network. The specific process is as follows: the pipeline segmentation module performs two convolution operations on the feature map output by the backbone network, and finally obtains a single-channel output map. The size of each value in the output map represents the probability value of whether the point is a pipeline. The threshold is 0.5, and values ​​greater than 0.5 are considered to be the area where the pipeline is located, and finally a segmentation map of the pipeline is obtained; the feature value of the pipeline area in the segmentation map is set to 1, and the non-pipeline area is set to 0; after the setting is completed, it is multiplied with the 64-channel feature map output by the backbone network to obtain a pipeline area feature map.

[0028] Further improvement, in step 4), the specific process of processing the pipeline area feature map obtained in step 3) by the dripping state recognition module is as follows:

[0029] Step 4.1) Send the pipeline area feature map obtained in step 3) to the detection branch. After 4 convolutions and downsampling, the detection branch obtains a 64*64*512d feature map, and sends the feature map to the detection head module;

[0030] Step 4.2) The detection head performs a convolution operation on the input feature map. The size of the output feature map is 64*64*54. The 54 channels indicate that each pixel outputs 9 groups of prediction values. Each group represents a possible detection result. Each group has 6 outputs. The first value represents the foreground score, followed by the next 4 values ​​representing the target coordinate prediction values, and the last value represents the drip prediction score value.

[0031] Step 4.3) If the foreground probability exceeds the set threshold and the drip classification score also exceeds the threshold, it means that the predicted value is a drip detection target, and the coordinate value indicates the location of the drip;

[0032] Step 4.4) After obtaining all the drip target prediction values, perform a maximum value suppression operation to remove redundant and repeated predictions;

[0033] Step 4.5) If a leaking target is detected, it means that there is a leaking phenomenon in the pipe in this picture.

[0034] The beneficial effects of the present invention are:

[0035] 1. Compared with manual inspection, the detection efficiency is greatly improved.

[0036] 2. Using deep learning technology, a multi-task network model system was designed. The model analyzes the collected photos to determine whether there is leakage in the pipeline. The environment where the pipeline is located in the power plant is relatively complex, and there is a lot of background information. If the picture is directly detected, it will greatly affect the recognition accuracy. Removing background information through deep learning technology greatly improves the recognition accuracy, which is of great significance for ensuring the tightness of the pipeline and the safety of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a flowchart of the overall execution of the present invention;

[0039] Figure 2 Identify network diagrams for pipeline leaks;

[0040] Figure 3 This is the Bottleneck module structure diagram;

[0041] Figure 4 This is the structure diagram of the downsample module;

[0042] Figure 5 This is the structural diagram of the detection head;

[0043] Figure 6 Schematic diagram of the picture taken for the inspection robot. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] A specific embodiment of the present invention is as follows Figure 1 As shown, the specific process is as follows:

[0046] Step 1: Image preprocessing module:

[0047] Step 1.1: When the inspection robot is inspecting, it usually patrols each location by taking photos at fixed points. When there is a pipeline at the shooting point, it will take pictures like Figure 6The visible light picture shown. The color texture of the pipe drip target is generally light and easily disturbed by background information. Before reasoning on the picture, the size needs to be adjusted. The long side of the picture is adjusted to 512 pixels, and the short side is adjusted to the corresponding size in proportion. After the adjustment, the short board is filled to ensure that the size of the picture sent to the network is 512*512. Next, each pixel value of the three channels is divided by 255. Make the pixel value range between 0 and 1. Then subtract the mean 0.485, 0.456, and 0.406 from the R channel, G channel, and B channel respectively, and then divide them by the standard deviation 0.229, 0.224, and 0.225 to get the final input of the network.

[0048] Step 2 Basic feature extraction module:

[0049] Step 2.1: The basic feature extraction module mainly extracts basic features through the backbone network, such as Figure 2 As shown in the figure, the backbone network is a U-shaped network, which includes an encoding part and a decoding part. The encoding part consists of three stages, and the size of the feature map gradually decreases in each stage, but the number of channels gradually increases. In the first stage, the input 3-channel image with a size of 512*512 is convolved, and the number of channels becomes 64. The feature map is then further extracted through the Bottleneck module. Bottleneck is a bottleneck operation, such as Figure 3 This layer uses a residual structure, which can avoid network degradation while deepening the network. Bottleneck first uses 1x1 convolution for dimensionality increase, then uses depthwise separable convolution for convolution operation, then uses 1x1 convolution for dimensionality reduction, and finally performs a short-circuit operation. The depthwise separable convolution used in the Bottleneck operation can greatly reduce the parameters of the model, thereby saving computing power and improving the model prediction speed. The activation function of Bottleneck is FRelu, and its expression is as follows:

[0050] y=FRelu(x)=max(x,f(x))

[0051] Where f(x) is a convolution based on a local neighborhood, expressed as follows:

[0052] f(x)=W*x+b

[0053] After completing the first stage, the downsampling operation is performed. The downsampling operation is similar to the Bottleneck module, such as Figure 4 As shown, the stride is set to 2 during convolution. The purpose of downsampling is to halve the size of the feature map to better extract global features. After completing the three stages in sequence, the resolution changes from 512*512 to 64*64, and the number of channels finally becomes 512.

[0054] Step 2.2, after completing the encoding module, enter the decoding part. The decoding part is also divided into 3 stages. Starting from the 64*64 feature map obtained by encoding, a bilinear interpolation upsampling operation is first performed to increase the resolution to 128*128. A concat operation is performed on the feature map with the same resolution corresponding to the encoding part, and the number of channels of the feature map becomes 768. Then a convolution operation is performed, and the number of channels becomes 256. Finally, a Bottleneck operation is performed to further extract features. After completing the three stages in sequence, the feature map resolution becomes 512*512 and the number of channels becomes 192. After the convolution operation, the backbone network outputs a 512*512*64 feature map.

[0055] Step 3 Pipeline segmentation module:

[0056] The pipeline segmentation module performs two convolution operations on the feature map output by the backbone network, and finally obtains a single-channel output map. The size of each value in the output map represents the probability value of whether the point is a pipeline. The threshold is 0.5, and the value greater than 0.5 is considered to be the area where the pipeline is located, and finally the pipeline segmentation map is obtained. The feature value of the pipeline area in the segmentation map is set to 1, and the non-pipeline area is set to 0. After the setting is completed, it is multiplied with the 64-channel feature map output by the backbone network to obtain the pipeline area feature map.

[0057] Step 4: Drip status recognition module:

[0058] The pipeline area feature map obtained in step 3 is sent to the detection branch. After 4 convolutions and downsampling, the detection branch obtains a 64*64*512d feature map, which is sent to the detection head module. Figure 5 As shown in the figure, the detection head performs a convolution operation on the input feature map. The size of the output feature map is 64*64*54. The 54 channels indicate that each pixel outputs 9 groups of prediction values. Each group represents a possible detection result. Each group has 6 outputs. The first value represents the foreground score, followed by the next 4 values ​​representing the target coordinate prediction values, and the last value represents the drip prediction score value. If the foreground probability exceeds the set threshold and the drip classification score also exceeds the threshold, it means that the prediction value is a drip detection target, and the coordinate value indicates the location of the drip. After obtaining all the drip target prediction values, a maximum value suppression operation is required. Maximum value suppression is to remove redundant and repeated predictions. After completing these operations, if a drip target is detected, it means that there is a drip phenomenon in the pipeline of this image.

[0059] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, the above is only a preferred embodiment of the present invention. Since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with the technical field is within the technical scope disclosed by the present invention. For ordinary technicians in the technical field, changes or replacements that can be easily thought of should be covered within the protection scope of the present invention without departing from the principle of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A pipeline leakage status recognition system based on deep learning, characterized by: It includes a shooting module, an image preprocessing module, a basic feature extraction module, a pipeline segmentation module and a dripping state recognition module; the shooting module shoots pipeline images; the image preprocessing module preprocesses the captured pipeline images; the basic feature extraction module is a U-shaped network, which is divided into two parts, the encoding module and the decoding module. The encoding module is responsible for extracting the features of the preprocessed image, gradually reducing the input image and extracting high-level features; The decoding part gradually enlarges the features to the size of the original image through upsampling and skip links; the pipeline segmentation module segments the pipeline through the segmentation branch of the network, obtains the mask of the area where the pipeline is located, multiplies the obtained mask with the feature map obtained by the basic feature extraction module, obtains the pipeline area features, and shields the related background area features; the drip state recognition module processes the feature map, outputs the inference result, and then performs a maximum suppression operation on the inference result to finally obtain the recognition result of whether the pipeline is dripping.

2. A pipeline leakage state recognition method based on deep learning, characterized in that The following steps are involved: Step 1) The shooting module of the inspection robot shoots the pipeline image and sends it to the image preprocessing module to preprocess the image and obtain the input data of the image; Step 2) Input the input data preprocessed in step 1) into the backbone network of the basic feature extraction module. The backbone network is a U-shaped network, including an encoding module and a decoding module. The encoding module is responsible for extracting features, gradually reducing the input image and extracting high-level features; the decoding part gradually enlarges the features to the size of the original image through upsampling and skip links to obtain a feature map; Step 3) The pipeline segmentation module segments the pipeline through the segmentation branch of the network, obtains the mask of the area where the pipeline is located, and then multiplies the obtained mask with the feature map obtained in step 2) to obtain the pipeline area feature map while shielding the relevant background area features; Step 4) The pipeline area feature map obtained in step 3) is processed by the dripping state recognition module. After passing through several layers of convolution and detection heads, the inference result is output, and then the maximum value suppression operation is performed on the inference result to finally obtain the pipeline dripping recognition result.

3. The method for identifying pipeline leakage status based on deep learning according to claim 2, characterized in that: In the preprocessing process of step 1), the image is first resized and padded to ensure that the input data meets the neural network input requirements, and then the adjusted image is normalized, the mean is subtracted and divided by the variance to obtain the input data of the image.

4. A pipeline leakage state recognition method based on deep learning according to claim 2 or 3, characterized in that: The pretreatment process of step 1) is specifically as follows: 1.1) Adjust the image size, adjust the long side of the image to 512 pixels, and adjust the short side to the corresponding size proportionally; 1.2) Fill the short board to ensure that the size of the image sent to the network is 512*512; 1.3) Divide each pixel value of the three channels by 255 to make the pixel value range between 0 and 1, then subtract the mean 0.485, 0.456, and 0.406 of the R channel, G channel, and B channel respectively, and then divide them by the standard deviation 0.229, 0.224, and 0.225 to obtain the final input of the network.

5. The method for identifying pipeline leakage status based on deep learning according to claim 2, characterized in that: The backbone network of the basic feature extraction module in step 2) is a U-shaped network, which includes an encoding part and a decoding part. The specific extraction process is as follows: Step 2.1) The encoding part includes 3 stages. In each stage, the size of the feature map gradually decreases, but the number of channels gradually increases. The resolution changes from 512*512 to 64*64, and the number of channels finally becomes 512. Step 2.2) The decoding part includes 3 stages. In each stage, the size of the feature map gradually decreases, but the number of channels gradually decreases. The feature map resolution becomes 512*512 and the number of channels becomes 192. After the convolution operation, the backbone network finally outputs a feature map of 512*512*64.

6. The method for identifying pipeline leakage status based on deep learning according to claim 5, characterized in that: Step 2.1) The specific process of encoding is as follows: Step 2.11) Perform convolution operation on the input 3-channel image of size 512*512 to change the number of channels to 64; Step 2.12) The special diagnosis is further extracted through the Bottleneck module. Using the residual structure, the Bottleneck module first uses 1x1 convolution for dimensionality increase, then uses depthwise separable convolution for convolution operation, then uses 1x1 convolution for dimensionality reduction, and finally performs a short-circuit operation; the activation function of Bottleneck is FRelu, and its expression is as follows: y=FRelu(x)=max(x,f(x)) Where f(x) is a convolution based on a local neighborhood, expressed as follows: f(x) = W*x+b; Step 2.13) performs downsampling operation. Based on the Bottleneck module in step 2.12), the step size is set to 2 during convolution to halve the size of the feature map to better extract global features.

7. The method for identifying pipeline leakage status based on deep learning according to claim 5, characterized in that: Step 2.2) The specific process of decoding is as follows: Step 2.21) Perform bilinear interpolation upsampling on the 64*64 feature map obtained from the encoding part to increase the resolution to 128*128; Step 2.22) performs concat operation with the feature map of the same resolution corresponding to the encoded part, and the number of channels of the feature map becomes 768. Then, the number of channels becomes 256 after the convolution operation. Step 2.23) Further extract features through the Bottleneck module to make the feature map resolution become 512*512 and the number of channels become 192. After the convolution operation, the backbone network finally outputs a feature map of 512*512*64.

8. The method for identifying pipeline leakage status based on deep learning according to claim 2, characterized in that: Step 3) The pipeline segmentation module segments the pipeline through the segmentation branch of the network. The specific process is as follows: the pipeline segmentation module performs two convolution operations on the feature map output by the backbone network, and finally obtains a single-channel output map. The size of each value in the output map represents the probability value of whether the point is a pipeline. The threshold is 0.5, and values ​​greater than 0.5 are considered to be the area where the pipeline is located, and finally a segmentation map of the pipeline is obtained; the feature value of the pipeline area in the segmentation map is set to 1, and the non-pipeline area is set to 0; after the setting is completed, it is multiplied with the 64-channel feature map output by the backbone network to obtain a pipeline area feature map.

9. A pipeline leakage state recognition method based on deep learning according to claim 2 or 8, characterized in that: Step 4) The specific process of processing the pipeline area feature map obtained in step 3) by the dripping state recognition module is as follows: Step 4.1) Send the pipeline area feature map obtained in step 3) to the detection branch. After 4 convolutions and downsampling, the detection branch obtains a 64*64*512d feature map, and sends the feature map to the detection head module; Step 4.2) The detection head performs a convolution operation on the input feature map. The size of the output feature map is 64*64*54. The 54 channels indicate that each pixel outputs 9 groups of prediction values. Each group represents a possible detection result. Each group has 6 outputs. The first value represents the foreground score, followed by the next 4 values ​​representing the target coordinate prediction values, and the last value represents the drip prediction score value. Step 4.3) If the foreground probability exceeds the set threshold and the drip classification score also exceeds the threshold, it means that the predicted value is a drip detection target, and the coordinate value indicates the location of the drip; Step 4.4) After obtaining all the drip target prediction values, perform a maximum value suppression operation to remove redundant and repeated predictions; Step 4.5) If a leaking target is detected, it means that there is a leaking phenomenon in the pipe in this picture.

Images