Method, device and storage medium for identifying defects in a pipe

Abstract

The application provides a pipeline defect identification method and device and a storage medium, and relates to the field of computer vision and image processing. The method comprises the following steps: acquiring a color image and a depth image corresponding to a pipeline, performing noise reduction processing on the depth image to obtain a target depth image; performing mask processing on the target depth image to obtain a mask plate, fusing the mask plate with the color image to obtain a fused image; acquiring a fusion effect and a fusion factor according to the fused image, and optimizing the fusion factor according to the fusion effect to obtain a target fusion factor and a target fused image corresponding to the fused image; acquiring a convolutional neural network model, improving the convolutional neural network model to obtain a target neural network model, and inputting the target fused image into the target neural network model to obtain an online defect identification result corresponding to the pipeline. The method provided in the application improves the accuracy of unmanned aerial vehicle vision detection of pipeline defects.

Description

Methods, equipment, and storage media for identifying pipeline defects Technical Field

[0001] This application relates to the field of computer vision and image processing, and in particular to a method, device and storage medium for identifying pipeline defects. Background Technology

[0002] With the development of overhead pipeline oil and gas transportation, pipeline operation problems have increased due to factors such as falling rocks and corrosion, leading to various types of defects. Unmanned aerial vehicle (UAV) visual inspection, as one of the important methods for detecting defects in overhead pipelines, has great application potential.

[0003] Due to the complex and variable environment surrounding overhead pipelines, pipeline defects can exhibit diverse forms and manifestations, resulting in poor accuracy and low precision in UAV visual inspection. Furthermore, traditional background removal methods for images obtained through UAV visual inspection suffer from low signal-to-noise ratios and long processing times. Summary of the Invention

[0004] This application provides a method, device, and storage medium for identifying pipeline defects, in order to solve the technical problems of poor accuracy and low precision in visual inspection of pipeline defects by UAVs.

[0005] In a first aspect, this application provides a method for identifying pipeline defects, the method comprising:

[0006] The first camera acquires a color image of the pipe, the second camera acquires a depth image of the pipe, and the depth image is denoised to obtain the target depth image.

[0007] The target depth image is masked to obtain a mask corresponding to the target depth image. The mask is then fused with the color image to obtain a fused image.

[0008] The fusion effect and fusion factor are obtained from the fused image, and the fusion factor is optimized based on the fusion effect to obtain the target fusion factor and target fused image corresponding to the fused image.

[0009] Obtain a convolutional neural network model, improve the convolutional neural network model to obtain a target neural network model, input the target fused image into the target neural network model, and obtain the online defect identification result corresponding to the pipeline.

[0010] Optionally, the depth image is denoised to obtain the target depth image, including:

[0011] An adaptive median filter is obtained, and the adaptive median filter is improved to obtain a target adaptive median filter. The target adaptive median filter is then used to filter out target pixels in the depth image to obtain the first result.

[0012] Based on the first result, an opening operation is used to denoise the first result to obtain the second result, and a closing operation is used to denoise the second result to obtain the target depth image.

[0013] Optionally, a masking process is performed on the target depth image to obtain a mask corresponding to the target depth image. The mask is then fused with a color image to obtain a fused image, including:

[0014] Obtain the pipe portion and background portion from the target depth image, filter the background portion, and create a mask corresponding to the target depth image based on the pipe portion and background portion.

[0015] Map the corresponding first pixel in the mask to the three-dimensional space to obtain the corresponding first spatial coordinates, and project the first spatial coordinates onto the two-dimensional plane corresponding to the color image to obtain the aligned pixels;

[0016] Based on the aligned pixels, the mask and the color image are fused together, and the background of the color image is removed based on the mask to obtain the fused image.

[0017] Optionally, the fusion effect and fusion factor are obtained from the fused image, and the fusion factor is optimized based on the fusion effect to obtain the target fusion factor and the target fused image corresponding to the fused image, including:

[0018] Obtain an image segmentation model, use the image segmentation model to segment the fused image, obtain the segmentation result, and obtain the fusion effect corresponding to the fused image based on the segmentation result;

[0019] Based on the fusion effect, the fused image is divided into effective images and invalid images, and the proportion of effective images and invalid images and the fusion factor are obtained based on the effective images and invalid images.

[0020] An image segmentation model is used to optimize the fusion factor, resulting in the target fusion factor and the target fusion image corresponding to the fused image.

[0021] Optionally, an image segmentation model is used to optimize the fusion factor to obtain the target fusion factor and the target fusion image corresponding to the fused image, including:

[0022] Based on the current fusion factor, the segmentation results are fused to obtain the first fused image;

[0023] The first fused image is compared with the fused image to determine whether the proportion meets the preset conditions; if so, the target fusion factor and the target fused image are obtained.

[0024] Optionally, a convolutional neural network model is obtained, and the convolutional neural network model is improved to obtain a target neural network model, including:

[0025] Obtain the features of the target fused image, and based on the features of the target fused image, obtain the classification depth convolution operator;

[0026] Obtain the convolutional neural network model, and replace the preset convolutional part in the convolutional neural network model with the classification depthwise convolution operator to obtain the target neural network model.

[0027] Optionally, the target fused image is input into the target neural network model to obtain the online defect identification result corresponding to the pipeline, including:

[0028] The target neural network model is trained to obtain the first target neural network model;

[0029] The target fusion image is input into the first target neural network model to obtain the online defect identification result corresponding to the pipeline.

[0030] Secondly, this application provides a device for identifying pipeline defects, comprising:

[0031] The acquisition module is used to acquire a color image corresponding to the pipe based on the first camera, acquire a depth image corresponding to the pipe based on the second camera, and perform noise reduction processing on the depth image to obtain a target depth image.

[0032] The first processing module is used to perform masking processing on the target depth image to obtain a mask corresponding to the target depth image, and to fuse the mask with the color image to obtain a fused image.

[0033] The second processing module is used to obtain the fusion effect and fusion factor based on the fused image, and to optimize the fusion factor based on the fusion effect to obtain the target fusion factor and the target fused image corresponding to the fused image.

[0034] The input module is used to acquire a convolutional neural network model, improve the convolutional neural network model to obtain a target neural network model, and input the target fused image into the target neural network model to obtain the online defect identification result corresponding to the pipeline. In one possible implementation, the ... module is also used for:

[0035] Optionally, the acquisition module is also used for:

[0036] An adaptive median filter is obtained, and the adaptive median filter is improved to obtain a target adaptive median filter. The target adaptive median filter is then used to filter out target pixels in the depth image to obtain the first result.

[0037] Based on the first result, an opening operation is used to denoise the first result to obtain the second result, and a closing operation is used to denoise the second result to obtain the target depth image.

[0038] Optionally, the first processing module is also used for:

[0039] Obtain the pipe portion and background portion from the target depth image, filter the background portion, and create a mask corresponding to the target depth image based on the pipe portion and background portion.

[0040] Map the corresponding first pixel in the mask to the three-dimensional space to obtain the corresponding first spatial coordinates, and project the first spatial coordinates onto the two-dimensional plane corresponding to the color image to obtain the aligned pixels;

[0041] Based on the aligned pixels, the mask and the color image are fused together, and the background of the color image is removed based on the mask to obtain the fused image.

[0042] Optionally, the second processing module is also used for:

[0043] Obtain an image segmentation model, use the image segmentation model to segment the fused image, obtain the segmentation result, and obtain the fusion effect corresponding to the fused image based on the segmentation result;

[0044] Based on the fusion effect, the fused image is divided into effective images and invalid images, and the proportion of effective images and invalid images and the fusion factor are obtained based on the effective images and invalid images.

[0045] An image segmentation model is used to optimize the fusion factor, resulting in the target fusion factor and the target fusion image corresponding to the fused image.

[0046] Optionally, the second processing module is also used for:

[0047] Based on the current fusion factor, the segmentation results are fused to obtain the first fused image;

[0048] The first fused image is compared with the fused image to determine whether the proportion meets the preset conditions; if so, the target fusion factor and the target fused image are obtained.

[0049] Optionally, the input module is also used for:

[0050] Obtain the features of the target fused image, and based on the features of the target fused image, obtain the classification depth convolution operator;

[0051] Obtain the convolutional neural network model, and replace the preset convolutional part in the convolutional neural network model with the classification depthwise convolution operator to obtain the target neural network model.

[0052] Optionally, the input module is also used for:

[0053] The target neural network model is trained to obtain the first target neural network model;

[0054] The target fusion image is input into the first target neural network model to obtain the online defect identification result corresponding to the pipeline.

[0055] Thirdly, this application provides a device for identifying pipeline defects, comprising:

[0056] Processor and memory;

[0057] The memory stores the instructions that the computer executes;

[0058] The processor executes computer execution instructions stored in memory, causing the pipeline defect identification device to perform any of the pipeline defect identification methods in the first aspect.

[0059] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a pipeline defect identification method as described in any of the first aspects.

[0060] The pipeline defect identification method, device, and storage medium provided in this application acquire color and depth images. Combining color and depth images provides a more comprehensive description of the scene, facilitating more accurate pipeline identification and analysis. Noise reduction processing of the depth image improves the clarity and accuracy of depth information, thereby reducing noise interference in image analysis. Masking the target depth image and fusing it with the color image yields a fused image, enabling real-time background removal, improving the signal-to-noise ratio, and enhancing pipeline identification and analysis capabilities. Optimization based on the fusion effect and fusion factor yields a target fusion factor and a target fused image. This target fused image is then input into an improved convolutional neural network model to achieve online pipeline defect identification. By combining deep learning technology with image information, pipeline defect identification can be automated and efficient, improving detection accuracy and efficiency. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 is a schematic flowchart of a pipeline defect identification method provided in an embodiment of this application;

[0063] Figure 2 is a schematic flowchart of a pipeline defect identification method provided in an embodiment of this application;

[0064] Figure 3 is a schematic diagram illustrating the principle of noise reduction processing and mask creation in a pipeline defect identification method provided in an embodiment of this application;

[0065] Figure 4 is a schematic flowchart of a pipeline defect identification method provided in an embodiment of this application;

[0066] Figure 5 is a schematic flowchart of a pipeline defect identification method provided in an embodiment of this application;

[0067] Figure 6 is a schematic diagram of the classification depth convolution operator in a pipeline defect identification method provided in an embodiment of this application;

[0068] Figure 7 is a schematic diagram of the structure of a pipeline defect identification device provided in an embodiment of this application;

[0069] Figure 8 is a schematic diagram of the hardware structure of a pipeline defect identification device provided in an embodiment of this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] Oil and gas pipelines in areas with complex terrain, such as mountains, hills, deserts, and shallow seas, require overhead pipeline construction. Inland overhead pipelines are subject to stress and deformation from rockfalls, while cross-sea overhead pipelines face high temperatures, high humidity, high salt spray concentrations, and strong ultraviolet radiation corrosion, leading to pipeline defects and increasing transportation challenges.

[0073] Unmanned aerial vehicle (UAV) pipeline inspection technology is a non-destructive testing technology for pipelines, widely used in industries such as manufacturing, construction, and oil and gas. Compared to traditional manual inspection, UAVs offer greater flexibility and adaptability, capable of handling complex and changing environments and operating conditions. The principles of UAV pipeline inspection technology primarily involve remote sensing and image processing technologies used by the UAV.

[0074] Machine vision inspection, as a non-destructive testing method for surface defects, faces challenges in pipeline defect detection. Complex background environments, lighting, and other factors often result in low signal-to-noise ratios, making it difficult to detect weak signals or effectively distinguish them from noise. Furthermore, the diverse range of objects being inspected, the numerous types and forms of surface defects, and the complex backgrounds present challenges. The mechanisms underlying various defect types and the relationships between their external manifestations remain unclear, leading to insufficient defect descriptions, low effectiveness in feature extraction, and difficulties in defect segmentation. Additionally, finding "standard" images for reference further complicates defect detection and classification. Online inspection, in particular, is characterized by massive data volumes, abundant redundant information, and high feature space dimensionality. Considering the diversity of objects and problems faced by true machine vision, algorithms for extracting limited defect information from massive datasets are insufficient, resulting in low real-time performance.

[0075] This application provides a method, device, and storage medium for identifying pipeline defects. A first camera acquires a color image, and a second camera acquires a depth image. Noise reduction processing is applied to the depth image to effectively obtain visual information about the pipeline and reduce noise in the depth image. By masking the target depth image and fusing it with the color image, a fused image is obtained. This helps combine color and depth information, improving the ability to identify and analyze pipelines, and the fusion processing of the mask and color image is real-time. The fusion effect and fusion factor are obtained from the fused image. The fusion factor is optimized based on the effect to obtain the target fusion factor and target fused image, improving the quality of image fusion and further enhancing the accuracy of pipeline detection and identification. The target fused image is input into an improved convolutional neural network model to obtain online pipeline defect identification results. By utilizing deep learning technology combined with image information for automated defect identification, the efficiency and accuracy of pipeline detection are improved.

[0076] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0077] Figure 1 is a schematic flowchart of a pipeline defect identification method provided in an embodiment of this application.

[0078] As shown in Figure 1, this embodiment provides a method for identifying pipeline defects, including:

[0079] S101. Obtain the color image corresponding to the pipe from the first camera, and obtain the depth image corresponding to the pipe from the second camera. Perform noise reduction processing on the depth image to obtain the target depth image.

[0080] In this embodiment, the first camera refers to a color camera, and the second camera refers to a depth camera. The color image of the pipe captured by the first camera contains information about the pipe defects, the pipe itself, and the background, where the background is redundant. At the same viewpoint, the depth image of the pipe captured by the second camera contains information about the pipe defects, the pipe itself, the background, and noise. To eliminate the influence of noise in the depth image, denoising processing is required to obtain a denoised depth image.

[0081] It should be noted that both the first and second cameras were mounted on drones for pipe photography.

[0082] S102. Perform masking on the target depth image to obtain a mask corresponding to the target depth image. Then, fuse the mask with the color image to obtain a fused image.

[0083] In this embodiment, the contrast between pipes and the background is relatively high in depth images, while the contrast between defects and pipes is low. In color images, the interlacing pipes, differences in pipe colors, and interference from complex backgrounds make it difficult to accurately identify individual target pipes and defects from complex information. While depth images can accurately identify pipes, noise prevents accurate defect identification.

[0084] To accurately identify target information, the target depth image needs to be masked. This involves setting a threshold to segment the depth values ​​into target and background regions, generating a binary mask image. This mask image serves as a template, and the color image and the depth image mask are then fused. Pixel-level fusion algorithms, such as weighted fusion, can be used. The fusion weights between the color and depth images are adjusted based on the mask values ​​of the target depth image to achieve the desired fusion effect. The fused image is then output as the final fused image, preserving depth information while enhancing the three-dimensionality of the color image.

[0085] S103. Obtain the fusion effect and fusion factor based on the fused image, and optimize the fusion factor based on the fusion effect to obtain the target fusion factor and target fused image corresponding to the fused image.

[0086] In this embodiment, the fusion effect typically refers to the visual effect of the fused image, i.e., the overall effect after different images are merged. The fusion factor refers to the parameters or weights used in the fusion process, affecting the generation of the fused image. By observing and analyzing the fused image, subjective and objective evaluations of the fusion effect can be obtained. Simultaneously, the fusion factor is crucial for controlling the fusion effect, and it is necessary to optimize the fusion result by adjusting these factors. Through optimization, adjusted target fusion factors are obtained; these factors will better control the fusion process to achieve a more ideal fusion effect. Based on the optimized fusion factors, the image fusion process can be repeated to generate the final target fused image.

[0087] S104. Obtain the convolutional neural network model, improve the convolutional neural network model to obtain the target neural network model, input the target fused image into the target neural network model, and obtain the online defect identification result corresponding to the pipeline.

[0088] In this embodiment, the convolutional neural network models used for image classification include VGG16, InceptionV3, ResNet50, and MobileNet. For this invention, a lightweight neural network model more suitable for the training samples is needed to achieve high speed while maintaining accuracy, thus reducing model training time. Among these, the MobileNet series models are lightweight neural network models. By using depthwise convolution instead of traditional convolution, the MobileNet series models reduce the number of parameters in the model, resulting in lower time complexity and significantly reducing training time. Therefore, the MobileNet series models are improved to obtain the Pipe-MobileNet model, which serves as the target neural network model. The target fused image is input into the target neural network model for processing to obtain the corresponding online defect recognition results.

[0089] Convolutional Neural Networks (CNNs) are a type of deep learning model that extracts and represents features from input data through convolution operations. A CNN consists of multiple convolutional and pooling layers, which perform convolution and pooling operations on the input data, respectively.

[0090] This application provides a method for identifying pipeline defects. The method acquires a color image and a depth image corresponding to the pipeline. The depth image is denoised to obtain a target depth image, improving the signal-to-noise ratio. The target depth image is then masked to obtain a mask, which is fused with the color image to obtain a fused image, representing the effect of the mask and color image fusion. This fusion process is real-time. The fusion effect and fusion factor are obtained from the fused image, and the fusion factor is optimized based on the fusion effect to obtain the target fusion factor and target fused image, improving the quality of image fusion and further enhancing the accuracy of pipeline detection and identification. A convolutional neural network model is acquired and improved to obtain a target neural network model. The target fused image is input into the target neural network model to obtain the online defect identification result corresponding to the pipeline. By utilizing deep learning technology combined with image information for automated defect identification, the efficiency and accuracy of pipeline detection are improved.

[0091] Figure 2 is a schematic flowchart of a pipeline defect identification method provided in an embodiment of this application. This embodiment is based on the embodiment in Figure 1 and provides a detailed description of the pipeline defect identification method.

[0092] As shown in Figure 2, this embodiment provides a method for identifying pipeline defects, including:

[0093] S201. Obtain the color image corresponding to the pipe using the first camera, and obtain the depth image corresponding to the pipe using the second camera.

[0094] Step S201 is similar to step S101 above, and will not be repeated here.

[0095] S202. Obtain the adaptive median filter, improve the adaptive median filter to obtain the target adaptive median filter, and use the target adaptive median filter to filter out the target pixels in the depth image to obtain the first result.

[0096] In this embodiment, the adaptive median filter is a filter that dynamically adjusts based on pixel values ​​to remove noise and abnormal pixels (noise points) from an image. The adaptive median filter is improved by adjusting parameters, changing the filtering strategy, and increasing threshold settings to enhance its filtering effect.

[0097] Specifically, since the depth image changes with the distance between the second camera and the pipe, the adaptive median filter cannot be set to a constant threshold. The adaptive median filter is improved by using the difference between the value of a point with a large color difference in the depth image and its neighboring pixels as the threshold, resulting in a target adaptive median filter. Therefore, to eliminate the influence of noise points in the depth image, the target adaptive median filter replaces the value of a point with a large color difference in the depth image with the median of its neighborhood, eliminating isolated noise points and obtaining the first result. Here, the isolated noise point is the target pixel.

[0098] It should be noted that the mathematical expression of the target adaptive median filter is as follows:

[0099]

[0100]

[0101] Where, p i,j p represents the pixel value at (i,j) in the depth image. L p represents the pixel difference threshold; m Represents the image (p) i-N,j-N )~(p i+N,j+N The median value of all pixels within the range; N represents the target adaptive median filter size. If p m If the value is within the threshold range, then replace p with that value. i,j The process then slides a window to process the next pixel; otherwise, the target adaptive median filter size is increased before continuing. Since the image is a three-channel image, the pixel difference is calculated by summing the values ​​for each channel separately. To reduce computational complexity, only the absolute value of the diagonal element difference for each channel is calculated. A three-channel image typically refers to an RGB (Red, Green, Blue) image, also known as a color image. In computer image processing, color images usually consist of three channels, each corresponding to the three primary colors: red, green, and blue. Various colors can be represented by combining and superimposing different channels.

[0102] S203. Based on the first result, the first result is denoised using an opening operation to obtain the second result, and the second result is denoised using a closing operation to obtain the target depth image.

[0103] In this embodiment, the first result is processed by an opening operation, which eliminates small and irregular noise points in the image while preserving the main contours. The opening operation is followed by erosion and dilation operations to eliminate fine protrusions in the first result, yielding the second result. The second result is then processed by a closing operation, which fills in small holes and defects in the image while smoothing the contours. The closing operation is followed by dilation and erosion operations to obtain the target depth image, i.e., the image after the closing operation. This image is smoother and clearer, and noise and irregular points inside the pipes in the image have been removed.

[0104] It should be noted that the mathematical expressions for image opening and closing operations are as follows: Let A be the original image, and B and C be structuring element images, then:

[0105]

[0106] S204. Obtain the pipe portion and background portion in the target depth image, filter the background portion, and create a mask corresponding to the target depth image based on the pipe portion and background portion.

[0107] In this embodiment, image segmentation techniques, such as thresholding, edge detection, and region growing, can be used to segment the pipe portion and background portion in the target depth image, thereby obtaining binary images of the pipe portion and background portion. The method for obtaining the pipe portion and background portion in the target depth image is not limited here. Based on the difference between the background portion and the pipe portion, the background portion is filtered and removed, retaining the main target in the target depth image. Using the binary images of the pipe portion and background portion, the separated background in the target depth image is used as a mask in the color image. By fusing and overlaying to remove the background in the color image, a mask is created that retains only the pipe portion and defect information.

[0108] A binary image is an image type in which each pixel has only two possible values, typically 0 and 1. In a binary image, a pixel value of 0 represents black (or background), and a pixel value of 1 represents white (or foreground); therefore, it is often called a black and white image or a 1-bit image.

[0109] S205. Map the corresponding first pixel in the mask to the three-dimensional space to obtain the corresponding first spatial coordinates, and project the first spatial coordinates onto the two-dimensional plane corresponding to the color image to obtain the aligned pixels.

[0110] In this embodiment, since the color image and depth image are acquired by different cameras, and different cameras correspond to different sensors, the color image and depth image correspond to different coordinate systems. Therefore, pixel alignment is required. This pixel alignment process is implemented using Python programming.

[0111] S206. Based on the aligned pixels, the mask and the color image are fused together, and the background of the color image is removed based on the mask to obtain the fused image.

[0112] In this embodiment, the correspondence between the mask and the color image is ensured to be accurate based on the aligned pixels. At the aligned pixels, the pixel values ​​in the mask are fused with the pixel values ​​in the color image. Using the mask, background removal is performed on the color image during the fusion process. Depending on the pixel values ​​in the mask (e.g., 0 represents background, 1 represents foreground), background pixels can be set to transparent values ​​or replaced according to a specific color.

[0113] It should be noted that the fusion formula for blending the mask and the color image is as follows:

[0114]

[0115] Where α, β, and c are fusion factors, g(x) is the fused image, f1(x) is the color image, and f2(x) is the depth image.

[0116] For a single color image, the fused image g(x) is (r, g, b), and the three-channel histograms are h, g, b, and ... R (r), h G (g), h B (b)

[0117] Calculate the cumulative distribution function (CDF) for each channel. R (r), cdf G (g), cdf B (b):

[0118]

[0119] Equalization function:

[0120]

[0121] Where M is the width of the image and N is the height of the image.

[0122] The brightness of this image can be represented as L:

[0123]

[0124] The average brightness of N images is:

[0125]

[0126] but:

[0127]

[0128] The adjusted image g'(x) is (f R '(r),f G '(g),f B '(b)).

[0129] The color image after removing the background is:

[0130]

[0131] S207. Obtain the fusion effect and fusion factor based on the fused image, and optimize the fusion factor based on the fusion effect to obtain the target fusion factor and target fused image corresponding to the fused image.

[0132] Step S207 is similar to step S103 above, and will not be described again here.

[0133] S208. Obtain the convolutional neural network model, improve the convolutional neural network model to obtain the target neural network model, input the target fused image into the target neural network model, and obtain the online defect identification result corresponding to the pipeline.

[0134] Step S208 is similar to step S104 above, and will not be described again here.

[0135] This application provides a method for identifying pipeline defects. The method acquires a color image and a depth image, and uses an adaptive median filter to filter target pixels in the depth image. The improved filter can process images in various complex environments, is more versatile, and can better remove noise and improve image quality. Noise reduction is performed on the processed result through opening and closing operations to further optimize the quality of the depth image. Based on the pipeline and background portions in the depth image, a mask corresponding to the depth image is created. The mask maps pixels in the depth image to three-dimensional space and a two-dimensional plane, achieving pixel alignment. The aligned pixels are fused with the color image, and the background is removed to obtain a fused image. Combining depth and color information improves the overall image quality. The fusion effect and fusion factor are obtained from the fused image and optimized to obtain the target fusion factor and target fused image, thereby achieving real-time background removal, significantly improving the signal-to-noise ratio and recognition accuracy. Online defect recognition is performed using a convolutional neural network model. The fused image is input into the model to obtain the pipeline defect recognition result.

[0136] Figure 3 is a schematic diagram illustrating the principle of noise reduction and mask creation in a pipe defect identification method provided in this application embodiment. This embodiment is based on the embodiment in Figure 2, and provides a detailed explanation of steps S202-S204. As shown in Figure 3, a depth image is first input. This depth image is then processed by a target adaptive median filter to replace the value of a point with a large color difference in the image with the median of its neighborhood, eliminating isolated noise points and obtaining a first result. Then, an opening operation is used to perform an erosion followed by a dilation operation on the first result, thereby eliminating fine protrusions in the image and obtaining a second result. Finally, a closing operation is used to perform a dilation followed by an erosion operation on the second result, thereby eliminating noise inside the pipe in the image and obtaining a target depth image. After the above process, noise and defect features are removed simultaneously, achieving smoothing and eliminating noise. In the processed target depth image, the pipe portion and the background portion have high contrast. The background portion is filtered, and a mask is created. Here, squares represent pixels.

[0137] Figure 4 is a flowchart illustrating a pipeline defect identification method according to an embodiment of this application. This embodiment, based on the embodiment shown in Figure 2, details the acquisition of fusion effect and fusion factor from the fused image, and the optimization of the fusion factor based on the fusion effect to obtain the target fusion factor and target fused image corresponding to the fused image. As shown in Figure 4, the pipeline defect identification method provided by this embodiment includes:

[0138] S401. Obtain the image segmentation model, use the image segmentation model to segment the fused image, obtain the segmentation result, and obtain the fusion effect corresponding to the fused image based on the segmentation result.

[0139] In this embodiment, the image segmentation model used is the Segment Anything Model (SAM). This model has zero-sample transfer capability and can generate masks for any object in any image or video, even objects and image types that did not appear during training. The SAM model is used to segment the fused image to obtain the segmentation results. Based on the segmentation results, pixel-level analysis metrics are used to analyze the fusion effect and obtain the accuracy and quality of the segmentation results.

[0140] It should be noted that in pipeline defect identification with complex backgrounds, SAM is more suitable as an upstream module, rather than using SAM to complete image segmentation and defect identification entirely.

[0141] S402. Based on the fusion effect, the fused image is divided into valid images and invalid images, and the proportion of valid images and invalid images and the fusion factor are obtained based on the valid images and invalid images.

[0142] In this embodiment, in the segmentation result of the target fused image, the pipe portion is noise-free, and the RGB values ​​of the pixels in the background portion are all 0. Such a standard image background portion does not possess sub-region contour features, and the pipe portion, apart from defects, does not possess other contour features. Fusion images exhibiting this segmentation result are considered valid images (Apic), and fusion images with other segmentation results are considered invalid images (Fpic). To obtain the target fused image, a SAM model is introduced to analyze the fusion effect. Based on the feedback from the SAM model output, the proportion of valid to invalid images (Acc) can be obtained:

[0143]

[0144] The fusion factor can be a parameter used to adjust the weights or proportions of different modules during the fusion process in order to achieve the target fusion effect.

[0145] S403. Based on the current fusion factor, the segmentation results are fused to obtain the first fused image.

[0146] In this embodiment, based on the current fusion factor, the valid image and the invalid image are fused according to a preset ratio or weight, and the two images are merged together. After the fusion process, a first fused image is obtained.

[0147] S404. Compare the first fused image with the fused image to determine whether the proportion meets the preset conditions.

[0148] In this embodiment, the first fused image is compared with the second fused image to determine whether the proportion of valid and invalid images in the first fused image has increased. If it has increased, the fusion factor is updated according to the adjustment strategy of the fusion factor (e.g., gradually increasing or decreasing the factor), and the fusion is performed again. This process continues until the proportion meets the preset condition.

[0149] S405. If yes, then the target fusion factor and the target fusion image are obtained; if not, then step S401 is executed again.

[0150] In this embodiment, when the proportion meets the preset conditions, the target fusion factor and the target fusion image are obtained.

[0151] Specifically, in this embodiment, 95% is used as the preset condition for judgment. After 10 iterations, the proportion of each type of image is 100%, that is, all images after fusion are valid images.

[0152] This application provides a method for identifying pipeline defects. This method uses an image segmentation model to segment a fused image, obtaining segmentation results. This facilitates further understanding and analysis of the fused image. Based on the segmentation results, the fused image is divided into valid and invalid images. The proportion of valid and invalid images and the fusion factor are then calculated, enabling the determination of the proportion of valid information in the image and adjusting the fusion effect accordingly to obtain the target fusion factor and the target fused image. Through image segmentation, fusion effect judgment, and iterative optimization, real-time dynamic adjustment and optimization of the image fusion effect are achieved, resulting in higher-quality image results. This iterative optimization method helps improve the efficiency and accuracy of image processing, bringing richer technical means to the field of image processing.

[0153] Figure 5 is a flowchart illustrating a pipeline defect identification method according to an embodiment of this application. This embodiment is based on the embodiment shown in Figure 2, and details the acquisition of a convolutional neural network model, the improvement processing of the convolutional neural network model to obtain a target neural network model, and the input of the target fused image into the target neural network model to obtain the online defect identification result corresponding to the pipeline. As shown in Figure 5, the pipeline defect identification method provided by this embodiment includes:

[0154] S501. Obtain the features of the target fused image, and obtain the classification depth convolution operator based on the features of the target fused image.

[0155] In this embodiment, the target fusion image features obtained through feature extraction can be processed through operations such as dimensionality processing and normalization. Based on the dimensionality and characteristics of the target fusion image features, a suitable classification depthwise convolution operator is constructed.

[0156] Specifically, Figure 6 is a schematic diagram of the structure of a classification depth convolution operator in a pipeline defect identification method provided in this application embodiment. As shown in Figure 6, firstly, partial images of the background and pipeline parts in the target fusion image are selected respectively to obtain a first image, a second image, and a third image. The pixel values ​​of the first and second images are both (0, 0, 0). Feature extraction processing is performed on the first, second, and third images using convolutional kernels in a convolutional neural network. For the target fusion image, since the color information around each pixel needs to be considered, a 3×3 convolutional kernel is used. Each convolutional kernel operates on three channels during the convolution operation, finally obtaining a feature image where each channel contains feature information of the corresponding color channel. The classification depth convolution operator is used to determine the element values ​​of the four corner points of the convolution kernel in the input target fusion image. If all element values ​​are equal to a certain value, the convolution kernel is a type A convolution kernel, which only performs convolution operations on a single channel and assigns the value to other channels; otherwise, it is a type B convolution kernel, which operates in the form of conventional convolution. When a previous type A convolutional kernel exists, when determining if the next convolutional kernel is also a type A convolutional kernel, it checks if the element value is the same as an element in the previous type A convolutional kernel. If so, no convolution is needed, and the result of the previous type A convolutional kernel is assigned to the output. If not, the type A convolutional kernel operation is performed, but only on a single channel, and the result is assigned to the other channels before outputting the convolution result.

[0157] The equality of element values ​​at the four corner points of the convolution kernel is used to determine the type of convolution kernel, while the feature that the background pixel values ​​of the target fused image are all (0, 0, 0) is used to implement the specific application scenarios of this operator.

[0158] It's important to note that this classification depthwise convolution operator is applicable only when there are large connected regions consisting entirely of identical pixels. If the distribution is more dispersed, it means the total boundary length is longer, and the elements to be judged are not limited to the four corner points of the convolution kernel; otherwise, it will affect the final training results. If a certain prior feature is absent in the image, this classification depthwise convolution operator is not applicable.

[0159] S502. Obtain the convolutional neural network model, and replace the preset convolutional part in the convolutional neural network model using the classification depth convolution operator to obtain the target neural network model.

[0160] In this embodiment, the MobileNet series model is obtained and then improved by replacing the 3×3 convolutional part with a classification depthwise convolution operator to obtain the Pipe-MobileNet model, which is the target neural network model.

[0161] S503. Train the target neural network model to obtain the first target neural network model.

[0162] In this embodiment, in order to improve the generalization ability of the target neural network model to a certain extent, the target neural network model needs to be trained to obtain the first target neural network model.

[0163] Specifically, in this embodiment, 1000 images were captured under different lighting conditions (noon, dawn, dusk, and LED lights), different flight altitudes of the drone (300mm to 500mm above the pipe), different background environments of the pipe, and different camera angles. Sixty-three images containing multiple defects were removed. The remaining images were categorized and labeled into seven types: normal, deformed, holes, grooves, welds, cracks, and scratches. These images were then expanded using random rotation, distortion, salt-and-pepper noise, Gaussian noise, and adjustments to brightness and contrast, resulting in 10,000 images. Eight thousand images were used as the training set, and 2,000 images as the test set. After processing the training set images through steps S101 to S103, they were input into the target neural network model for offline training, thereby improving the generalization ability of the target neural network model and obtaining the first trained target neural network model.

[0164] S504. Input the target fusion image into the first target neural network model to obtain the online defect identification result corresponding to the pipeline.

[0165] In this embodiment, the target fused image is input into a first target neural network model for processing. The first target neural network model performs image recognition and defect detection based on the features and patterns it has learned. After processing, the first target neural network model outputs the online defect recognition result corresponding to the pipeline.

[0166] This application provides a method for identifying pipeline defects. This method improves the performance of the neural network on this task by designing a deep convolution operator suitable for the task based on the features of the target image. A customized target neural network model is obtained by replacing the preset convolution part with a classification deep convolution operator. Improving specific parts of the model can typically enhance the performance of the neural network on a specific task. The target neural network model is trained to obtain a first target neural network model, thus improving the generalization ability of the target neural network model. The target fused image is input into the trained target neural network model to obtain the online defect identification result corresponding to the pipeline. Through the improvement, training, and application of the neural network model, online defect identification of pipeline images is achieved, providing an automated and intelligent solution for pipeline maintenance and improving work efficiency, accuracy, and precision of detection.

[0167] Figure 7 is a schematic diagram of the structure of a pipeline defect identification device provided in an embodiment of this application. The device in this embodiment can be in the form of software and / or hardware. As shown in Figure 7, the pipeline defect identification device 700 provided in this embodiment includes: an acquisition module 701, a first processing module 702, a second processing module 703, and an input module 704.

[0168] The acquisition module 701 is used to acquire a color image corresponding to the pipe based on the first camera, acquire a depth image corresponding to the pipe based on the second camera, and perform noise reduction processing on the depth image to obtain a target depth image.

[0169] The first processing module 702 is used to perform masking processing on the target depth image to obtain a mask corresponding to the target depth image, and to fuse the mask with the color image to obtain a fused image.

[0170] The second processing module 703 is used to obtain the fusion effect and fusion factor based on the fused image, and to optimize the fusion factor based on the fusion effect to obtain the target fusion factor and the target fused image corresponding to the fused image.

[0171] Input module 704 is used to acquire a convolutional neural network model, improve the convolutional neural network model to obtain a target neural network model, and input the target fused image into the target neural network model to obtain the online defect identification result corresponding to the pipeline. In one possible implementation, the ... module is also used for:

[0172] In one possible implementation, the acquisition module 701 is further used for:

[0173] An adaptive median filter is obtained, and the adaptive median filter is improved to obtain a target adaptive median filter. The target adaptive median filter is then used to filter out target pixels in the depth image to obtain the first result.

[0174] Based on the first result, an opening operation is used to denoise the first result to obtain the second result, and a closing operation is used to denoise the second result to obtain the target depth image.

[0175] In one possible implementation, the first processing module 702 is further configured to:

[0176] Obtain the pipe portion and background portion from the target depth image, filter the background portion, and create a mask corresponding to the target depth image based on the pipe portion and background portion.

[0177] Map the corresponding first pixel in the mask to the three-dimensional space to obtain the corresponding first spatial coordinates, and project the first spatial coordinates onto the two-dimensional plane corresponding to the color image to obtain the aligned pixels;

[0178] Based on the aligned pixels, the mask and the color image are fused together, and the background of the color image is removed based on the mask to obtain the fused image.

[0179] In one possible implementation, the second processing module 703 is further used for:

[0180] Obtain an image segmentation model, use the image segmentation model to segment the fused image, obtain the segmentation result, and obtain the fusion effect corresponding to the fused image based on the segmentation result;

[0181] Based on the fusion effect, the fused image is divided into effective images and invalid images, and the proportion of effective images and invalid images and the fusion factor are obtained based on the effective images and invalid images.

[0182] An image segmentation model is used to optimize the fusion factor, resulting in the target fusion factor and the target fusion image corresponding to the fused image.

[0183] In one possible implementation, the second processing module 703 is further used for:

[0184] Based on the current fusion factor, the segmentation results are fused to obtain the first fused image;

[0185] The first fused image is compared with the fused image to determine whether the proportion meets the preset conditions; if so, the target fusion factor and the target fused image are obtained.

[0186] In one possible implementation, the input module 704 is also used for:

[0187] Obtain the features of the target fused image, and based on the features of the target fused image, obtain the classification depth convolution operator;

[0188] Obtain the convolutional neural network model, and replace the preset convolutional part in the convolutional neural network model with the classification depthwise convolution operator to obtain the target neural network model.

[0189] In one possible implementation, the input module 704 is also used for:

[0190] The target neural network model is trained to obtain the first target neural network model;

[0191] The target fusion image is input into the first target neural network model to obtain the online defect identification result corresponding to the pipeline.

[0192] Figure 8 is a hardware structure diagram of the pipeline defect identification device provided in an embodiment of this application. As shown in Figure 8, the pipeline defect identification device 800 includes:

[0193] Processor 801 and memory 802;

[0194] The memory stores the instructions that the computer executes;

[0195] The processor executes the computer execution instructions stored in memory 802, causing the pipeline defect identification device to perform the pipeline defect identification method described above.

[0196] It should be understood that the processor 801 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The memory 802 may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0197] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a method for identifying pipeline defects.

[0198] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, products, or apparatus.

[0199] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0200] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0201] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0202] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0203] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0204] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0205] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0206] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for identifying pipeline defects, characterized in that, include: A color image corresponding to the pipe is acquired using a first camera, and a depth image corresponding to the pipe is acquired using a second camera. The depth image is then denoised to obtain a target depth image. The pipe portion and the background portion in the target depth image are then acquired, the background portion is filtered, and a mask corresponding to the target depth image is created based on the pipe portion and the background portion. The first pixel in the mask is mapped to a three-dimensional space to obtain the corresponding first spatial coordinates. The first spatial coordinates are projected onto a two-dimensional plane corresponding to the color image to obtain aligned pixels. Based on the aligned pixels, the mask and the color image are fused, and the background of the color image is removed based on the mask to obtain a fused image. An image segmentation model is obtained, and the fused image is segmented using the image segmentation model to obtain a segmentation result. Based on the segmentation result, the fusion effect corresponding to the fused image is obtained. Based on the fusion effect, the fused image is divided into valid images and invalid images, and the proportion of valid images and invalid images and the fusion factor are obtained based on the valid images and invalid images. The fusion factor is optimized using the image segmentation model to obtain the target fusion factor and the target fused image corresponding to the fused image. A convolutional neural network model is obtained, and the convolutional neural network model is improved to obtain a target neural network model. The target fused image is input into the target neural network model to obtain the online defect identification result corresponding to the pipeline.

2. The method according to claim 1, characterized in that, The step of denoising the depth image to obtain a target depth image includes: acquiring an adaptive median filter; improving the adaptive median filter to obtain a target adaptive median filter; using the target adaptive median filter to filter out target pixels in the depth image to obtain a first result; and based on the first result, performing denoising on the first result using an opening operation to obtain a second result, and performing denoising on the second result using a closing operation to obtain the target depth image.

3. The method according to claim 1, characterized in that, The step of optimizing the fusion factor using the image segmentation model to obtain the target fusion factor and the target fusion image corresponding to the fusion image includes: fusing the segmentation results according to the current fusion factor to obtain a first fusion image; comparing the first fusion image with the fusion image to determine whether the proportion meets a preset condition; if so, obtaining the target fusion factor and the target fusion image.

4. The method according to claim 1, characterized in that, The step of obtaining a convolutional neural network model and improving the convolutional neural network model to obtain a target neural network model includes: obtaining features of the target fused image; obtaining a classification depthwise convolution operator based on the features of the target fused image; obtaining a convolutional neural network model; and replacing the preset convolutional part in the convolutional neural network model using the classification depthwise convolution operator to obtain the target neural network model.

5. The method according to claim 4, characterized in that, The step of inputting the target fused image into the target neural network model to obtain the online defect identification result corresponding to the pipeline includes: training the target neural network model to obtain a first target neural network model; and inputting the target fused image into the first target neural network model to obtain the online defect identification result corresponding to the pipeline.

6. A device for identifying pipeline defects, characterized in that, include: The acquisition module is used to acquire a color image corresponding to the pipe based on the first camera, acquire a depth image corresponding to the pipe based on the second camera, and perform noise reduction processing on the depth image to obtain a target depth image. The first processing module is used to perform masking processing on the target depth image to obtain a mask corresponding to the target depth image, and to fuse the mask with the color image to obtain a fused image; The second processing module is used to obtain the fusion effect and fusion factor based on the fused image, and to optimize the fusion factor based on the fusion effect to obtain the target fusion factor and target fused image corresponding to the fused image; The input module is used to acquire a convolutional neural network model, improve the convolutional neural network model to obtain a target neural network model, input the target fused image into the target neural network model, and obtain the online defect identification result corresponding to the pipe; the first processing module is specifically used to: acquire the pipe part and the background part in the target depth image, filter the background part, and create a mask corresponding to the target depth image based on the pipe part and the background part; The first processing module maps the corresponding first pixel in the mask to a three-dimensional space to obtain the corresponding first spatial coordinates, projects the first spatial coordinates onto a two-dimensional plane corresponding to the color image to obtain aligned pixels; based on the aligned pixels, the mask and the color image are fused, and the color image is processed to remove background based on the mask to obtain the fused image; the second processing module is specifically used to: obtain an image segmentation model, use the image segmentation model to segment the fused image to obtain a segmentation result, and obtain the fusion effect corresponding to the fused image based on the segmentation result; based on the fusion effect, the fused image is divided into valid images and invalid images, and based on the valid images and invalid images, the proportion of the valid images and invalid images and the fusion factor are obtained; the fusion factor is optimized using the image segmentation model to obtain the target fusion factor and the target fused image corresponding to the fused image.

7. A device for identifying pipeline defects, comprising: Processor and memory; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the pipeline defect identification device to perform the pipeline defect identification method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for identifying pipeline defects as described in any one of claims 1 to 5.

Images