Defect identification method and device for pipeline image based on convolutional neural network

Abstract

The invention provides a defect identification method and device for a pipeline image based on a convolutional neural network, and relates to the technical field of image identification, and the method comprises the steps: carrying out the piecewise linear gray-scale transformation processing of a to-be-identified drainage pipe network image and a standard drainage pipe network image, and obtaining a first gray-scale image and a second gray-scale image; the gray mapping relation can be adjusted according to different regional characteristics of the image content, so that various types of defect details can be more effectively highlighted. And the gray pixel difference value between the first gray image and the second gray image is determined, so that the defect area in the image can be accurately positioned. The feature matrix is generated based on the gray pixel difference value, and feature recognition is performed on the feature matrix through the preset convolutional neural network to determine the defect category of the drainage pipe network indicated by the drainage pipe network image, so that the degree of manual participation is greatly reduced, and relatively high performance can be maintained under different illumination conditions and noise levels.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for defect recognition of pipeline images based on a convolutional neural network. Background Art

[0002] The sewage pipe network is called the "vein" of the city. It is responsible for the collection, transportation and discharge of industrial and domestic wastewater. Therefore, it plays a very critical role in maintaining the daily operation of the city. Therefore, it is necessary to maintain and inspect the sewage pipe network from time to time to observe and judge whether there are defects inside the sewage pipe network. Commonly used detection methods include CCTV detection technology, QV detection technology, water quality and quantity analysis technology, bottom-penetrating radar detection technology, etc. Generally speaking, manual interpretation is still used to identify whether defects exist in the inspection video or inspection image, as well as the defect type, level, etc.; of course, there are also many image recognition technologies applied to the identification and extraction of defective images in sewage pipe network videos. How to improve the accuracy of image recognition? The key is to effectively process the image. At present, traditional image processing algorithms include: filtering, edge detection, image synthesis, image segmentation, etc. These methods mainly rely on manually designed feature extractors, such as Sobel, Canny, etc., and traditional machine learning algorithms for classification. These image processing methods have various shortcomings; for example, (1) in the grayscale process, the contrast, color channel and grayscale pixel features cannot be effectively extracted; (2) in the grayscale process, the global grayscale transformation of the image is used, and this method applies the same transformation rules to the entire image, and cannot optimize the details of different areas, etc.

[0003] Therefore, for the complex and changeable internal environment of the pipeline, especially when the defect characteristics are not obviously different from the background, the global grayscale transformation is difficult to effectively highlight the defects. In addition, since the diversity of the internal structure of the pipeline and the difference in defect types are not taken into account, it is difficult for traditional methods to accurately identify specific types of defects. This leads to inaccurate detection results and may miss some subtle but important defects. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a pipeline image defect recognition method and device based on convolutional neural network, which can significantly improve the effect of pipeline image defect recognition.

[0005] In a first aspect, an embodiment of the present invention provides a defect recognition method for pipeline images based on a convolutional neural network, the method comprising: obtaining a drainage network image to be identified, and a standard drainage network image corresponding to the drainage network image; performing piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image, respectively, to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image; determining the grayscale pixel difference between the first grayscale image and the second grayscale image; generating a feature matrix based on the grayscale pixel difference, performing feature recognition on the feature matrix through a preset convolutional neural network, and outputting a recognition result of the drainage network image; based on the recognition result, determining the defect category of the drainage network indicated by the drainage network image.

[0006] In combination with the first aspect, an embodiment of the present invention also provides a first implementation of the first aspect, wherein the step of determining the grayscale pixel difference between the first grayscale image and the second grayscale image includes: based on a preset segmentation size, segmenting the first grayscale image and the second grayscale image to obtain a plurality of first segmented images corresponding to the first grayscale image and a plurality of second segmented images corresponding to the second grayscale image; subtracting the pixel grayscale value of each first segmented image from the pixel grayscale value of the corresponding second segmented image to obtain the grayscale pixel difference of the drainage network image at the corresponding segmentation position.

[0007] In combination with the first aspect, an embodiment of the present invention also provides a second implementation of the first aspect, wherein the step of generating a feature matrix based on grayscale pixel differences includes: determining the absolute value of the grayscale pixel difference of the drainage network image at each segmentation position; and generating a feature matrix corresponding to the drainage network image based on the absolute value.

[0008] In combination with the first aspect, an embodiment of the present invention also provides a third implementation of the first aspect, wherein the steps of performing piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image respectively to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image include: calculating the initial grayscale values ​​corresponding to the drainage network image and the standard drainage network image respectively; obtaining multiple pre-set grayscale transformation thresholds, comparing the initial grayscale value with the grayscale transformation threshold, and determining the transformation interval corresponding to the current initial grayscale value; the transformation interval includes a preset transformation slope; based on the transformation interval, performing linear transformation processing on the initial grayscale value to obtain the transformed grayscale value; based on the transformed grayscale value, determining the first grayscale image corresponding to the drainage network image, or the second grayscale image corresponding to the standard drainage network image.

[0009] In combination with the first aspect, an embodiment of the present invention also provides a fourth implementation of the first aspect, wherein the steps of calculating the initial grayscale values ​​corresponding to the drainage network image and the standard drainage network image respectively include: obtaining the red pixel component, green pixel component and blue pixel component corresponding to each pixel in the drainage network image or the standard drainage network image; calculating the weighted sum of the red pixel component, the green pixel component and the blue pixel component according to preset component weights; and determining the initial grayscale value corresponding to the drainage network image or the initial grayscale value corresponding to the standard drainage network image based on the weighted sum.

[0010] In combination with the first aspect, an embodiment of the present invention also provides a fifth implementation of the first aspect, wherein the above method further includes: performing noise reduction processing on the grayscale image based on the local area variance of the grayscale image of the drainage network image.

[0011] In combination with the first aspect, an embodiment of the present invention also provides a sixth implementation of the first aspect, wherein the above method also includes: training a preset convolutional neural network through a pre-constructed training sample set to obtain a trained convolutional neural network; wherein the training sample set includes a defect feature matrix corresponding to the defective pipeline image, and the convolutional neural network includes multiple convolutional layers, multiple pooling layers and fully connected layers, and the multiple convolutional layers and the multiple pooling layers are alternately arranged; the fully connected layer is set with a preset Dropout rate.

[0012] In combination with the first aspect, an embodiment of the present invention also provides a seventh implementation of the first aspect, wherein each convolution layer includes a convolution kernel of a preset size and a preset convolution step size.

[0013] In combination with the first aspect, an embodiment of the present invention also provides an eighth implementation of the first aspect, wherein each pooling layer includes a preset operation scale and a preset moving step size.

[0014] In a second aspect, an embodiment of the present invention provides a defect recognition device for pipeline images based on a convolutional neural network, wherein the device includes: an image acquisition module, used to acquire a drainage network image to be identified, and a standard drainage network image corresponding to the drainage network image; a data processing module, used to perform piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image, respectively, to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image; a calculation module, used to determine the grayscale pixel difference between the first grayscale image and the second grayscale image; an execution module, used to generate a feature matrix based on the grayscale pixel difference, perform feature recognition on the feature matrix through a preset convolutional neural network, and output the recognition result of the drainage network image; an output module, used to determine the defect category of the drainage network indicated by the drainage network image based on the recognition result.

[0015] The embodiments of the present invention bring the following beneficial effects: The embodiments of the present invention provide a method and device for defect recognition of pipeline images based on convolutional neural networks, which combine piecewise linear grayscale transformation with feature recognition based on deep learning, so that the system can maintain high performance under different lighting conditions and noise levels. In addition, the method can adapt to new defect types by continuously updating the training set, and has good generalization ability and long-term applicability. Moreover, the embodiments of the present invention use pre-trained convolutional neural networks to analyze these features, realizing an automated mapping process from image data to defect categories. This step greatly reduces the degree of manual participation, improves work efficiency, and ensures the consistency and objectivity of the classification results.

[0016] Among them, the embodiment of the present invention adopts piecewise linear grayscale transformation processing, which can adjust the grayscale mapping relationship according to the characteristics of different regions of the image content, so as to more effectively highlight the details of various types of defects. This method not only enhances the overall contrast of the image, but also pays special attention to the changes in local areas, making small or hidden defects more obvious. The grayscale pixel difference can accurately locate the defective area in the image. Compared with the traditional global processing method, this method can more finely distinguish normal structures from abnormal situations and improve the accuracy of defect detection.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flow chart of a method for defect recognition of pipeline images based on a convolutional neural network provided by an embodiment of the present invention is shown; Figure 2 A flow chart of another defect recognition method for pipeline images based on a convolutional neural network provided by an embodiment of the present invention is shown; Figure 3A schematic diagram of the structure of a pipeline image defect recognition device based on a convolutional neural network provided by an embodiment of the present invention is shown; Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The embodiments of the present invention provide a pipeline image defect recognition method and device based on a convolutional neural network, which can significantly improve the effect of pipeline image defect recognition.

[0023] For ease of understanding, a defect recognition method for pipeline images based on a convolutional neural network provided in an embodiment of the present invention is first described. Figure 1 A flow chart of a method for defect recognition of pipeline images based on a convolutional neural network provided by an embodiment of the present invention is shown. Figure 1 , the method comprises the following steps: Step S102, obtaining a drainage network image to be identified and a standard drainage network image corresponding to the drainage network image.

[0024] Step S104 , performing piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image respectively, to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image.

[0025] Step S106, determining the grayscale pixel difference between the first grayscale image and the second grayscale image.

[0026] Step S108, generating a feature matrix based on the grayscale pixel difference, performing feature recognition on the feature matrix through a preset convolutional neural network, and outputting the recognition result of the drainage network image.

[0027] Step S110: determining the defect category of the drainage network indicated by the drainage network image based on the recognition result.

[0028] The embodiment of the present invention performs piecewise linear grayscale transformation processing on the drainage network image to be identified and the corresponding standard drainage network image, respectively, to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image, and can adjust the grayscale mapping relationship according to the characteristics of different regions of the image content, so as to more effectively highlight the details of various types of defects. And determine the grayscale pixel difference between the first grayscale image and the second grayscale image, and accurately locate the defective area in the image. Then, a feature matrix is ​​generated based on the grayscale pixel difference, and feature recognition is performed on the feature matrix through a preset convolutional neural network to determine the defect category of the drainage network indicated by the drainage network image, which not only greatly reduces the degree of manual participation, but also can maintain high performance under different lighting conditions and noise levels.

[0029] Furthermore, based on the above embodiments, the embodiments of the present invention also provide another defect recognition method for pipeline images based on convolutional neural networks for illustration. Figure 2 A flowchart of another defect recognition method for pipeline images based on a convolutional neural network provided by an embodiment of the present invention is shown. Figure 2 , the method comprises the following steps: Step S202: acquiring a drainage network image to be identified and a standard drainage network image corresponding to the drainage network image.

[0030] Step S204 , performing piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image respectively, to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image.

[0031] The embodiment of the present invention performs piecewise linear transformation on the image to determine the corresponding transformation interval based on the different grayscale conditions in the image, effectively highlighting the details of various types of defects. In the specific implementation, the corresponding grayscale image is determined through the following steps: 1) Calculate the initial grayscale values ​​corresponding to the drainage network image and the standard drainage network image respectively. In the specific implementation, by obtaining the red pixel component, green pixel component and blue pixel component corresponding to each pixel in the drainage network image or the standard drainage network image; according to the preset component weights, calculate the weighted sum of the red pixel component, the green pixel component and the blue pixel component; based on the weighted sum, determine the initial grayscale value corresponding to the drainage network image, or the initial grayscale value corresponding to the standard drainage network image.

[0032] The initial grayscale value is represented by f(i, j), and the calculation formula for the initial grayscale value of the drainage network image and the standard drainage network image is as follows: f(i,j)=0.299×R(i,j)+0.587×G(i,j)+0.114×B(i,j) Where (i, j) represents the coordinates of the image pixel, f(i, j) is the initial grayscale value of pixel (i, j), R(i, j), G(i, j) and B(i, j) are the RGB values ​​of pixel (i, j); (R, G, B represent the pixel components of the red, green and blue channels respectively).

[0033] 2) Obtain multiple pre-set grayscale transformation thresholds, compare the initial grayscale value with the grayscale transformation threshold, and determine the transformation interval corresponding to the current initial grayscale value. Based on the transformation interval, perform linear transformation processing on the initial grayscale value to obtain the transformed grayscale value. Based on the transformed grayscale value, determine the first grayscale image corresponding to the drainage network image, or the second grayscale image corresponding to the standard drainage network image.

[0034] The specific transformation is shown in the following calculation formula: ①When 0 ≤ f ( i , j )< a hour:

[0035] Indicates that the transformation will f ( x , y ) has a value from [0, a ) linearly maps to [0, c ).

[0036] ② When a ≤ f ( i , j )< b hour:

[0037] Indicates that the transformation will f ( x , y ) value from [ a , b ) maps to [ c , d ).

[0038] ③When b ≤ f ( x , y ) ≤ M f hour:

[0039] Indicates that the transformation will f ( i , j ) value from [ b , M f ] maps to [ d , M g ].

[0040] in ，f ( i , j ) is the original function; g ( i , j ) represents the transformed function; M f Represents the original function f ( i , j )’s maximum value; M g Represents the transformed function g ( i , j ) is the maximum value of . a The first threshold representing the original function value is used to determine the boundaries of the segmentation; b A second threshold representing the original function value, used to determine the boundaries of the segments; c Represents the transformed function g ( i , j ) minimum value; d Represents the transformed function g ( i , j ) to determine the range of the transformation.

[0041] Furthermore, the embodiment of the present invention also performs noise reduction processing on the drainage network image. In one embodiment, the grayscale image is subjected to noise reduction processing to improve the image quality. In a specific implementation, the embodiment of the present invention performs noise reduction processing on the grayscale image based on the local area variance of the grayscale image of the drainage network image, and can adaptively adjust the noise reduction intensity, thereby better retaining the image details while removing the noise.

[0042] In the specific implementation, the grayscale image is denoised through the following steps: 1. For each pixel (X i ,Y i ), collect the pixels in its neighborhood.

[0043] 2. Use the weight calculation formula to calculate the weight for each pixel in the neighborhood. The calculation formula is as follows: (1) (2) (3) (4) in, It is an adaptive threshold function that dynamically adjusts the threshold according to local image characteristics. MaxVar is the maximum value of the variance of all local regions in the image and is used for normalization; is a small positive number used to avoid division by zero. Var(X i , Y i ) is (X i , Y i ) is used to adjust the influence of the grayscale difference term. α Represents the standard deviation in the spatial domain, which is used to measure the distance between a pixel and its center point in space; β Represents the standard deviation of the grayscale domain, which is used to measure the distance between a pixel and its center point in terms of grayscale value; where d i is the distance between the ith pixel and the center of its neighborhood, is the average of these distances, g i is the gray value of the ith pixel, is the average of these grayscale values, and N is the number of pixels in the neighborhood.

[0044] W: weight of the current image pixel; K i Represents a constant for any small image, used to adjust the scale of the weight; (X i , Y i ) represents the coordinates of the current pixel; (X0, Y0) represents the coordinates of the center point of the image; g(X i , Y i ) represents the pixel (X i ,Y i ); g(X0, Y0) represents the gray value of the center point (X0, Y0) of the image.

[0045] Among them, the weight calculation formula of the embodiment of the present invention includes a spatial distance term, a grayscale difference term and a gradient term (that is, the above-mentioned adaptive threshold function), which not only measures the distance between the pixel point and the center point, but also takes into account the grayscale value difference of the pixel point and the variance of the local area, which is used to suppress noise. The weight calculation formula comprehensively considers multiple factors, not only the spatial distance, but also the pixel intensity difference and the local variance, to more comprehensively process image noise. In addition, the calculation process considers the adaptive threshold function, which can better retain the edge information in the image while removing noise. It ensures that the weight is lower in the edge area, thereby avoiding blurred edges. Combined with the variance term Var(X i , Y i ), different weights can be used in different areas to adapt to different parts of the image, improving the noise reduction effect while maintaining details. Moreover, the weight calculation formula is nonlinear and can handle complex structures in the image more flexibly.

[0046] 3. Perform weighted average of the pixels in the neighborhood. Taking the current pixel as the center, the weighted average of the pixels in the neighborhood will be used as the pixel value after noise reduction. The formula is as follows:

[0047] Among them, g(X i ,Y i ) means (X i ,Y i ) gray value, g′(X i ,Y i ) represents the grayscale value after noise reduction.

[0048] 4. g′(X i ,Y i ) is assigned to (X i ,Y i ) to update the pixel value and repeat the above operation until the image is free of noise.

[0049] Here, the noise reduction process is explained: Assume that a 3×3 image area is processed and the coordinates of the central pixel are (X i ,Y i ), the grayscale values ​​of the pixels in its neighborhood are as follows:

[0050] Assume that the coordinates of the center point (X0, Y0) of the image are the same as (X i ,Y i ) is the same, that is, (X0,Y0)=(X i ,Y i), and g(X0,Y0)=50. Correspondingly, based on the corresponding data, we get: standard deviation α=5, β=10, local variance Var(X i ,Y i )=100, adaptive threshold function , constant K i =1.

[0051] For each pixel in the neighborhood (X j ,Y j ), calculate the weight W(X j ,Y j ). i−1 ,Y i ) as an example, its gray value is 20, and the weight is calculated as follows: W(X i−1 ,Y i )≈0.704.

[0052] Similarly, following the above steps, all weights can be obtained:

[0053] Finally, it can be calculated that g′(Xi,Yi)≈56.38.

[0054] Step S206 , segmenting the first grayscale image and the second grayscale image based on a preset segmentation size to obtain a plurality of first segmented images corresponding to the first grayscale image and a plurality of second segmented images corresponding to the second grayscale image.

[0055] The embodiment of the present invention compares the image of the drainage network to be tested with the image of the standard drainage network to locate the defect area based on the pixel difference. In the specific implementation, the image is divided into regions of preset sizes, and the defect location analysis is performed based on the pixel difference of each region.

[0056] Step S208 : Subtract the pixel grayscale value of each first segmented image from the pixel grayscale value of the corresponding second segmented image to obtain the grayscale pixel difference value of the drainage network image at the corresponding segmented position.

[0057] Step S210, determining the absolute value of the grayscale pixel difference of the drainage network image at each segmentation position; based on the absolute value, generating a feature matrix corresponding to the drainage network image.

[0058] Specifically, the drainage network image and the standard drainage network image are evenly divided into N×N grids, where the N×N small grid images are called the first segmented image and the second segmented image, respectively. The grayscale pixel values ​​at the same position of the first segmented image and the second segmented image are subtracted, and the absolute value of the subtraction constitutes a feature matrix. The image edge is filled with 0, and the input layer constitutes a feature matrix with the image grayscale value. Assuming that it is cut into 245 images, the initial feature matrix is ​​245×245.

[0059] Step S212: Perform feature recognition on the feature matrix through a preset convolutional neural network, and output the recognition result of the drainage network image.

[0060] Step S214: determining the defect category of the drainage network indicated by the drainage network image based on the recognition result.

[0061] The embodiment of the present invention recognizes images through a convolutional neural network, wherein the convolutional neural network can be trained by a training sample set corresponding to the above-mentioned drainage network image, and the original normal pipeline image and the original defect image can be screened out by a traversal method, thereby performing defect recognition.

[0062] Among them, the convolutional neural network of the embodiment of the present invention includes multiple convolutional layers, multiple pooling layers and fully connected layers, and multiple convolutional layers and multiple pooling layers are alternately arranged; the fully connected layer is set with a preset Dropout rate. Each convolutional layer includes a convolution kernel of a preset size and a preset convolution step; each pooling layer includes a preset operation scale and a preset moving step. Specifically, the training or recognition process of the convolutional neural network is as follows.

[0063] The first convolution is performed on the initial feature matrix. The convolution kernel size can be 5×5 (can be adjusted according to actual conditions) and the convolution step size is 2. The size of the first output feature matrix after convolution is 123×123. The first output feature matrix is ​​used as the input layer, and the second convolution layer is performed. The convolution kernel size is 3×3 and the convolution step size is 2. The convolution is performed on the first output feature matrix. The size of the second output feature matrix is ​​61×61. The third layer is the pooling layer. The average pooling layer is used for the second output feature matrix. The pooling layer operation scale is 3×3 and the moving step size is Strides=2. The size of the third output feature matrix is ​​30×30. The fourth layer is the convolution layer. The convolution is performed on the third output feature matrix. The convolution kernel size is 3×3 and the convolution step size is 2. The size of the fourth output feature matrix is ​​14×14. The fifth layer is the maximum pooling layer, which operates on the fourth output feature matrix with an operation size of 2×2 and a moving step size of Strides=1. It can be obtained that the size of the fifth output feature matrix is ​​7×7.

[0064] The embodiments of the present invention can extract features at multiple scales through convolution kernels of different sizes and different step sizes, which helps to capture the details and overall structure in the image. The alternating use of average pooling and maximum pooling can reduce the data dimension while retaining key features. Average pooling can smooth features, while maximum pooling can highlight the most significant local features. Through a series of convolution and pooling operations, the size of the feature map is gradually reduced, while the level of abstraction of the features is increased, which helps to extract higher-level feature representations.

[0065] The sixth layer is a fully connected layer, and the input layer is the features of the output layer of the fifth layer. Dropout=0.15 is set, which means that the probability of each neuron being discarded during the training process of the neural network is 0.15. After the neurons in the fifth layer are discarded, they are connected to the fully connected layer containing 1024 neuron nodes through the action of the activation function. The seventh fully connected layer, the input layer is the output features of the sixth layer, that is, the 1024 neurons output by the sixth layer, which are discarded by dropout=0.15, and then fully connected with the 1024 neuron nodes in the seventh layer through the action of the activation function (the activation function uses softmax). The embodiment of the present invention uses multiple Dropouts in the fully connected layer, and each time the probability is set to 0.15 to randomly discard neurons, which helps prevent overfitting of the model and improve generalization ability.

[0066] In the eighth fully connected layer, there are N kinds of defective images in the drainage pipe, so the N defective images are fully connected to the 1024 neuron nodes output by the upper layer, and then the calculated value is output. After the feature map before the fully connected layer is flattened, high-dimensional feature mapping is performed through two fully connected layers of 1024 neurons, which can effectively learn complex feature relationships. The fully connected layer directly corresponds to N defect categories, and uses the softmax function to output the probability of each category, which is suitable for multi-classification tasks. In summary, the convolutional neural network of the embodiment of the present invention can effectively extract and distinguish different types of pipeline defects while maintaining computational efficiency, thereby improving the accuracy and robustness of recognition.

[0067] Another defect recognition method for pipeline images based on convolutional neural networks provided by an embodiment of the present invention determines the pixel difference corresponding to the standard image after performing piecewise linear transformation on the drainage network image to be recognized, and extracts global features of the drainage network image, which can not only enhance the key features in the image, especially under poor lighting conditions, but also capture local details while retaining global structural information, which helps to improve detection accuracy.

[0068] Moreover, the embodiment of the present invention trains and recognizes images through a convolutional neural network, uses multi-layer convolution operations, and gradually extracts features at different levels. Combining average pooling and maximum pooling, it can retain the main features and reduce the amount of data. The embodiment of the present invention can quickly and accurately identify the defects in the image.

[0069] In combination with the above embodiments, the present invention also provides a pipeline image defect recognition device based on a convolutional neural network. Figure 3 FIG. 1 shows a schematic diagram of a defect recognition device for pipeline images based on a convolutional neural network according to an embodiment of the present invention. Figure 3 The device includes: an image acquisition module 100, which is used to acquire a drainage network image to be identified and a standard drainage network image corresponding to the drainage network image; a data processing module 200, which is used to perform piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image, respectively, to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image; a calculation module 300, which is used to determine the grayscale pixel difference between the first grayscale image and the second grayscale image; an execution module 400, which is used to generate a feature matrix based on the grayscale pixel difference, perform feature recognition on the feature matrix through a preset convolutional neural network, and output the recognition result of the drainage network image; an output module 500, which is used to determine the defect category of the drainage network indicated by the drainage network image based on the recognition result.

[0070] A defect recognition device for pipeline images based on a convolutional neural network provided in an embodiment of the present invention has the same technical features as a defect recognition method for pipeline images based on a convolutional neural network provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0071] Furthermore, based on the above embodiments, the embodiments of the present invention also provide another defect recognition device for pipeline images based on convolutional neural networks, wherein the above-mentioned calculation module 300 is also used to segment the first grayscale image and the second grayscale image based on a preset segmentation size to obtain multiple first segmented images corresponding to the first grayscale image and multiple second segmented images corresponding to the second grayscale image; subtract the pixel grayscale value of each first segmented image from the pixel grayscale value of the corresponding second segmented image to obtain the grayscale pixel difference value of the drainage network image at the corresponding segmentation position.

[0072] The execution module 400 is further used to determine the absolute value of the grayscale pixel difference of the drainage network image at each segmentation position; based on the absolute value, generate a feature matrix corresponding to the drainage network image.

[0073] The data processing module 200 is also used to calculate the initial grayscale values ​​corresponding to the drainage network image and the standard drainage network image respectively; obtain multiple pre-set grayscale transformation thresholds, compare the initial grayscale value with the grayscale transformation threshold, and determine the transformation interval corresponding to the current initial grayscale value; the transformation interval includes a preset transformation slope; based on the transformation interval, the initial grayscale value is linearly transformed to obtain the transformed grayscale value; based on the transformed grayscale value, the first grayscale image corresponding to the drainage network image, or the second grayscale image corresponding to the standard drainage network image, is determined.

[0074] The data processing module 200 is also used to obtain the red pixel component, green pixel component and blue pixel component corresponding to each pixel in the drainage network image or the standard drainage network image; calculate the weighted sum of the red pixel component, the green pixel component and the blue pixel component according to the preset component weights; and determine the initial grayscale value corresponding to the drainage network image or the initial grayscale value corresponding to the standard drainage network image based on the weighted sum.

[0075] The data processing module 200 is further used to perform noise reduction processing on the grayscale image of the drainage pipe network image based on the local area variance of the grayscale image.

[0076] The execution module 400 is also used to train the preset convolutional neural network through a pre-constructed training sample set to obtain a trained convolutional neural network; wherein the training sample set includes a defect feature matrix corresponding to the defective pipeline image, and the convolutional neural network includes multiple convolutional layers, multiple pooling layers and fully connected layers, and the multiple convolutional layers and the multiple pooling layers are alternately arranged; the fully connected layer is set with a preset Dropout rate. Each convolutional layer includes a convolution kernel of a preset size and a preset convolution step size. Each pooling layer includes a preset operation scale and a preset moving step size.

[0077] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above Figure 1 to Figure 2 The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor when it is run. Figure 1 to Figure 2 The embodiment of the present invention also provides a structural diagram of an electronic device, such as Figure 4 FIG. 4 is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 41 and a memory 40, the memory 40 stores computer executable instructions that can be executed by the processor 41, and the processor 41 executes the computer executable instructions to implement the above Figure 1 to Figure 2 Any of the methods shown. Figure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43, wherein the processor 41, the communication interface 43 and the memory 40 are connected via the bus 42. The memory 40 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 42 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but it does not mean that there is only one bus or one type of bus. The processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 41 or the instructions in the form of software. The above-mentioned processor 41 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register or other mature storage media in the art. The storage medium is located in the memory, and the processor 41 reads the information in the memory and completes the above-mentioned Figure 1 to Figure 2 Any of the methods shown.

[0078] A computer program product of a method and device for defect recognition of pipeline images based on a convolutional neural network provided in an embodiment of the present invention includes a computer-readable storage medium storing a program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can refer to the method embodiment, which will not be repeated here. A person skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the previous method embodiment, which will not be repeated here. In addition, in the description of the embodiment of the present invention, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood in specific circumstances. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, and other media that can store program codes. In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate the orientation or position relationship based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0079] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A pipeline image defect recognition method based on convolutional neural network, characterized in that: The method comprises: Acquire a drainage network image to be identified and a standard drainage network image corresponding to the drainage network image; Performing piecewise linear grayscale transformation processing on the drainage pipe network image and the standard drainage pipe network image respectively to obtain a first grayscale image corresponding to the drainage pipe network image and a second grayscale image corresponding to the standard drainage pipe network image; determining a grayscale pixel difference between the first grayscale image and the second grayscale image; Generate a feature matrix based on the grayscale pixel difference, perform feature recognition on the feature matrix through a preset convolutional neural network, and output a recognition result of the drainage network image; Based on the recognition result, a defect category of the drainage network indicated by the drainage network image is determined.

2. The method according to claim 1, characterized in that The step of determining a grayscale pixel difference between the first grayscale image and the second grayscale image comprises: Based on a preset segmentation size, segment the first grayscale image and the second grayscale image to obtain a plurality of first segmented images corresponding to the first grayscale image and a plurality of second segmented images corresponding to the second grayscale image; The pixel grayscale value of each first segmented image is subtracted from the corresponding pixel grayscale value of the second segmented image to obtain the grayscale pixel difference value of the drainage pipe network image at the corresponding segmented position.

3. The method according to claim 2, characterized in that The step of generating a feature matrix based on the grayscale pixel difference comprises: Determine the absolute value of the grayscale pixel difference of the drainage network image at each segmentation position; Based on the absolute value, a feature matrix corresponding to the drainage network image is generated.

4. The method according to claim 1, characterized in that The step of performing piecewise linear grayscale transformation processing on the drainage network image and the standard drainage network image respectively to obtain a first grayscale image corresponding to the drainage network image and a second grayscale image corresponding to the standard drainage network image comprises: Calculating initial grayscale values ​​corresponding to the drainage pipe network image and the standard drainage pipe network image respectively; Acquire a plurality of preset grayscale transformation thresholds, compare the initial grayscale value with the grayscale transformation threshold, and determine a transformation interval corresponding to the current initial grayscale value; the transformation interval includes a preset transformation slope; Based on the transformation interval, the initial grayscale value is linearly transformed to obtain a transformed grayscale value; Based on the transformed grayscale value, a first grayscale image corresponding to the drainage pipe network image or a second grayscale image corresponding to the standard drainage pipe network image is determined.

5. The method according to claim 4, characterized in that The step of calculating the initial grayscale values ​​corresponding to the drainage network image and the standard drainage network image respectively comprises: Obtaining the drainage pipe network image, or the red pixel component, the green pixel component and the blue pixel component corresponding to each pixel in the standard drainage pipe network image; Calculating a weighted sum of the red pixel component, the green pixel component, and the blue pixel component according to preset component weights; Based on the weighted sum, an initial grayscale value corresponding to the drainage network image, or an initial grayscale value corresponding to the standard drainage network image is determined.

6. The method according to claim 1, characterized in that The method further comprises: Based on the local area variance of the grayscale image of the drainage pipe network image, noise reduction processing is performed on the grayscale image.

7. The method according to claim 1, characterized in that The method further comprises: The preset convolutional neural network is trained by using a pre-constructed training sample set to obtain a trained convolutional neural network; Among them, the training sample set includes a defect feature matrix corresponding to the defective pipeline image, the convolutional neural network includes multiple convolutional layers, multiple pooling layers and a fully connected layer, and the multiple convolutional layers and the multiple pooling layers are alternately arranged; the fully connected layer is set with a preset Dropout rate.

8. The method according to claim 7, characterized in that Each of the convolutional layers includes a convolution kernel of a preset size and a preset convolution step size.

9. The method according to claim 7, characterized in that: Each of the pooling layers includes a preset operation scale and a preset moving step size.

10. A defect recognition device for pipeline images based on convolutional neural network, characterized in that: The device comprises: An image acquisition module, used to acquire a drainage network image to be identified and a standard drainage network image corresponding to the drainage network image; A data processing module, used for performing piecewise linear grayscale transformation processing on the drainage pipe network image and the standard drainage pipe network image respectively, to obtain a first grayscale image corresponding to the drainage pipe network image and a second grayscale image corresponding to the standard drainage pipe network image; a calculation module, configured to determine a grayscale pixel difference between the first grayscale image and the second grayscale image; An execution module, used for generating a feature matrix based on the grayscale pixel difference, performing feature recognition on the feature matrix through a preset convolutional neural network, and outputting a recognition result of the drainage network image; An output module is used to determine the defect category of the drainage network indicated by the drainage network image based on the recognition result.

Images