Defect recognition method and device for pipe image based on convolutional neural network

Abstract

The application provides a pipe image defect recognition method and device based on a convolutional neural network, relates to the technical field of image recognition, and can obtain a first gray image and a second gray image by performing segmented linear gray scale transformation processing on a to-be-recognized drainage pipe network image and a standard drainage pipe network image, can adjust a gray scale mapping relationship according to different regional characteristics of image content, and can more effectively highlight defect details of various types. The gray scale pixel difference between the first gray image and the second gray image is determined, and a defect region in the image can be accurately located. A feature matrix is generated based on the gray scale pixel difference, feature recognition is performed on the feature matrix by using a preset convolutional neural network, the defect category of the drainage pipe network indicated by the drainage pipe network image is determined, the degree of manual participation is greatly reduced, and high performance can be maintained under different illumination conditions and noise levels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a pipe image defect recognition method and device based on a convolutional neural network. BACKGROUND

[0002] Sewer networks are known as the "veins" of the city, and bear the role of collecting, transporting and discharging industrial wastewater and domestic sewage in the city, so they play a very key role in maintaining the daily operation of the city. Therefore, it is necessary to maintain and inspect the sewer network from time to time in order to observe and judge whether there are defects inside the sewer network. Common detection methods include CCTV detection technology, QV detection technology, water quality and quantity analysis technology, and bottom radar detection technology.

[0003] For detection videos or images, manual interpretation is generally used to identify whether there are defects, defect types, levels, etc. Of course, there are also many image recognition technologies applied to the recognition and extraction of defect images in sewer network videos. How to improve the accuracy of image recognition, and the key is to effectively process the images. The current traditional image processing algorithms are: filtering, edge detection, image synthesis, image segmentation, etc. These methods mainly rely on manually designed feature extractors such as Sobel, Canny, etc., and traditional machine learning algorithms for classification. These image processing methods have various shortcomings; such as (1) in the process of grayscale, it is difficult to effectively extract contrast, color channel and grayscale pixel features; (2) in the process of grayscale, global grayscale transformation of the image is used, and this method applies the same conversion rule to the entire image, which cannot optimize the details of different areas, etc.

[0004] Therefore, for complex and variable internal environments of pipes, especially when the defect features and background differences are not obvious, global grayscale transformation cannot effectively highlight the defects. In addition, due to the lack of consideration of the diversity of internal pipe structures and the differences in defect types, traditional methods are difficult to accurately identify specific types of defects. This leads to inaccurate detection results, which may miss some subtle but important defects. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a pipe image defect recognition method and device based on a convolutional neural network, which can significantly improve the effect of pipe image defect recognition.

[0006] In a first aspect, an embodiment of the present application provides a method for defect recognition of a pipeline image based on a convolutional neural network, the method comprising: obtaining a drainage pipe network image to be recognized and a standard drainage pipe network image corresponding to the drainage pipe network image; performing segmented linear gray scale transformation processing on the drainage pipe network image and the standard drainage pipe network image respectively to obtain a first gray scale image corresponding to the drainage pipe network image and a second gray scale image corresponding to the standard drainage pipe network image; determining a gray scale pixel difference value between the first gray scale image and the second gray scale image; generating a feature matrix based on the gray scale pixel difference value, performing feature recognition on the feature matrix by using a preset convolutional neural network, and outputting a recognition result of the drainage pipe network image; and determining a defect category of a drainage pipe network indicated by the drainage pipe network image based on the recognition result.

[0007] In combination with the first aspect, an embodiment of the present application further provides a first implementation manner of the first aspect, wherein the step of determining the gray scale pixel difference value between the first gray scale image and the second gray scale image comprises: segmenting the first gray scale image and the second gray scale image based on a preset segmentation size to obtain a plurality of first segmented images corresponding to the first gray scale image and a plurality of second segmented images corresponding to the second gray scale image; and subtracting a pixel gray scale value of each first segmented image from a pixel gray scale value of a corresponding second segmented image to obtain a gray scale pixel difference value of the drainage pipe network image at a corresponding segmentation position.

[0008] In combination with the first aspect, an embodiment of the present application further provides a second implementation manner of the first aspect, wherein the step of generating the feature matrix based on the gray scale pixel difference value comprises: determining an absolute value of the gray scale pixel difference value of the drainage pipe network image at each segmentation position; and generating a feature matrix corresponding to the drainage pipe network image based on the absolute value.

[0009] In combination with the first aspect, an embodiment of the present application further provides a third implementation manner of the first aspect, wherein the step of performing segmented linear gray scale transformation processing on the drainage pipe network image and the standard drainage pipe network image respectively to obtain the first gray scale image corresponding to the drainage pipe network image and the second gray scale image corresponding to the standard drainage pipe network image comprises: calculating initial gray scale values respectively corresponding to the drainage pipe network image and the standard drainage pipe network image; obtaining a plurality of gray scale transformation threshold values set in advance, comparing the initial gray scale values with the gray scale transformation threshold values, and determining a transformation interval corresponding to a current initial gray scale value; the transformation interval comprises a preset transformation slope; performing linear transformation processing on the initial gray scale value based on the transformation interval to obtain a transformed gray scale value; and determining the first gray scale image corresponding to the drainage pipe network image or the second gray scale image corresponding to the standard drainage pipe network image based on the transformed gray scale value.

[0010] With reference to the first aspect, the embodiment of the present application further provides a fourth implementation manner of the first aspect, and the step of calculating the initial gray value corresponding to the sewer network image and the standard sewer network image respectively comprises: acquiring the red pixel component, the green pixel component and the blue pixel component corresponding to each pixel in the sewer network image or the standard sewer network image; calculating the weighted sum of the red pixel component, the green pixel component and the blue pixel component according to the preset component weight; and determining the initial gray value corresponding to the sewer network image or the initial gray value corresponding to the standard sewer network image based on the weighted sum.

[0011] With reference to the first aspect, the embodiment of the present application further provides a fifth implementation manner of the first aspect, and the method further comprises: performing noise reduction processing on the gray image based on the local region variance of the gray image of the sewer network image.

[0012] With reference to the first aspect, the embodiment of the present application further provides a sixth implementation manner of the first aspect, and the method further comprises: training the preset convolutional neural network through a pre-constructed training sample set to obtain a trained convolutional neural network; wherein the training sample set comprises a defect feature matrix corresponding to a defect pipeline image, the convolutional neural network comprises a plurality of convolutional layers, a plurality of pooling layers and a full connection layer, the plurality of convolutional layers and the plurality of pooling layers are arranged alternately, and the full connection layer is provided with a preset Dropout rate.

[0013] With reference to the first aspect, the embodiment of the present application further provides a seventh implementation manner of the first aspect, and each convolutional layer comprises a convolution kernel with a preset size and a preset convolution step.

[0014] With reference to the first aspect, the embodiment of the present application further provides an eighth implementation manner of the first aspect, and each pooling layer comprises a preset operation scale and a preset moving step.

[0015] The second aspect, the embodiment of the present application provides a pipeline image defect identification device based on a convolutional neural network, wherein the device comprises: an image acquisition module, configured to acquire a sewer network image to be identified and a standard sewer network image corresponding to the sewer network image; a data processing module, configured to perform segmented linear gray scale transformation processing on the sewer network image and the standard sewer network image respectively to obtain a first gray image corresponding to the sewer network image and a second gray image corresponding to the standard sewer network image; a calculation module, configured to determine a gray pixel difference value between the first gray image and the second gray image; an execution module, configured to generate a feature matrix based on the gray pixel difference value, perform feature recognition on the feature matrix through a preset convolutional neural network, and output an identification result of the sewer network image; and an output module, configured to determine a defect category of a sewer network indicated by the sewer network image based on the identification result.

[0016] The embodiments of the present application bring the following beneficial effects: the defect identification method and device for pipeline images based on a convolutional neural network provided by the embodiments of the present application combine piecewise linear gray scale transformation and deep learning-based feature recognition, so that the system can maintain high performance under different illumination conditions and noise levels. In addition, the method can adapt to new defect types by continuously updating the training set, and has good generalization ability and long-term applicability. Moreover, the embodiments of the present application use a pre-trained convolutional neural network to analyze these features, realizing an automatic mapping process from image data to defect categories. This step greatly reduces the degree of human participation, improves work efficiency, and at the same time ensures the consistency and objectivity of the classification results.

[0017] Among them, the embodiments of the present application adopt piecewise linear gray scale transformation processing, which can adjust the gray scale mapping relationship according to the different regional characteristics of image content, thereby more effectively highlighting the details of various types of defects. This method not only enhances the overall contrast of the image, but also pays special attention to the changes in local areas, making small or hidden defects more obvious. The gray pixel difference can accurately locate the defect area in the image. Compared with the traditional global processing method, this method can more finely distinguish between normal structures and abnormal conditions, improving the accuracy of defect detection.

[0018] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the specification, claims and drawings.

[0019] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 A flowchart of a defect identification method for pipeline images based on a convolutional neural network provided by the embodiments of the present application is shown;

[0022] Figure 2 A flowchart of another defect identification method for pipeline images based on a convolutional neural network provided by the embodiments of the present application is shown;

[0023] Figure 3 A structural schematic diagram of a pipeline image defect identification device based on a convolutional neural network is shown.

[0024] Figure 4 A structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with the embodiments. Obviously, the described embodiments are only some, but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0026] The embodiments of the present application provide a pipeline image defect identification method and device based on a convolutional neural network, which can significantly improve the effect of pipeline image defect identification.

[0027] For ease of understanding, first, a pipeline image defect identification method based on a convolutional neural network is described. Figure 1 A flowchart of a pipeline image defect identification method based on a convolutional neural network is shown, referring to Figure 1 The method comprises the following steps:

[0028] Step S102, obtaining a drainage pipe network image to be identified and a standard drainage pipe network image corresponding to the drainage pipe network image.

[0029] Step S104, performing segmented linear gray scale transformation processing on the drainage pipe network image and the standard drainage pipe network image respectively to obtain a first gray scale image corresponding to the drainage pipe network image and a second gray scale image corresponding to the standard drainage pipe network image.

[0030] Step S106, determining the gray scale pixel difference value between the first gray scale image and the second gray scale image.

[0031] Step S108, generating a feature matrix based on the gray scale pixel difference value, performing feature recognition on the feature matrix through a preset convolutional neural network, and outputting the identification result of the drainage pipe network image.

[0032] Step S110, determining the defect category of the drainage pipe network indicated by the drainage pipe network image based on the identification result.

[0033] The embodiment of the present application can effectively highlight various types of defect details by respectively performing segmented linear gray scale transformation processing on the to-be-identified drainage pipe network image and the corresponding standard drainage pipe network image, obtaining a first gray scale image corresponding to the drainage pipe network image and a second gray scale image corresponding to the standard drainage pipe network image, and adjusting the gray scale mapping relationship according to the different regional characteristics of the image content. The gray scale pixel difference between the first gray scale image and the second gray scale image can be determined to accurately locate the defect region in the image. Then, a feature matrix is generated based on the gray scale pixel difference, and the feature matrix is subjected to feature recognition by a preset convolutional neural network to determine the defect category of the drainage pipe network indicated by the drainage pipe network image, which not only greatly reduces the degree of human participation, but also maintains high performance under different illumination conditions and noise levels.

[0034] Further, on the basis of the above-mentioned embodiment, the embodiment of the present application also provides another defect identification method for a pipe image based on a convolutional neural network. Figure 2 A flowchart of another defect identification method for a pipe image based on a convolutional neural network provided by the embodiment of the present application is shown, which is described with reference to Figure 2 The method comprises the following steps:

[0035] In step S202, a to-be-identified drainage pipe network image and a standard drainage pipe network image corresponding to the drainage pipe network image are obtained.

[0036] In step S204, segmented linear gray scale transformation processing is performed on the drainage pipe network image and the standard drainage pipe network image respectively, to obtain a first gray scale image corresponding to the drainage pipe network image and a second gray scale image corresponding to the standard drainage pipe network image.

[0037] In the embodiment of the present application, the segmented linear transformation processing is performed on the image to determine the corresponding transformation interval based on different gray scale conditions in the image, so as to effectively highlight various types of defect details. In the specific implementation, the corresponding gray scale image is determined by the following steps:

[0038] 1) The initial gray scale values corresponding to the drainage pipe network image and the standard drainage pipe network image are calculated. In the specific implementation, the red pixel component, the green pixel component and the blue pixel component corresponding to each pixel in the drainage pipe network image or the standard drainage pipe network image are obtained; the weighted sum of the red pixel component, the green pixel component and the blue pixel component is calculated according to the preset component weight; and the initial gray scale value corresponding to the drainage pipe network image or the initial gray scale value corresponding to the standard drainage pipe network image is determined based on the weighted sum.

[0039] In the embodiment of the present application, the initial gray scale value is represented by f(i,j), and the calculation formula of the initial gray scale value of the drainage pipe network image and the standard drainage pipe network image is as follows:

[0040] f(i, j) = 0.299 x R(i, j) + 0.587 x G(i, j) + 0.114 x B(i, j)

[0041] where (i, j) represents the coordinates of the image pixel, f(i, j) is the initial gray value of the pixel (i, j), R(i, j), G(i, j) and B(i, j) are the RGB values of the pixel (i, j); (R, G, B represent the red, green and blue three-channel pixel components, respectively).

[0042] 2) Obtain a plurality of preset gray scale transformation thresholds, compare the initial gray value with the gray scale transformation threshold, and determine the transformation interval corresponding to the current initial gray value. Based on the transformation interval, the initial gray value is linearly transformed to obtain the transformed gray value. Based on the transformed gray value, the first gray image corresponding to the drainage pipe network image is determined, or the second gray image corresponding to the standard drainage pipe network image is determined.

[0043] The specific transformation is shown in the following calculation formula:

[0044] ① When 0 ≤ f(i, j) < T1: f ( i , j )< a

[0045]

[0046] represents that the transformation linearly maps the value of (i, j) from [0, T1) to [0, T2). f ( x , y ). a c ② When T1 ≤ f(i, j) < T2:

[0047] a ≤ f ( i , j )< b

[0048]

[0049] represents that the transformation maps the value of (i, j) from [T1, T2) to [T2, T3). f ( x , y ). a b c d ③ When T2 ≤ f(i, j) < T3:

[0050] b ≤ f ( x , y ) ≤​​​​​​​​M f Time:

[0051]

[0052] representing the transformation maps the values of f ( i , j ) from the interval [ b , M f ] to the interval [ d , M g ].

[0053] where ，f ( i , j ) is the original function; g ( i , j ) represents the transformed function; M f representing the maximum value of the original function f ( i , j ); M g representing the maximum value of the transformed function g ( i , j ). a representing the first threshold value of the original function value, used to determine the boundary of the segmentation; b representing the second threshold value of the original function value, used to determine the boundary of the segmentation; c representing the minimum value of the transformed function g ( i , j ); d representing the intermediate value of the transformed function g ( i , j ), used to determine the range of the transformation.

[0054] Further, the embodiment of the present application also performs noise reduction processing on the sewer network image. In one implementation, the gray-scale image is processed to improve the image quality. In the specific implementation, the embodiment of the present application performs noise reduction processing on the gray-scale image based on the local area variance of the gray-scale image of the sewer network image, and can adaptively adjust the noise reduction strength, so as to better retain the image details while removing the noise.

[0055] In the specific implementation, the gray-scale image is processed by the following steps:

[0056] 1. For each pixel point (X iY i ), collect the pixel points in its neighborhood.

[0057] 2. Calculate the weight of each pixel point in the neighborhood using the weight calculation formula, which is as follows:

[0058] (1)

[0059] (2)

[0060] (3)

[0061] (4)

[0062] wherein, is an adaptive threshold function that can dynamically adjust the threshold according to the local image characteristics. MaxVar is the maximum value of the variance of all local regions in the image, used for normalization; is a small positive number, used to avoid division by zero. Var(X i , Y i ) is the variance of the local region centered at (X i , Y i ), used to adjust the influence of the gray difference term. α represents the standard deviation in the spatial domain, used to measure the distance of the pixel point from the center point in the spatial position; β represents the standard deviation in the gray domain, used to measure the distance of the pixel point from the center point in the gray value; wherein, d i is the distance of the i-th pixel point from the center point of the neighborhood, is the average of these distances, g i is the gray value of the i-th pixel point, is the average of these gray values, and N is the number of pixel points in the neighborhood.

[0063] W: the weight of the current image pixel point; K i represents a constant of any small image, used to adjust the scale of the weight; (X i , Y i ) represents the coordinates of the current pixel point; (X0, Y0) represents the coordinates of the image center point; g(X i , Y i ) represents the gray value of the pixel point (X i , Y i ); and g(X0, Y0) represents the gray value of the image center point (X0, Y0).

[0064] The weight calculation formula of the embodiment of the present application contains a spatial distance item, a gray difference item and a gradient item (i.e. the adaptive threshold function described above), which not only measures the distance of the pixel point from the center point, but also considers the gray value difference of the pixel point and the variance of the local region, for suppressing noise. The weight calculation formula comprehensively considers multiple factors, not only considering the spatial distance, but also considering the pixel intensity difference and the local variance, and more comprehensively processes image noise. In addition, the calculation process considers the adaptive threshold function, which can better preserve the edge information in the image while removing noise. It ensures that the weight in the edge region is low, thereby avoiding blurred edges. Combined with the variance term Var(X i , Y i ), different weights can be used in different regions, thereby adapting to different parts of the image, improving the noise reduction effect while maintaining details. Moreover, the weight calculation formula is nonlinear, which can more flexibly process complex structures in the image.

[0065] 3. Perform weighted average on the pixel points in the neighborhood, with the current pixel point as the center, and the weighted average value of the pixel points in the neighborhood will be used as the pixel value after noise reduction. The formula is as follows:

[0066]

[0067] wherein g(X i ,Y i ) represents the gray value of (X i ,Y i ), and g'(X i ,Y i ) represents the gray value after noise reduction.

[0068] 4. Assign g'(X i ,Y i ) to (X i ,Y i ) to update the pixel value, and repeat the above operation until the image is de-noised.

[0069] wherein the noise reduction process is described: assuming that a 3x3 image region is processed, the coordinates of the center pixel point are (X i ,Y i ), and the gray values of the pixel points in the neighborhood are as follows:

[0070]

[0071] Assuming that the coordinates of the image center point (X0,Y0) are the same as (X i ,Y i ), i.e. (X0,Y0)=(X i ,Y i), and g(X0, Y0) = 50. Correspondingly, based on the corresponding data, the standard deviation a = 5, b = 10, the local variance Var(X i ,Y i ) = 100, the adaptive threshold function , and the constant K i = 1 are obtained.

[0072] For each pixel point (X j ,Y j ) in the neighborhood, the weight W(X j ,Y j ) is calculated. Taking (X i−1 ,Y i ) as an example, the gray value of which is 20, the weight is calculated as follows: W(X i−1 ,Y i ) ≈ 0.704.

[0073] Similarly, according to the above steps, all the weights can be obtained.

[0074]

[0075] Finally, g'(Xi, Yi) ≈ 56.38 can be calculated.

[0076] In step S206, the first gray-scale image and the second gray-scale image are segmented based on a preset segmentation size, to obtain a plurality of first segmented images corresponding to the first gray-scale image and a plurality of second segmented images corresponding to the second gray-scale image.

[0077] In the embodiment of the present application, the image to be tested and the standard sewer network image are compared, and the defect area is located based on the pixel difference value. In the specific implementation, the image is segmented into regions of a preset size, and the defect is located and analyzed based on the pixel difference value of each region.

[0078] In step S208, the pixel gray value of each first segmented image is subtracted from the pixel gray value of the corresponding second segmented image, to obtain the gray pixel difference value of the sewer network image at the corresponding segmentation position.

[0079] In step S210, the absolute value of the gray pixel difference value of the sewer network image at each segmentation position is determined, and a feature matrix corresponding to the sewer network image is generated based on the absolute value.

[0080] Specifically, the sewer network image and the standard sewer network image are averagely divided into N*N grids, and the N*N small grid images are respectively referred to as a first segmentation image and a second segmentation image. The gray pixel values at the same positions of the first segmentation image and the second segmentation image are subtracted, and the absolute values of the subtraction form a feature matrix. The image edge is filled with 0, and the input layer is formed by the feature matrix of the image gray value. Assuming that the image is cut into 245 images, the initial feature matrix is 245*245.

[0081] In step S212, the feature matrix is subjected to feature recognition through a preset convolutional neural network, and an identification result of the sewer network image is output.

[0082] In step S214, based on the identification result, a defect category of the sewer network indicated by the sewer network image is determined.

[0083] In the embodiment of the application, the image is recognized through the convolutional neural network, which can be trained through the training sample set corresponding to the sewer network image, and the original normal pipe image and the original defect image can be screened out through the traversal method, so that the defect recognition is performed.

[0084] The convolutional neural network of the embodiment of the application includes a plurality of convolutional layers, a plurality of pooling layers and a full connection layer, and the plurality of convolutional layers and the plurality of pooling layers are alternately arranged. The full connection layer is provided with a preset Dropout rate. Each convolutional layer includes a convolution kernel of a preset size and a preset convolution step; and each pooling layer includes a preset operation scale and a preset moving step. Specifically, the training or recognition process of the convolutional neural network is as follows.

[0085] The initial feature matrix is subjected to first layer convolution calculation, the convolution kernel size can be 5*5 (which can be adjusted according to the actual situation), and the convolution step is 2. The first output feature matrix size obtained after the convolution is 123*123. The first output feature matrix is taken as the input layer, and the second layer convolution layer calculation is performed. The convolution kernel size is 3*3, the convolution step is 2, the first output feature matrix is subjected to convolution calculation, and the second output feature matrix size obtained is 61*61. The third layer is a pooling layer, the second output feature matrix is processed by an average pooling layer, the pooling layer operation scale is 3*3, and the moving step Strides=2. The third output feature matrix size obtained is 30*30. The fourth layer is a convolution layer, the third output feature matrix is subjected to convolution calculation, the convolution kernel size is 3*3, the convolution step is 2, and the fourth output feature matrix size obtained is 14*14. The fifth layer is a maximum pooling layer, the fourth output feature matrix is subjected to operation, the operation size is 2*2, and the moving step Strides=1. The fifth output feature matrix size obtained is 7*7.

[0086] The embodiment of the present application can extract features on multiple scales through different sizes of convolution kernels and different strides, which helps to capture details and overall structure in the image. The average pooling and the maximum pooling are used alternately, which can not only reduce the data dimension, but also retain key features. The average pooling can smooth the features, while the maximum pooling can highlight the most significant local features. Through a series of convolution and pooling operations, the size of the feature map is gradually reduced, while the abstract level of the features is increased, which helps to extract higher-level feature representations.

[0087] The sixth layer is a full connection layer, the input layer is the output layer of the fifth layer, the dropout is set to 0.15, which represents the probability that each neuron is discarded in the training process of the neural network. After the neurons of the fifth layer are discarded, the full connection layer with 1024 neuron nodes is connected through the action of the activation function. The seventh layer is a full connection layer, the input layer is the output features of the sixth layer, that is, 1024 neurons output by the sixth layer, which are discarded through the dropout of 0.15, and then connected to the 1024 neuron nodes of the seventh layer through the action of the activation function (the activation function adopts softmax). The embodiment of the present application uses multiple Dropout in the full connection layer, each of which is set to a probability of 0.15 to randomly discard neurons, which helps to prevent model overfitting and improve generalization ability.

[0088] The eighth layer is a full connection layer, and the N defect images of the drainage pipeline exist, so the defect images in N are fully connected with the 1024 neuron nodes output by the upper layer, and then the calculation value is output. After the feature map before the full connection layer is flattened, high-dimensional feature mapping is performed through two full connection layers with 1024 neurons, which can effectively learn the complex feature relationship. The full connection layer directly corresponds to N defect categories, and the softmax function is used to output the probability of each category, which is suitable for multi-classification tasks. In summary, the convolutional neural network of the embodiment of the present application can effectively extract and distinguish different types of pipeline defects while maintaining the calculation efficiency, thereby improving the accuracy and robustness of recognition.

[0089] Another drainage pipeline image defect recognition method based on a convolutional neural network provided by the embodiment of the present application can determine the pixel difference value corresponding to the standard image after segmenting and linearly transforming the drainage pipeline network image to be recognized, and can extract the global features of the drainage pipeline network image, which can not only enhance the key features in the image, especially in poor light conditions, but also capture local details while retaining global structure information, which helps to improve the detection accuracy.

[0090] And, the embodiment of the present application trains and recognizes the image through the convolutional neural network, uses the multi-layer convolution operation, and gradually extracts the features of different levels. The average pooling and the maximum pooling are combined, so that the main features can be retained and the data amount can be reduced. The embodiment of the present application can quickly and accurately determine the defects existing in the image.

[0091] In combination with the above embodiment, the embodiment of the present application further provides a defect recognition device for pipeline images based on a convolutional neural network, Figure 3 The structure schematic diagram of the defect recognition device for pipeline images based on the convolutional neural network is shown, and the embodiment of the present application is described in detail. Figure 3 The device comprises an image acquisition module 100, which is used to acquire a drainage pipe network image to be recognized and a standard drainage pipe network image corresponding to the drainage pipe network image; a data processing module 200, which is used to perform segmented linear gray scale conversion processing on the drainage pipe network image and the standard drainage pipe network image respectively, to obtain a first gray scale image corresponding to the drainage pipe network image and a second gray scale image corresponding to the standard drainage pipe network image; a calculation module 300, which is used to determine the gray scale pixel difference value between the first gray scale image and the second gray scale image; an execution module 400, which is used to generate a feature matrix based on the gray scale pixel difference value, perform feature recognition on the feature matrix through a preset convolutional neural network, and output the recognition result of the drainage pipe network image; and an output module 500, which is used to determine the defect category of the drainage pipe network indicated by the drainage pipe network image based on the recognition result.

[0092] The defect recognition device for pipeline images based on the convolutional neural network provided by the embodiment of the present application has the same technical features as the defect recognition method for pipeline images based on the convolutional neural network provided by the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0093] Further, on the basis of the above embodiment, the embodiment of the present application further provides another defect recognition device for pipeline images based on the convolutional neural network, wherein the calculation module 300 is further used to segment the first gray scale image and the second gray scale image based on a preset segmentation size, to obtain a plurality of first segmentation images corresponding to the first gray scale image and a plurality of second segmentation images corresponding to the second gray scale image; and subtract the pixel gray scale value of each first segmentation image from the pixel gray scale value of the corresponding second segmentation image, to obtain the gray scale pixel difference value of the drainage pipe network image at the corresponding segmentation position.

[0094] The execution module 400 is further used to determine the absolute value of the gray scale pixel difference value of the drainage pipe network image at each segmentation position; and generate the feature matrix corresponding to the drainage pipe network image based on the absolute value.

[0095] The data processing module 200 is further configured to calculate initial gray values corresponding to the sewer network image and the standard sewer network image respectively, acquire a plurality of preset gray transformation thresholds, compare the initial gray values with the gray transformation thresholds, determine a transformation interval corresponding to the current initial gray value, and determine a transformation slope in the transformation interval. The data processing module 200 is further configured to perform linear transformation processing on the initial gray values based on the transformation interval to obtain transformed gray values, and determine a first gray image corresponding to the sewer network image or a second gray image corresponding to the standard sewer network image based on the transformed gray values.

[0096] The data processing module 200 is further configured to acquire red, green and blue pixel components corresponding to each pixel in the sewer network image or the standard sewer network image, calculate a weighted sum of the red, green and blue pixel components according to a preset component weight, and determine an initial gray value corresponding to the sewer network image or an initial gray value corresponding to the standard sewer network image based on the weighted sum.

[0097] The data processing module 200 is further configured to perform noise reduction processing on the gray image based on a local area variance of the gray image of the sewer network image.

[0098] The execution module 400 is further configured to train the preset convolutional neural network by using a pre-constructed training sample set to obtain a trained convolutional neural network, wherein the training sample set includes a defect feature matrix corresponding to a defect pipeline image, the convolutional neural network includes a plurality of convolutional layers, a plurality of pooling layers and a full connection layer, the plurality of convolutional layers and the plurality of pooling layers are arranged alternately, and the full connection layer is provided with a preset Dropout rate. Each convolutional layer includes a convolution kernel with a preset size and a preset convolution step. Each pooling layer includes a preset operation scale and a preset moving step.

[0099] The embodiment of the present application further provides an electronic device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above. The embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of any of the methods described above. Figures 1 to 2 The embodiment of the present application further provides a structural schematic diagram of an electronic device, as shown in Figures 1 to 2 The embodiment of the present application further provides a structural schematic diagram of an electronic device, as shown in Figure 4 The embodiment of the present application further provides a structural schematic diagram of an electronic device, as shown in Figures 1 to 2 The embodiment of the present application further provides a structural schematic diagram of an electronic device, as shown inFigure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43, wherein the processor 41, the communication interface 43 and the memory 40 are connected through the bus 42. The memory 40 can include a high-speed random access memory (RAM) and can further include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through the at least one communication interface 43 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 42 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The processor 41 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor 41 or instructions in the form of software. The processor 41 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or other mature storage media in the art. The memory is located in the storage, and the processor 41 reads information in the memory and combines it with its hardware to complete the foregoingFigures 1 to 2 Any of the methods shown.

[0100] The computer program product of the pipeline image defect recognition method and apparatus based on convolutional neural network provided in this invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the preceding method embodiments, which will not be repeated here. In addition, in the description of the embodiments of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0101] Finally, it should be noted that the above examples are merely specific embodiments of the present application, and are used to illustrate the technical solutions of the present application, but are not intended to limit the present application. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing examples, or make equivalent replacements to some of the technical features, within the technical range disclosed by the present application. These modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for defect recognition of pipe images based on a convolutional neural network, characterized in that, The method comprises: acquiring a sewer network image to be identified and a standard sewer network image corresponding to the sewer network image; performing segmented linear gray scale transformation processing on the sewer network image and the standard sewer network image respectively to obtain a first gray scale image corresponding to the sewer network image and a second gray scale image corresponding to the standard sewer network image; determining a gray scale pixel difference value between the first gray scale image and the second gray scale image; generating a feature matrix based on the gray scale pixel difference value, performing feature recognition on the feature matrix through a preset convolutional neural network, and outputting an identification result of the sewer network image; based on the identification result, determining a defect category of a sewer network indicated by the sewer network image; The step of performing segmented linear gray scale transformation processing on the sewer network image and the standard sewer network image respectively to obtain a first gray scale image corresponding to the sewer network image and a second gray scale image corresponding to the standard sewer network image comprises: calculating initial gray scale values respectively corresponding to the sewer network image and the standard sewer network image; acquiring a plurality of gray scale transformation threshold values set in advance, comparing the initial gray scale values with the gray scale transformation threshold values, and determining a transformation interval corresponding to a current initial gray scale value; based on the transformation interval, performing linear transformation processing on the initial gray scale value to obtain a transformed gray scale value; wherein the transformation interval comprises a preset transformation slope; the transformation slope is calculated based on a preset threshold value corresponding to the initial gray scale value and a preset threshold value corresponding to the transformed gray scale value; based on the transformed gray scale value, determining the first gray scale image corresponding to the sewer network image, or the second gray scale image corresponding to the standard sewer network image; wherein the method further comprises: based on the local area variance, the standard deviation of the spatial domain and the standard deviation of the gray scale domain of the first gray scale image of the sewer network image, performing noise reduction processing on the first gray scale image; the local area variance is calculated based on the gray scale values of the pixel points in the neighborhood of the first gray scale image; the transformation slope comprises: When 0 ≤ f ( i , j ) < a : ; When a < f(i, j) < b: ; When b < f(i,j) < Mf: ; wherein ，f ( i , j ) is the original function; g ( i , j ) is the transformed function; M f is the maximum value of the original function f ( i , j ) ; M g is the maximum value of the transformed function g ( i , j ) ; a represents a first threshold value of the original function value for determining the limits of the segments; b represents a second threshold value of the original function value for determining the limits of the segments; c represents the minimum value of the transformed function g ( i , j ) ; d represents the intermediate value of the transformed function g ( i , j ) for determining the range of the transformation; The step of calculating initial gray scale values respectively corresponding to the sewer network image and the standard sewer network image comprises: acquiring a red pixel component, a green pixel component and a blue pixel component corresponding to each pixel in the sewer network image or the standard sewer network image; calculating a weighted sum of the red pixel component, the green pixel component and the blue pixel component according to a preset component weight; based on the weighted sum, determining the initial gray scale value corresponding to the sewer network image, or the initial gray scale value corresponding to the standard sewer network image.

2. The method of claim 1, wherein, The step of determining the gray scale pixel difference value between the first gray scale image and the second gray scale image comprises: based on a preset segmentation size, segmenting the first gray scale image and the second gray scale image to obtain a plurality of first segmented images corresponding to the first gray scale image and a plurality of second segmented images corresponding to the second gray scale image; Subtracting pixel gray value of each of the first segmented image from pixel gray value of corresponding second segmented image, to obtain gray pixel difference value of the sewer network image at corresponding segmented position.

3. The method of claim 2, wherein, The step of generating a feature matrix based on the gray pixel difference value comprises: Determining absolute value of the gray pixel difference value of the sewer network image at each segmented position; Generating a feature matrix corresponding to the sewer network image based on the absolute value.

4. The method of claim 1, wherein, The method further comprises: Training a preset convolutional neural network through a pre-constructed training sample set to obtain a trained convolutional neural network; The training sample set comprises a defect feature matrix corresponding to a defect pipeline image, the convolutional neural network comprises a plurality of convolutional layers, a plurality of pooling layers and a full connection layer, the plurality of convolutional layers and the plurality of pooling layers are arranged alternately, and the full connection layer is provided with a preset Dropout rate.

5. The method of claim 4, wherein, Each of the convolutional layers comprises a convolution kernel with a preset size and a preset convolution step.

6. The method of claim 4, wherein, Each of the pooling layers comprises a preset operation scale and a preset moving step. 7.A defect recognition apparatus for pipe images based on a convolutional neural network, characterized by The device comprises: An image acquisition module configured to acquire a sewer network image to be identified and a standard sewer network image corresponding to the sewer network image; A data processing module configured to perform segmented linear gray scale transformation processing on the sewer network image and the standard sewer network image respectively to obtain a first gray scale image corresponding to the sewer network image and a second gray scale image corresponding to the standard sewer network image; A calculation module configured to determine a gray pixel difference value between the first gray scale image and the second gray scale image; An execution module configured to generate a feature matrix based on the gray pixel difference value, perform feature recognition on the feature matrix through a preset convolutional neural network, and output an identification result of the sewer network image; An output module configured to determine a defect category of a sewer network indicated by the sewer network image based on the identification result; The data processing module is further configured to: calculate initial gray scale values respectively corresponding to the sewer network image and the standard sewer network image; acquire a plurality of preset gray scale transformation thresholds; compare the initial gray scale values with the gray scale transformation thresholds to determine a transformation interval corresponding to a current initial gray scale value; perform linear transformation processing on the initial gray scale value based on the transformation interval to obtain a transformed gray scale value; wherein the transformation interval comprises a preset transformation slope; the transformation slope is calculated based on a preset threshold corresponding to the initial gray scale value and a preset threshold corresponding to the transformed gray scale value; and determine the first gray scale image corresponding to the sewer network image or the second gray scale image corresponding to the standard sewer network image based on the transformed gray scale value; The data processing module is further configured to: perform noise reduction processing on the first gray scale image based on a local area variance, a spatial domain standard deviation and a gray scale domain standard deviation of the first gray scale image; and the local area variance is calculated based on gray scale values of pixel points in a neighborhood of the first gray scale image. The transformation slope comprises: When 0 ≤ f ( i , j ) < a : ; When a < f(i, j) < b: ; When b < f(i,j) < Mf: ; wherein ，f ( i , j ) is the original function; g ( i , j ) is the transformed function; M f is the maximum value of the original function f ( i , j ) ; M g is the maximum value of the transformed function g ( i , j ) ; a represents a first threshold value of the original function value for determining the limits of the segments; b represents a second threshold value of the original function value for determining the limits of the segments; c represents the minimum value of the transformed function g ( i , j ) ; d represents the intermediate value of the transformed function g ( i , j ) for determining the range of the transformation; The data processing module is further configured to: acquire a red pixel component, a green pixel component and a blue pixel component corresponding to each pixel in the sewer network image or the standard sewer network image; calculate a weighted sum of the red pixel component, the green pixel component and the blue pixel component according to a preset component weight; and determine an initial gray value corresponding to the sewer network image or an initial gray value corresponding to the standard sewer network image based on the weighted sum.

Images