Dike surface defect detection method based on deep learning image processing

Abstract

The invention discloses an embankment surface defect detection method based on deep learning image processing. The method comprises the following steps: acquiring embankment surface defect image data of an embankment surface and real-time monitoring video image data of the embankment surface; marking defect positions, defect types and defect hazard levels in the dike surface defect image data to obtain marked image data; preprocessing the marked image data to obtain a training data set; training a preset deep learning multi-task fusion network according to the training data set to obtain an embankment surface defect detection model, preprocessing the real-time monitoring video image data, and inputting the preprocessed real-time monitoring video image data into the embankment surface defect detection model, and outputting a dike surface defect detection result. According to the invention, the dike surface defect detection efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for detecting embankment surface defects based on deep learning image processing. Background Art

[0002] Traditional methods for detecting surface defects in levees usually rely on manual inspections and manual testing, which is not only inefficient but also has high safety risks and subjective biases. Manual inspections require a lot of manpower and are easily affected by external factors such as weather, light, and time, resulting in missed or false detections. In addition, manual inspections have low accuracy and it is difficult to ensure comprehensive detection of every potential defect in the levee. Traditional automated detection methods, such as simple algorithms based on image processing, although effective in some cases, mostly rely on manually set thresholds and fixed rules, have slow processing speeds, and are prone to misidentification in complex environments. Therefore, how to improve the detection efficiency and accuracy of levee surface defects has always been a focus of attention in the engineering field.

[0003] Therefore, the applicant has developed a method for detecting embankment surface defects based on deep learning image processing to solve the above problems. Summary of the invention

[0004] The present invention proposes a levee surface defect detection method based on deep learning image processing to solve the problems of low detection efficiency and easy misidentification in complex environments in existing levee surface defect detection methods.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] The present invention provides a method for detecting embankment surface defects based on deep learning image processing, comprising:

[0007] Acquire embankment surface defect image data and real-time monitoring video image data of the embankment surface;

[0008] Annotating the defect position, defect type, and defect hazard level in the levee surface defect image data to obtain annotated image data;

[0009] Preprocess the labeled image data to obtain a training data set;

[0010] A preset deep learning multi-task fusion network is trained according to the training data set to obtain a dike surface defect detection model, which is used to first perform feature extraction and downsampling processing on the preprocessed image data through a first CNN feature downsampling module, and then process the output of the first CNN feature downsampling module through three task branch modules: a defect category task module, a defect hazard level classification task module, and a defect location task module, wherein:

[0011] The defect category classification task module is used to process the output of the first CNN feature downsampling module through the second CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect category through the fully connected module;

[0012] The defect hazard level classification task module is used to process the output of the first CNN feature downsampling module through the third CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect hazard level through the fully connected module;

[0013] The defect location task module is used to process the output of the first CNN feature downsampling module through the CNN feature upsampling module to output the pixel coordinates of the defect;

[0014] The real-time monitoring video image data is preprocessed and the preprocessed real-time monitoring video image data is input into the levee surface defect detection model to output the levee surface defect detection result.

[0015] Furthermore, the defect location, defect type, and defect hazard level in the levee surface defect image data are marked, including:

[0016] Using a polygonal frame to mark the coordinate range of the defect area in the embankment surface defect image data;

[0017] Classifying the embankment surface defect image data into different defect categories according to the type, shape and nature of the defects, the defect categories including cracks, subsidence, spalling and water seepage;

[0018] The embankment surface defect image data is graded according to the threat to the embankment stability, and is classified into normal, slight, medium and severe.

[0019] Furthermore, the annotated image data is preprocessed, including:

[0020] Performing image enhancement processing on the annotated image data, wherein the image enhancement processing includes rotation, scaling, cropping, and adjusting light;

[0021] The annotated levee surface defect image data and labels are automatically generated according to the enhancement processing method, and the automatically generated labels are processed with one-hot and relationship mapping to obtain an enhanced data set;

[0022] The enhanced dataset is uniformly resized to be the same as the network input size, and then the image is normalized so that each pixel value ranges from 0 to 1.

[0023] Furthermore, uniformly resizing the enhanced data set includes:

[0024] The image data is resized uniformly through the resize method to obtain the image pixel value of the uniform size , the adjusted size is 512*512*3;

[0025] Perform maximum and minimum normalization on the resized image to obtain the normalized image pixel value , so that each pixel value ranges between 0 and 1, the normalization formula is:

[0026] .

[0027] Furthermore, the loss function of the preset deep learning multi-task fusion network trained according to the training data set is:

[0028]

[0029] The true label and predicted label of defect category classification are , , the true label and predicted label of the hazard level classification are , , the true label and predicted label of the defect location are , , represents the weights of the defect category classification task and the defect hazard level classification task, represents the weight of the defect location task, is the cross entropy loss function.

[0030] Furthermore, the input size of the first CNN feature downsampling module is 512*512*3. The first CNN feature downsampling module uses 512, 256, and 128 convolution kernel modules in sequence to perform feature extraction operations on the input, and performs 1*2 pooling to obtain a 64*64*128 feature matrix.

[0031] Furthermore, the defect category classification task module uses 128, 64, and 32 convolution kernel modules in sequence to extract features from the output of the first CNN feature downsampling module, flattens the extracted features, and then uses 512-dimensional full connection and 512-dimensional long short-term memory networks to compress the features, and then splices the compressed features. Finally, the spliced ​​features are converged through a 5-dimensional full connection as the spatial feature vector of the defect category.

[0032] Furthermore, the defect hazard level classification task module uses 128, 64, and 32 convolution kernel modules to extract features from the output of the first CNN feature downsampling module in sequence, and then flattens the extracted features and uses 512-dimensional fully connected and 512-dimensional long short-term memory networks to compress the features, splices the compressed features, and finally uses 4-dimensional fully connected features to perform final feature convergence as the spatial feature vector of the hazard level.

[0033] Furthermore, the defect location task module uses a deconvolution module with convolution kernel parameters of 64, 16, and 1 on the output of the first CNN feature downsampling module to obtain a 512*512*1 spatial matrix as the defect location.

[0034] Furthermore, it also includes generating a levee surface defect report based on the levee surface defect detection results, the levee surface defect report includes basic report information, basic defect information and additional information, the basic report information includes report number, detection time, detection personnel, monitoring video source, levee area and report generation time, the basic defect information includes a detailed description of the defect number, type, hazard level, location and related images, the related images include original images, enhanced images and marked images, and the additional information includes defect handling means information.

[0035] The beneficial effects of the present invention are:

[0036] The levee surface defect detection method based on deep learning image processing proposed in the present invention realizes the real-time location detection of levee surface defects, defect category identification and hazard level determination through the innovatively constructed levee surface defect detection model architecture, greatly improving the defect detection efficiency. The model can process multiple information simultaneously in the same network to realize comprehensive analysis of levee surface defects, avoid data transfer and repeated calculations between multiple models, improve detection speed and accuracy, and reduce the consumption of hardware resources. Through a comprehensive deep learning model, the present invention can simplify the architecture of the entire detection system and provide more efficient and comprehensive defect detection and analysis tools for levee managers. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the dike surface defect detection method based on deep learning image processing in this application;

[0038] Figure 2 A schematic diagram of the model architecture of the dike surface defect detection model in the embodiment of the present application;

[0039] Figure 3 This is a schematic diagram of the network structure of the dike surface defect detection model in the embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the structure of the Block module in the embodiment of the present application;

[0041] Figure 5 This is a schematic diagram of the structure of the Dec module in the embodiment of the present application;

[0042] Figure 6 A schematic diagram of a template for a dike surface defect report generated in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0046] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0047] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0048] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0050] like Figure 1 As shown, a method for detecting embankment surface defects based on deep learning image processing comprises:

[0051] Obtaining levee surface defect image data and real-time monitoring video image data of the levee surface, wherein the levee surface monitoring video is subjected to frame extraction through video processing technology, and each frame in the video is converted into a static image. This process is performed at a fixed frame rate, extracting one frame of image per second, and then filtering out images containing levee surface defects;

[0052] Annotating the defect position, defect type, and defect hazard level in the levee surface defect image data to obtain annotated image data;

[0053] Preprocess the labeled image data to obtain a training data set;

[0054] The preset deep learning multi-task fusion network is trained according to the training data set to obtain a levee surface defect detection model, such as Figure 2 As shown, the levee surface defect detection model is used to first extract features and perform downsampling processing on the preprocessed image data through the first CNN feature downsampling module, and then process the output of the first CNN feature downsampling module through three task branch modules: defect category task module, defect hazard level classification task module and defect location task module, wherein:

[0055] The defect category classification task module is used to process the output of the first CNN feature downsampling module through the second CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect category through the fully connected module;

[0056] The defect hazard level classification task module is used to process the output of the first CNN feature downsampling module through the third CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect hazard level through the fully connected module;

[0057] The defect location task module is used to process the output of the first CNN feature downsampling module through the CNN feature upsampling module to output the pixel coordinates of the defect;

[0058] The real-time monitoring video image data is preprocessed and the preprocessed real-time monitoring video image data is input into the levee surface defect detection model to output the levee surface defect detection result.

[0059] In some embodiments, marking the defect location, defect type, and defect hazard level in the levee surface defect image data includes:

[0060] Using a polygonal frame to mark the coordinate range of the defect area in the embankment surface defect image data;

[0061] Classifying the embankment surface defect image data into different defect categories according to the type, shape and nature of the defects, the defect categories including cracks, subsidence, spalling and water seepage;

[0062] The embankment surface defect image data is graded according to the threat to the embankment stability, and is classified into normal, slight, medium and severe.

[0063] In some embodiments, preprocessing the annotated image data includes:

[0064] Performing image enhancement processing on the annotated image data, wherein the image enhancement processing includes rotation, scaling, cropping, and adjusting light;

[0065] The method for jointly adjusting brightness and contrast is as follows:

[0066]

[0067] in represents the pixel value of the adjusted image, is the contrast adjustment parameter, is the brightness adjustment parameter, is the pixel value of the image, and x and y are the coordinates of the pixel.

[0068] The annotated levee surface defect image data and labels are automatically generated according to the enhancement processing method, and the automatically generated labels are processed with one-hot and relationship mapping to obtain an enhanced data set. The data set is divided into a training set, a validation set, and a test set in a ratio of 7:2:1 for deep learning multi-task fusion network training;

[0069] The enhanced dataset is uniformly resized to be the same as the network input size, and then the image is normalized so that each pixel value ranges from 0 to 1.

[0070] In some embodiments, uniformly resizing the enhanced dataset includes:

[0071] The image data is resized uniformly through the resize method to obtain the image pixel value of the uniform size , the adjusted size is 512*512*3;

[0072] Perform maximum and minimum normalization on the resized image to obtain the normalized image pixel value , so that each pixel value ranges between 0 and 1, the normalization formula is:

[0073] .

[0074] In some embodiments, the processed image data is sent to the designed deep learning multi-task fusion network for high-order semantic extraction, and the inference result is compared with the label result, the error is calculated through the loss function, and the back propagation is performed through the optimizer to update the parameters in the network. When the loss function converges, the deep learning model file can be obtained, and the model training stops. The loss function of the deep learning multi-task fusion network preset for training according to the training data set is:

[0075]

[0076] The true label and predicted label of defect category classification are , , the true label and predicted label of the hazard level classification are , , the true label and predicted label of the defect location are , , represents the weights of the defect category classification task and the defect hazard level classification task, Represents the weight of the defect location task.

[0077] like Figure 3 , Figure 4 and Figure 5As shown: In some embodiments, the network is a single-input three-output structure, such as Figure 3 As shown in the structure, the input size is 512*512*3, and the network uses convolution, normalization and pooling as feature extraction modules, such as Figure 4 As shown in the structure, the module first uses the convolution kernel to extract features from the input feature matrix, superimposes the extracted results and the input data into a matrix and uses standardization, and finally performs pooling processing; upsampling and convolution are used as deconvolution modules, such as Figure 5 As shown in the structure, the network uses 512, 256, and 128 convolution kernel modules to extract features from three inputs, and performs 1*2 pooling to obtain a 64*64*128 feature matrix; then the feature extraction modules using 128, 64, and 32 are processed in sequence, and after flattening, 512-dimensional full connection and 512-dimensional long short-term memory network (LSTM) are used to compress features, and the compressed features are concatenated. Finally, the final feature convergence is performed through 5-dimensional full connection as the spatial feature vector of the defect category; similarly, the 64*6 The 4*128 feature matrix is ​​processed by the 128, 64, and 32 feature extraction modules in turn. After flattening, it is compressed by the 512-dimensional full connection and the 512-dimensional long short-term memory network (LSTM), and the compressed features are concatenated. Finally, the final feature convergence is performed by the 4-dimensional full connection as the spatial feature vector of the hazard level; the 64*64*128 feature matrix is ​​processed by the deconvolution module with convolution kernel parameters of 64, 16, and 1 to obtain a 512*512*1 spatial matrix. In the figure, Batch Normalization means batch normalization, Relu means Relu function, sigmoid means sigmoid function, dropout is a regularization technique used to prevent overfitting of neural networks, Upsampling means upsampling, Conv is the abbreviation of Convolution, which means convolution operation, and same means padding strategy, which is used in the convolution layer to ensure that the spatial size of input data and output data remains consistent after the convolution operation.

[0078] like Figure 6 As shown, in some embodiments, it also includes generating a levee surface defect report based on the levee surface defect detection results, the levee surface defect report includes basic report information, basic defect information and additional information, the basic report information includes report number, detection time, detection personnel, monitoring video source, levee area and report generation time, the basic defect information includes a detailed description of the defect number, type, hazard level, location and related images, the related images include original images, enhanced images and marked images, and the additional information includes defect handling means information.

[0079] The advantages of the present invention compared to the prior art are:

[0080] (1) The present invention simplifies the system architecture through the deep learning multi-task fusion model innovatively designed by the applicant.

[0081] A significant advantage of the present invention is that by designing a deep learning multi-task fusion network, one model can simultaneously complete multiple tasks, including real-time position detection of levee surface defects, defect category identification, and hazard level determination. Traditional image processing methods often require the design of multiple different models to perform different tasks, such as position detection, classification, and hazard assessment, which not only increases the complexity of the system, but also leads to a waste of computing resources. The deep learning multi-task learning strategy adopted by the present invention greatly improves the efficiency of the model by sharing the feature extraction process of the convolution layer. The model can process multiple types of information simultaneously in the same network to achieve a comprehensive analysis of levee surface defects. This method avoids data transfer and repeated calculations between multiple models, improves detection speed and accuracy, and reduces the consumption of hardware resources. Through a comprehensive deep learning model, the architecture of the entire detection system can be simplified, providing levee managers with more efficient and comprehensive defect detection and analysis tools.

[0082] (2) The present invention improves the accuracy and real-time performance of embankment surface defect detection.

[0083] Based on deep learning image processing technology, the present invention adopts a single-input multi-output neural network (CNN+LSTM) model to automatically detect and analyze surface defects of levees. Through the deep learning network, the model can automatically extract high-level features from the image and accurately identify and classify defects, especially in complex environments and different lighting conditions, which can greatly improve the accuracy of defect detection. Traditional manual detection methods rely on manual inspections, which are easily affected by human factors and environmental factors, and there is a risk of missed detection and false detection. Deep learning can identify subtle defects on the surface of levees in a short time through training of a large amount of historical data, and has high stability and robustness. In particular, the multi-output network design can not only identify the location of defects, but also determine the category of defects (such as cracks, corrosion, etc.) and hazard level at the same time, so as to achieve comprehensive defect recognition. This automated and intelligent detection method improves the efficiency and accuracy of levee management, can detect potential problems in a timely manner, and avoid levee damage or disasters.

[0084] (3) The present invention can adapt to complex environments and improve the intelligence level of embankment safety management.

[0085] Traditional methods for detecting surface defects of levees often rely on fixed rules or manually set thresholds. When faced with complex environmental conditions (such as changes in illumination, weather influences, etc.), the detection effect is unstable. However, the deep learning model can be continuously optimized through the self-learning process, adapt to various changing environments, and has stronger adaptability. In the present invention, a deep learning model based on convolutional neural networks and long short-term memory networks is adopted, which can continuously learn in various complex environments and automatically adjust the detection parameters to ensure the stability of the detection results. By training a large amount of image data, deep learning can monitor levees under changing weather conditions, different seasons and time periods, ensuring real-time detection and timely warning of levees. In addition, the present invention can integrate new monitoring data into the training process through an online learning mechanism, so that the model can continuously optimize itself, gradually adapt to new environmental changes, and improve the intelligent level of levee management. Real-time defect detection and hazard assessment can provide decision makers with timely management information and enhance the safety of levees.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting embankment surface defects based on deep learning image processing, characterized in that: include: Acquire embankment surface defect image data and real-time monitoring video image data of the embankment surface; Annotating the defect position, defect type, and defect hazard level in the levee surface defect image data to obtain annotated image data; Preprocess the labeled image data to obtain a training data set; A preset deep learning multi-task fusion network is trained according to the training data set to obtain a dike surface defect detection model, which is used to first perform feature extraction and downsampling processing on the preprocessed image data through a first CNN feature downsampling module, and then process the output of the first CNN feature downsampling module through three task branch modules: a defect category task module, a defect hazard level classification task module, and a defect location task module, wherein: The defect category classification task module is used to process the output of the first CNN feature downsampling module through the second CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect category through the fully connected module; The defect hazard level classification task module is used to process the output of the first CNN feature downsampling module through the third CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect hazard level through the fully connected module; The defect location task module is used to process the output of the first CNN feature downsampling module through the CNN feature upsampling module to output the pixel coordinates of the defect; The real-time monitoring video image data is preprocessed and the preprocessed real-time monitoring video image data is input into the levee surface defect detection model to output the levee surface defect detection result.

2. The method for detecting embankment surface defects based on deep learning image processing according to claim 1 is characterized in that: The defect location, defect type, and defect hazard level in the levee surface defect image data are marked, including: Using a polygonal frame to mark the coordinate range of the defect area in the embankment surface defect image data; Classifying the embankment surface defect image data into different defect categories according to the type, shape and nature of the defects, the defect categories including cracks, subsidence, spalling and water seepage; The embankment surface defect image data is graded according to the threat to the embankment stability, and is classified into normal, slight, medium and severe.

3. A method for detecting embankment surface defects based on deep learning image processing according to claim 1 or 2, characterized in that: Preprocess the annotated image data, including: Performing image enhancement processing on the annotated image data, wherein the image enhancement processing includes rotation, scaling, cropping, and adjusting light; The annotated levee surface defect image data and labels are automatically generated according to the enhancement processing method, and the automatically generated labels are processed with one-hot and relationship mapping to obtain an enhanced data set; The enhanced dataset is uniformly resized to be the same as the network input size, and then the image is normalized so that each pixel value ranges from 0 to 1.

4. The method for detecting embankment surface defects based on deep learning image processing according to claim 3 is characterized in that: The enhanced dataset is uniformly resized, including: The image data is resized uniformly through the resize method to obtain the image pixel value of the uniform size , the adjusted size is 512*512*3; Perform maximum and minimum normalization on the resized image to obtain the normalized image pixel value , so that each pixel value ranges between 0 and 1, the normalization formula is: 。 5. The method for detecting embankment surface defects based on deep learning image processing according to claim 4 is characterized in that: The loss function of the preset deep learning multi-task fusion network trained according to the training data set is: ， The true label and predicted label of defect category classification are , , the true label and predicted label of the hazard level classification are , , the true label and predicted label of the defect location are , , represents the weights of the defect category classification task and the defect hazard level classification task, Represents the weight of the defect location task.

6. The method for detecting embankment surface defects based on deep learning image processing according to claim 1 is characterized in that: The input size of the first CNN feature downsampling module is 512*512*3. The first CNN feature downsampling module uses 512, 256, and 128 convolution kernel modules in sequence to perform feature extraction operations on the input, and performs 1*2 pooling to obtain a 64*64*128 feature matrix.

7. The method for detecting embankment surface defects based on deep learning image processing according to claim 1 is characterized in that: The defect category classification task module uses 128, 64, and 32 convolution kernel modules in turn to extract features from the output of the first CNN feature downsampling module, flattens the extracted features, and then uses 512-dimensional full connection and 512-dimensional long short-term memory networks to compress the features, then splices the compressed features, and finally uses 5-dimensional full connection to perform final feature convergence on the spliced ​​features as the spatial feature vector of the defect category.

8. The method for detecting embankment surface defects based on deep learning image processing according to claim 1 is characterized in that: The defect hazard level classification task module uses 128, 64, and 32 convolution kernel modules to extract features from the output of the first CNN feature downsampling module, and then flattens the extracted features and uses 512-dimensional full connection and 512-dimensional long short-term memory networks to compress the features, splices the compressed features, and finally uses 4-dimensional full connection to perform final feature convergence on the spliced ​​features as the spatial feature vector of the hazard level.

9. The method for detecting embankment surface defects based on deep learning image processing according to claim 1, characterized in that: The defect location task module uses a deconvolution module with convolution kernel parameters of 64, 16, and 1 on the output of the first CNN feature downsampling module to obtain a 512*512*1 spatial matrix as the defect location.

10. The method for detecting embankment surface defects based on deep learning image processing according to claim 1, characterized in that: It also includes generating a levee surface defect report based on the levee surface defect detection results, the levee surface defect report including basic report information, basic defect information and additional information, the basic report information including report number, detection time, detection personnel, monitoring video source, levee area and report generation time, the basic defect information including a detailed description of the defect number, type, hazard level, location and related images, the related images including original images, enhanced images and marked images, the additional information including defect handling means information.

Images