A method for detecting embankment surface defects based on deep learning image processing

Abstract

This invention discloses a method for detecting levee surface defects based on deep learning image processing. The method comprises: acquiring levee surface defect image data and real-time monitoring video image data of the levee surface; annotating the defect location, defect category, and defect hazard level in the levee surface defect image data to obtain annotated image data; preprocessing the annotated image data to obtain a training dataset; training a preset deep learning multi-task fusion network based on the training dataset to obtain a levee surface defect detection model; preprocessing the real-time monitoring video image data and inputting the preprocessed real-time monitoring video image data into the levee surface defect detection model to output levee surface defect detection results. This invention improves the efficiency and accuracy of levee surface defect detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for detecting embankment surface defects based on deep learning image processing. Background Art

[0002] Traditional methods for detecting surface defects in levees usually rely on manual inspections and manual testing, which is not only inefficient but also has high safety risks and subjective biases. Manual inspections require a lot of manpower and are easily affected by external factors such as weather, light, and time, resulting in missed detections or false detections. In addition, the accuracy of manual inspections is low, making it difficult to ensure comprehensive detection of every potential defect in the levee. Traditional automated detection methods, such as simple algorithms based on image processing, although effective in some cases, often rely on manually set thresholds and fixed rules, have slow processing speeds, and are prone to misidentification in complex environments. Therefore, how to improve the efficiency and accuracy of detecting surface defects in levees has always been a focus of attention in the engineering field.

[0003] Therefore, the applicant has developed a method for detecting embankment surface defects based on deep learning image processing to solve the above problems. Summary of the Invention

[0004] The present invention proposes a levee surface defect detection method based on deep learning image processing to solve the problems of low detection efficiency and easy misidentification in complex environments in existing levee surface defect detection methods.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] The present invention provides a method for detecting embankment surface defects based on deep learning image processing, comprising:

[0007] Acquire embankment surface defect image data and real-time monitoring video image data of the embankment surface;

[0008] marking defect locations, defect categories, and defect hazard levels in the embankment surface defect image data to obtain marked image data;

[0009] Preprocess the labeled image data to obtain a training data set;

[0010] A preset deep learning multi-task fusion network is trained according to the training data set to obtain a levee surface defect detection model. The levee surface defect detection model is used to first perform feature extraction and downsampling processing on the preprocessed image data through a first CNN feature downsampling module, and then process the output of the first CNN feature downsampling module through three task branch modules: a defect category task module, a defect hazard level classification task module, and a defect location task module, wherein:

[0011] The defect classification task module is used to process the output of the first CNN feature downsampling module through the second CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect category through the fully connected module;

[0012] The defect hazard level classification task module is used to process the output of the first CNN feature downsampling module through the third CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect hazard level through the fully connected module;

[0013] The defect location task module is used to process the output of the first CNN feature downsampling module through the CNN feature upsampling module and output the pixel coordinates of the defect;

[0014] The real-time monitoring video image data is preprocessed and the preprocessed real-time monitoring video image data is input into the embankment surface defect detection model to output the embankment surface defect detection result.

[0015] Furthermore, the defect location, defect type, and defect hazard level in the embankment surface defect image data are marked, including:

[0016] Using a polygonal frame to mark the coordinate range of the defect area in the embankment surface defect image data;

[0017] Classifying the embankment surface defect image data into different defect categories according to the type, shape and nature of the defects, the defect categories including cracks, subsidence, spalling and water seepage;

[0018] The embankment surface defect image data is graded according to the threat it poses to the embankment stability, and is classified into normal, slight, medium, and severe.

[0019] Furthermore, the labeled image data is preprocessed, including:

[0020] Performing image enhancement processing on the annotated image data, wherein the image enhancement processing includes rotation, scaling, cropping, and light adjustment;

[0021] The annotated embankment surface defect image data and labels are automatically generated according to the enhancement processing method, and the automatically generated labels are processed with one-hot processing and relationship mapping to obtain an enhanced dataset;

[0022] The enhanced dataset is uniformly resized to the same size as the network input, and then the image is normalized so that each pixel value ranges between 0 and 1.

[0023] Furthermore, uniformly resizing the enhanced dataset includes:

[0024] The image data is resized uniformly using the resize method to obtain the image pixel values ​​of the uniform size. , the adjusted size is 512*512*3;

[0025] Perform maximum and minimum normalization on the resized image to obtain the normalized image pixel value , so that each pixel value ranges from 0 to 1, the normalization formula is:

[0026] .

[0027] Furthermore, the loss function of the preset deep learning multi-task fusion network trained according to the training data set is:

[0028]

[0029] The true labels and predicted labels of defect category classification are 、 , the true label and predicted label of the hazard level classification are 、 , the true label and predicted label of the defect location are 、 , represents the weights of the defect category classification task and the defect hazard level classification task, represents the weight of the defect location task, is the cross entropy loss function.

[0030] Furthermore, the input size of the first CNN feature downsampling module is 512*512*3. The first CNN feature downsampling module uses 512, 256, and 128 convolution kernel modules in sequence to perform feature extraction operations on the input, and performs 1*2 pooling to obtain a 64*64*128 feature matrix.

[0031] Furthermore, the defect category classification task module uses 128, 64, and 32 convolution kernel modules in sequence to extract features from the output of the first CNN feature downsampling module, flattens the extracted features, and then uses 512-dimensional full connection and 512-dimensional long short-term memory networks to compress the features, and then splices the compressed features. Finally, the spliced ​​features are converged through 5-dimensional full connection as the spatial feature vector of the defect category.

[0032] Furthermore, the defect hazard level classification task module uses 128, 64, and 32 convolution kernel modules to extract features from the output of the first CNN feature downsampling module in sequence, and then flattens the extracted features and uses 512-dimensional full connection and 512-dimensional long short-term memory networks to compress the features, splices the compressed features, and finally uses 4-dimensional full connection to perform final feature convergence on the spliced ​​features as the spatial feature vector of the hazard level.

[0033] Furthermore, the defect location task module uses a deconvolution module with convolution kernel parameters of 64, 16, and 1 on the output of the first CNN feature downsampling module to obtain a 512*512*1 spatial matrix as the defect location.

[0034] Furthermore, it also includes generating a levee surface defect report based on the levee surface defect detection results, the levee surface defect report includes basic report information, basic defect information and additional information, the basic report information includes report number, detection time, detection personnel, monitoring video source, levee area and report generation time, the basic defect information includes a detailed description of the defect number, type, hazard level, location and related images, the related images include original images, enhanced images and marked images, and the additional information includes defect handling means information.

[0035] The beneficial effects of the present invention are:

[0036] The proposed method for detecting levee surface defects based on deep learning image processing utilizes an innovatively constructed levee surface defect detection model architecture to achieve real-time location detection, defect classification, and hazard level determination of levee surface defects, significantly improving defect detection efficiency. The model can simultaneously process multiple types of information within the same network, enabling comprehensive analysis of levee surface defects. This avoids data transfer and repeated calculations between multiple models, improving detection speed and accuracy while reducing hardware resource consumption. Through a comprehensive deep learning model, the present invention simplifies the architecture of the entire detection system, providing levee managers with a more efficient and comprehensive defect detection and analysis tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the embankment surface defect detection method based on deep learning image processing in this application;

[0038] Figure 2 Schematic diagram of the model architecture of the dike surface defect detection model in an embodiment of the present application;

[0039] Figure 3 Schematic diagram of the network structure of the dike surface defect detection model in the embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the structure of the Block module in the embodiment of the present application;

[0041] Figure 5 This is a schematic diagram of the structure of the Dec module in the embodiment of the present application;

[0042] Figure 6 A schematic diagram of a template for a dike surface defect report generated in an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0046] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is conventionally placed when in use, or are the orientations or positional relationships conventionally understood by those skilled in the art. These are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0047] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0048] In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, terms such as "disposed" and "connected" should be understood in a broad sense. For example, "connected" can mean a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can also mean internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0049] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0050] like Figure 1 As shown, a method for detecting embankment surface defects based on deep learning image processing includes:

[0051] Obtaining embankment surface defect image data and real-time monitoring video image data of the embankment surface, wherein the embankment surface monitoring video is subjected to frame extraction through video processing technology, and each frame in the video is converted into a static image. This process is performed at a fixed frame rate, extracting one frame of image per second, and then filtering out images containing embankment surface defects;

[0052] marking defect locations, defect categories, and defect hazard levels in the embankment surface defect image data to obtain marked image data;

[0053] Preprocess the labeled image data to obtain a training data set;

[0054] The preset deep learning multi-task fusion network is trained according to the training data set to obtain the embankment surface defect detection model, such as Figure 2 As shown, the embankment surface defect detection model is used to first perform feature extraction and downsampling processing on the preprocessed image data through the first CNN feature downsampling module, and then process the output of the first CNN feature downsampling module through three task branch modules: defect category task module, defect hazard level classification task module, and defect location task module, wherein:

[0055] The defect classification task module is used to process the output of the first CNN feature downsampling module through the second CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect category through the fully connected module;

[0056] The defect hazard level classification task module is used to process the output of the first CNN feature downsampling module through the third CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect hazard level through the fully connected module;

[0057] The defect location task module is used to process the output of the first CNN feature downsampling module through the CNN feature upsampling module and output the pixel coordinates of the defect;

[0058] The real-time monitoring video image data is preprocessed and the preprocessed real-time monitoring video image data is input into the embankment surface defect detection model to output the embankment surface defect detection result.

[0059] In some embodiments, marking the defect location, defect type, and defect hazard level in the embankment surface defect image data includes:

[0060] Using a polygonal frame to mark the coordinate range of the defect area in the embankment surface defect image data;

[0061] Classifying the embankment surface defect image data into different defect categories according to the type, shape and nature of the defects, the defect categories including cracks, subsidence, spalling and water seepage;

[0062] The embankment surface defect image data is graded according to the threat it poses to the embankment stability, and is classified into normal, slight, medium, and severe.

[0063] In some embodiments, preprocessing the annotated image data includes:

[0064] Performing image enhancement processing on the annotated image data, wherein the image enhancement processing includes rotation, scaling, cropping, and light adjustment;

[0065] The method for jointly adjusting brightness and contrast is as follows:

[0066]

[0067] in represents the pixel value of the adjusted image, is the contrast adjustment parameter, is the brightness adjustment parameter, is the pixel value of the image, and x and y are the coordinates of the pixel.

[0068] The annotated embankment surface defect image data and labels are automatically generated according to the enhancement processing method. The automatically generated labels are processed with one-hot processing and relationship mapping to obtain an enhanced dataset. The dataset is divided into training set, validation set and test set in a ratio of 7:2:1 for deep learning multi-task fusion network training.

[0069] The enhanced dataset is uniformly resized to the same size as the network input size, and then the image is normalized so that each pixel value ranges between 0 and 1.

[0070] In some embodiments, uniformly resizing the enhanced dataset includes:

[0071] The image data is resized uniformly using the resize method to obtain the image pixel values ​​of the uniform size. , the adjusted size is 512*512*3;

[0072] Perform maximum and minimum normalization on the resized image to obtain the normalized image pixel value , so that each pixel value ranges from 0 to 1, the normalization formula is:

[0073] .

[0074] In some embodiments, the processed image data is fed into the designed deep learning multi-task fusion network for high-order semantic extraction, and the inference result is compared with the label result. The error is calculated through the loss function, and back propagation is performed through the optimizer to update the parameters in the network. When the loss function converges, the deep learning model file can be obtained, and the model training stops. The loss function of the preset deep learning multi-task fusion network trained according to the training data set is:

[0075]

[0076] The true labels and predicted labels of defect category classification are 、 , the true label and predicted label of the hazard level classification are 、 , the true label and predicted label of the defect location are 、 , represents the weights of the defect category classification task and the defect hazard level classification task, Represents the weight of the defect location task.

[0077] like Figure 3 、 Figure 4 and Figure 5As shown: In some embodiments, the network is a single-input three-output structure, such as Figure 3 As shown in the structure, the input size is 512*512*3, and the network uses convolution, normalization and pooling as feature extraction modules, such as Figure 4 As shown in the structure, the module first uses the convolution kernel to extract features from the input feature matrix, superimposes the extracted results and input data on the matrix and uses normalization processing, and finally performs pooling processing; upsampling and convolution are used as deconvolution modules, such as Figure 5 As shown in the structure, the network uses 512, 256, and 128 convolution kernel modules to extract features from three inputs, and performs 1*2 pooling to obtain a 64*64*128 feature matrix; then it processes the feature extraction modules using 128, 64, and 32 in sequence, and after flattening, it uses 512-dimensional full connection and 512-dimensional long short-term memory network (LSTM) to compress features, splices the compressed features, and finally uses 5-dimensional full connection to perform final feature convergence as the spatial feature vector of the defect category; similarly, the 64*6 The 4*128 feature matrix is ​​processed using the 128, 64, and 32 feature extraction modules, respectively. After flattening, it is compressed using a 512-dimensional fully connected network and a 512-dimensional long short-term memory (LSTM) network. The compressed features are concatenated and finally converged using a 4-dimensional fully connected network to form the spatial feature vector of the hazard level. The 64*64*128 feature matrix is ​​processed using the deconvolution module with kernel parameters of 64, 16, and 1, respectively, resulting in a 512*512*1 spatial matrix. In the figure, Batch Normalization represents batch normalization, Relu represents the Relu function, and sigmoid represents the sigmoid function. Dropout is a regularization technique used to prevent overfitting in neural networks. Upsampling represents upsampling. Conv is the abbreviation for the convolution operation. Same represents the padding strategy used in convolutional layers to ensure that the spatial dimensions of the input and output data remain consistent after the convolution operation.

[0078] like Figure 6 As shown, in some embodiments, it also includes generating a levee surface defect report based on the levee surface defect detection results, the levee surface defect report includes basic report information, defect basic information and additional information, the report basic information includes report number, detection time, detection personnel, monitoring video source, levee area and report generation time, the defect basic information includes a detailed description of the defect number, type, hazard level, location and related images, the related images include original images, enhanced images and marked images, and the additional information includes defect handling means information.

[0079] The advantages of the present invention compared to the prior art are:

[0080] (1) The present invention simplifies the system architecture through the deep learning multi-task fusion model innovatively designed by the applicant.

[0081] A significant advantage of the present invention is that by designing a deep learning multi-task fusion network, one model can simultaneously complete multiple tasks, including real-time location detection of levee surface defects, defect category identification, and hazard level determination. Traditional image processing methods often require the design of multiple different models to perform different tasks, such as location detection, classification, and hazard assessment, which not only increases the complexity of the system but also leads to a waste of computing resources. The deep learning multi-task learning strategy adopted by the present invention greatly improves the efficiency of the model by sharing the feature extraction process of the convolution layer. The model can process multiple types of information simultaneously in the same network to achieve a comprehensive analysis of levee surface defects. This method avoids data transfer and repeated calculations between multiple models, improves detection speed and accuracy, and reduces the consumption of hardware resources. Through a comprehensive deep learning model, the architecture of the entire detection system can be simplified, providing levee managers with more efficient and comprehensive defect detection and analysis tools.

[0082] (2) The present invention improves the accuracy and real-time performance of embankment surface defect detection.

[0083] Based on deep learning image processing technology, this invention employs a single-input, multi-output neural network (CNN+LSTM) model for automated detection and analysis of levee surface defects. This deep learning network automatically extracts high-level features from images and accurately identifies and classifies defects. This significantly improves defect detection accuracy, particularly in complex environments and under varying lighting conditions. Traditional manual inspection methods rely on manual inspections, which are susceptible to human and environmental factors and carry the risk of missed and false detections. Deep learning, trained on extensive historical data, can quickly identify subtle levee surface defects with high stability and robustness. In particular, the multi-output network design not only identifies defect locations but also simultaneously determines defect categories (such as cracks and corrosion) and hazard levels, achieving comprehensive defect identification. This automated and intelligent detection method improves the efficiency and accuracy of levee management, enabling timely identification of potential problems and preventing levee damage or disasters.

[0084] (3) The present invention adapts to complex environments and improves the intelligence level of embankment safety management.

[0085] Traditional methods for detecting surface defects in levees often rely on fixed rules or manually set thresholds, resulting in unstable detection results when faced with complex environmental conditions (such as varying lighting and weather). Deep learning models, on the other hand, can continuously optimize through a self-learning process, adapting to changing environments and offering greater adaptability. This invention utilizes a deep learning model based on convolutional neural networks and long-short-term memory networks. This model can continuously learn in complex environments and automatically adjust detection parameters to ensure stable detection results. By training on large amounts of image data, deep learning enables levee monitoring under varying weather conditions, seasons, and time periods, ensuring real-time detection and timely early warning. Furthermore, this invention incorporates new monitoring data into the training process through an online learning mechanism, enabling the model to continuously optimize and adapt to new environmental changes, thereby enhancing the intelligent level of levee management. Real-time defect detection and hazard assessment can provide decision makers with timely management information, enhancing levee safety.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting embankment surface defects based on deep learning image processing, characterized in that: include: Acquire embankment surface defect image data and real-time monitoring video image data of the embankment surface; marking defect locations, defect categories, and defect hazard levels in the embankment surface defect image data to obtain marked image data; Preprocess the labeled image data to obtain a training data set; A preset deep learning multi-task fusion network is trained according to the training data set to obtain a dike surface defect detection model. The dike surface defect detection model is used to first perform feature extraction and downsampling processing on the preprocessed image data through a first CNN feature downsampling module, and then process the output of the first CNN feature downsampling module through three task branch modules: a defect category task module, a defect hazard level classification task module, and a defect location task module, wherein: The defect classification task module is used to process the output of the first CNN feature downsampling module through the second CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect category through the fully connected module; The defect hazard level classification task module is used to process the output of the first CNN feature downsampling module through the third CNN feature downsampling module, then perform feature compression through the fully connected module and the LSTM module, and finally output the defect hazard level through the fully connected module; The defect location task module is used to process the output of the first CNN feature downsampling module through the CNN feature upsampling module and output the pixel coordinates of the defect; Preprocessing the real-time monitoring video image data and inputting the preprocessed real-time monitoring video image data into the embankment surface defect detection model to output the embankment surface defect detection result; Among them, the first CNN feature downsampling module, the second CNN feature downsampling module, the third CNN feature downsampling module, and the CNN feature upsampling module are all modules with decreasing number of convolution kernel channels; The input size of the first CNN feature downsampling module is 512*512*3. The first CNN feature downsampling module uses 512, 256, and 128 convolution kernel modules to perform feature extraction operations on the input, and performs 1*2 pooling to obtain a 64*64*128 feature matrix; The defect classification task module sequentially uses convolution kernel modules of 128, 64, and 32 to extract features from the output of the first CNN feature downsampling module. After flattening the extracted features, it uses a 512-dimensional fully connected network and a 512-dimensional long short-term memory network to compress the features. The compressed features are then concatenated. Finally, the concatenated features are converged using a 5-dimensional fully connected network to obtain the final feature vector of the defect category. The defect hazard level classification task module extracts features from the output of the first CNN feature downsampling module using convolution kernel modules of 128, 64, and 32, respectively. The extracted features are then flattened and compressed using a 512-dimensional fully connected network and a 512-dimensional long short-term memory network. The compressed features are concatenated and finally converged using a 4-dimensional fully connected network to obtain the spatial feature vector of the hazard level. The defect location task module uses a deconvolution module with convolution kernel parameters of 64, 16, and 1 on the output of the first CNN feature downsampling module to obtain a 512*512*1 spatial matrix as the defect location.

2. The method for detecting embankment surface defects based on deep learning image processing according to claim 1, characterized in that: The defect location, defect type, and defect hazard level in the embankment surface defect image data are marked, including: Using a polygonal frame to mark the coordinate range of the defect area in the embankment surface defect image data; Classifying the embankment surface defect image data into different defect categories according to the type, shape and nature of the defects, the defect categories including cracks, subsidence, spalling and water seepage; The embankment surface defect image data is graded according to the threat it poses to the embankment stability, and is classified into normal, slight, medium, and severe.

3. A method for detecting embankment surface defects based on deep learning image processing according to claim 1 or 2, characterized in that: Preprocess the annotated image data, including: Performing image enhancement processing on the annotated image data, wherein the image enhancement processing includes rotation, scaling, cropping, and light adjustment; The annotated embankment surface defect image data and labels are automatically generated according to the enhancement processing method, and the automatically generated labels are processed with one-hot processing and relationship mapping to obtain an enhanced dataset; The enhanced dataset is uniformly resized to the same size as the network input, and then the image is normalized so that each pixel value ranges between 0 and 1.

4. The method for detecting embankment surface defects based on deep learning image processing according to claim 3 is characterized in that: The enhanced dataset is uniformly resized, including: The image data is resized uniformly using the resize method to obtain the image pixel values ​​of the uniform size. , the adjusted size is 512*512*3; Perform maximum and minimum normalization on the resized image to obtain the normalized image pixel value , so that each pixel value ranges from 0 to 1, the normalization formula is: 。 5. The method for detecting embankment surface defects based on deep learning image processing according to claim 4 is characterized in that: The loss function of the preset deep learning multi-task fusion network trained according to the training data set is: ， The true labels and predicted labels of defect category classification are 、 , the true label and predicted label of the hazard level classification are 、 , the true label and predicted label of the defect location are 、 , represents the weights of the defect category classification task and the defect hazard level classification task, Represents the weight of the defect location task.

6. The method for detecting embankment surface defects based on deep learning image processing according to claim 1, characterized in that: It also includes generating a levee surface defect report based on the levee surface defect detection results, the levee surface defect report includes basic report information, basic defect information and additional information, the basic report information includes report number, detection time, detection personnel, monitoring video source, levee area and report generation time, the basic defect information includes a detailed description of the defect number, type, hazard level, location and related images, the related images include original images, enhanced images and marked images, and the additional information includes defect handling means information.

Images