Concrete crack detection and identification method, device, equipment and storage medium

Abstract

The application provides a concrete crack detection and identification method, device, equipment and storage medium, the method comprises the steps of: obtaining a plurality of crack image samples in a preset sample library; each crack image sample is manually marked respectively to obtain a first label image containing a first mark, wherein the first mark indicates the crack in the image; edge detection is performed on the first label image to obtain a second label image containing a second mark, wherein the second mark indicates the edge of the crack in the image; based on each crack image sample and its first label image and second label image, a pre-constructed crack detection and identification model is trained to obtain a trained crack detection and identification model; based on the trained crack detection and identification model, the image to be detected is subjected to crack detection and identification. The crack detection and identification method provided by the application can improve the detection and identification accuracy of fine cracks.

Description

Technical Field

[0001] This invention relates to the field of crack detection technology, and in particular to a method, apparatus, equipment and storage medium for detecting and identifying concrete cracks. Background Technology

[0002] In recent years, with the increasing scale of infrastructure construction, concrete has been used in a wide variety of projects. However, as construction time increases, concrete structures develop cracks, which, when they reach a certain width, can seriously affect the safety of the building. Therefore, crack detection and identification have become an important part of the durability assessment of concrete structures.

[0003] Common crack detection methods mainly include manual detection, edge detection, and threshold segmentation detection. With the rapid development of information technology, crack visual detection and recognition methods based on deep learning have become a new trend in the industry.

[0004] Manual inspection suffers from high subjectivity and low efficiency, while edge detection and threshold segmentation are easily affected by environmental factors such as blurring and shadows, leading to significant errors in traditional methods. Deep learning, on the other hand, utilizes convolutional networks to extract multi-layered features from images and trains specific network structures to perform tasks such as image classification, localization, and segmentation, improving accuracy compared to commonly used detection methods. However, existing deep learning-based models have relatively low accuracy in detecting and identifying minute cracks. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for detecting and identifying concrete cracks, in order to solve the problem that current models have low accuracy in detecting and identifying fine cracks.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting and identifying concrete cracks, including:

[0007] Obtain multiple crack image samples from a preset sample library;

[0008] Each crack image sample is manually labeled to obtain a first labeled image containing a first label, wherein the first label indicates the crack in the image;

[0009] Edge detection is performed on the first labeled image to obtain a second labeled image containing a second label, wherein the second label indicates the edge of the crack in the image;

[0010] Based on each crack image sample and its first label image and second label image, the pre-constructed crack detection and recognition model is trained to obtain the trained crack detection and recognition model. The crack detection and recognition model is based on channel convolution operation, which divides the Unet++ decoding network into a backbone segmentation network and an edge prediction branch network. The backbone segmentation network is used to extract all features of each crack image sample, and the edge prediction branch network is used to extract the edge crack features of the first label image.

[0011] The trained crack detection and recognition model is used to detect and recognize cracks in the image to be tested.

[0012] In one possible implementation, a pre-built crack detection and recognition model is trained based on each crack image sample and its first label image and second label image to obtain a trained crack detection and recognition model, which also includes:

[0013] The training weights of the crack image samples corresponding to the first label image are determined based on the crack area in the first label image and the crack edge area in the second label image.

[0014] The pre-built crack detection and recognition model is trained based on each crack image sample, its first label image, its second label image, and the training weights of each crack image sample.

[0015] In one possible implementation, the training weight λ of the crack image sample corresponding to the first labeled image is:

[0016] λ = 0.1 × P / Q;

[0017] Where P is the edge area of ​​the crack in the second label image, and Q is the crack area of ​​the crack in the first label image.

[0018] In one possible implementation, the pre-built crack detection and recognition model is trained based on each crack image sample, its first label image, its second label image, and the training weights of each crack image sample. The training also includes:

[0019] Based on the ground truth map of the first labeled image and the segmentation result of the third image, the backbone loss function of the backbone segmentation network is constructed; wherein, the third image is the image obtained after the crack image sample corresponding to the first labeled image is processed by the backbone segmentation network;

[0020] The prediction branch loss function of each prediction branch network is constructed based on the cross-entropy of the ground truth map of the second label image and the fourth image, as well as the training weights of the crack image samples; wherein, the fourth image is the feature map obtained by passing the crack image sample corresponding to the first label image through each prediction branch network.

[0021] The pre-built crack detection and recognition model is trained based on the backbone loss function, the prediction branch loss function, crack image samples, the first label image, and the second label image.

[0022] In one possible implementation, based on the ground truth map of the first labeled image and the segmentation result of the third image, the backbone loss function of the backbone segmentation network is constructed, including:

[0023] Based on the Dice correlation coefficient between the number of pixels in the ground truth map of the first labeled image and the number of pixels in the segmentation result of the third image, the backbone loss function of the backbone segmentation network is constructed.

[0024] Among them, the backbone loss function L Dice for:

[0025]

[0026] X * Y represents the segmentation result of the third image, and Y represents the ground truth image of the first label image. |X * | and |Y| represent the number of pixels in the ground truth image of the first label image and the number of pixels in the segmentation result of the third image, respectively. |X * ∩Y| represents the number of pixels where the ground truth image of the first labeled image intersects with the segmentation result of the third image.

[0027] In one possible implementation, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network.

[0028] The prediction branch loss function L for each layer of the prediction branch network b for:

[0029]

[0030] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nijThis is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0031] In one possible implementation, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network. The edge prediction branch network also has nn.Parameter for the adaptive weight of each feature map.

[0032] The prediction branch loss function L for each layer of the prediction branch network b for:

[0033]

[0034] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer, w k The weights of each feature map layer are W = {w1, w2, w3, w4, w5}. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nij This is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0035] Secondly, embodiments of the present invention provide a concrete crack detection and identification device, comprising:

[0036] The acquisition module is used to acquire multiple crack image samples from a preset sample library;

[0037] A labeling module is used to manually label each crack image sample to obtain a first labeled image containing a first label, wherein the first label indicates the crack in the image;

[0038] An edge detection module is used to perform edge detection on the first label image to obtain a second label image containing a second label, wherein the second label indicates the edge of the crack in the image;

[0039] The model training module is used to train the pre-built crack detection and recognition model based on each crack image sample and its first label image and second label image to obtain the trained crack detection and recognition model. The crack detection and recognition model is based on channel convolution operation, which divides the Unet++ decoding network into a backbone segmentation network and an edge prediction branch network. The backbone segmentation network is used to extract all features of each crack image sample, and the edge prediction branch network is used to extract the edge crack features of the first label image.

[0040] The detection and recognition module is used to detect and recognize cracks in the image to be detected based on the trained crack detection and recognition model.

[0041] In one possible implementation, the model training module is used to determine the training weights of the crack image samples corresponding to the first label image based on the crack area in the first label image and the edge area of ​​the crack in the second label image.

[0042] The pre-built crack detection and recognition model is trained based on each crack image sample, its first label image, its second label image, and the training weights of each crack image sample.

[0043] In one possible implementation, the training weight λ of the crack image sample corresponding to the first labeled image is:

[0044] λ = 0.1 × P / Q;

[0045] Where P is the edge area of ​​the crack in the second label image, and Q is the crack area of ​​the crack in the first label image.

[0046] In one possible implementation, the model training module is used to construct the backbone loss function of the backbone segmentation network based on the ground truth map of the first label image and the segmentation result of the third image; wherein, the third image is the image obtained after the crack image sample corresponding to the first label image is processed by the backbone segmentation network;

[0047] The prediction branch loss function of each prediction branch network is constructed based on the cross-entropy of the ground truth map of the second label image and the fourth image, as well as the training weights of the crack image samples; wherein, the fourth image is the feature map obtained by passing the crack image sample corresponding to the first label image through each prediction branch network.

[0048] The pre-built crack detection and recognition model is trained based on the backbone loss function, the prediction branch loss function, crack image samples, the first label image, and the second label image.

[0049] In one possible implementation, the model training module is used to construct the backbone loss function of the backbone segmentation network based on the Dice correlation coefficient between the number of pixels in the ground truth map of the first labeled image and the number of pixels in the segmentation result of the third image.

[0050] Among them, the backbone loss function L Dice for:

[0051]

[0052] X * Y represents the segmentation result of the third image, and Y represents the ground truth image of the first label image. |X * | and |Y| represent the number of pixels in the ground truth image of the first label image and the number of pixels in the segmentation result of the third image, respectively. |X * ∩Y| represents the number of pixels where the ground truth image of the first labeled image intersects with the segmentation result of the third image.

[0053] In one possible implementation, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network.

[0054] The prediction branch loss function L for each layer of the prediction branch network b for:

[0055]

[0056] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nij This is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0057] In one possible implementation, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network. The edge prediction branch network also has nn.Parameter for the adaptive weight of each feature map.

[0058] The prediction branch loss function L for each layer of the prediction branch network b for:

[0059]

[0060] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer, w k The weights of each feature map layer are W = {w1, w2, w3, w4, w5}. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nij This is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0061] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0062] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation thereof.

[0063] This invention provides a method, apparatus, device, and storage medium for concrete crack detection and identification. First, multiple crack image samples are acquired from a pre-set sample library. Then, each crack image sample is manually labeled to obtain a first-labeled image containing a first label. Next, edge detection is performed on the first-labeled image to obtain a second-labeled image containing a second label. Next, a pre-constructed crack detection and identification model is trained based on each crack image sample and its first and second-labeled images to obtain a trained crack detection and identification model. Finally, crack detection and identification are performed on the image to be detected based on the trained crack detection and identification model. The crack detection and identification model constructed in this invention, in addition to having a backbone segmentation network to extract all features of each crack image sample, also has an edge prediction branch network to extract edge crack features of the manually labeled crack image samples. This allows the detection and identification model constructed in this invention to not only learn all crack features but also focus on the edge features of the cracks, thereby improving the accuracy of the crack detection and identification model in detecting and identifying minute cracks. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating the implementation of the concrete crack detection and identification method provided in this embodiment of the invention.

[0066] Figure 2 This is the Sobel edge detection template provided in this embodiment of the invention;

[0067] Figure 3 This is a schematic diagram illustrating the training principle of the crack detection and recognition model provided in this embodiment of the invention;

[0068] Figure 4 This is a schematic diagram of the concrete crack detection and identification device provided in an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0070] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0071] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0072] Crack detection and identification is a crucial aspect of durability assessment for concrete structures. While existing end-to-end deep neural networks can learn some discriminative crack features, these networks still tend to fit less important features, failing to reinforce useful features (such as crack edges) during training. This can lead to localized model performance degradation, where the model collapses to poorly performing feature layers, resulting in missed or false alarms. Furthermore, the repeated downsampling operations in deep learning networks can cause the loss of edge features from minute cracks, also leading to missed detections of these fine cracks.

[0073] To address the problems of existing technologies, embodiments of the present invention provide a method, apparatus, device, and storage medium for detecting and identifying concrete cracks. The concrete crack detection and identification method provided by the embodiments of the present invention will be described first.

[0074] The entity executing the concrete crack detection and identification method can be a concrete crack detection and identification device, which can be an electronic device with a processor and memory, such as a mobile electronic device or a non-mobile electronic device. This invention does not impose specific limitations on the embodiments.

[0075] See Figure 1 The flowchart illustrating the implementation of the concrete crack detection and identification method provided in this embodiment of the invention is described in detail below:

[0076] Step S110: Obtain multiple crack image samples from the preset sample library.

[0077] In practice, a pre-built sample library needs to be constructed to train the pre-built crack detection and recognition model. This library includes multiple crack image samples containing cracks. The cracks in these samples vary in size, allowing for better training of the pre-built crack detection and recognition model. The crack images can be images of cracks in concrete.

[0078] Step S120: Manually label each crack image sample to obtain a first label image containing the first label.

[0079] The first marker indicates a crack in the image.

[0080] By manually labeling each crack image sample first, a first labeled image containing the cracks in the indicator image can be obtained, which can be a black and white ground truth map containing the cracks in the indicator image.

[0081] Step S130: Perform edge detection on the first label image to obtain a second label image containing the second label.

[0082] The second marker indicates the edge of the crack in the image.

[0083] Existing deep learning-based concrete crack detection and recognition models are prone to edge feature loss during feature encoding due to multiple downsampling operations, leading to missed detection of minute cracks and low accuracy in crack width perception. Therefore, this application uses crack edges as key target features for reinforcement learning. Specifically, edge detection is performed on the first labeled image to extract crack edge features, resulting in a second labeled image indicating the edges of cracks in the image, which serves as edge prediction and perception labeling information.

[0084] Specifically, since crack edges mainly contain two attributes—direction and amplitude—pixel brightness changes gradually along the crack boundary direction, while pixel brightness changes drastically perpendicular to the crack boundary direction. Therefore, the Sobel edge detection operator can be used to detect the ground truth map of crack edges in the first labeled image to obtain the second labeled image.

[0085] like Figure 2 As shown, the Sobel operator provides vertical ( Figure 2 (b) and level ( Figure 2 (c) Detection templates in two directions. The vertical detection template can detect vertical edges in the first label image, while the horizontal detection template can detect horizontal edges in the first label image. Let... Figure 2 (a) is the 3×3 first label image to be detected. The gradient values ​​of the vertical and horizontal directions of each point in the first label image can be calculated by the following formula. Then, the overall gradient magnitude of the pixel can be calculated by the last formula. Thus, it can be determined whether the pixel belongs to the edge of the crack image sample.

[0086] G y =(Z3+2Z6+Z9)-(Z1+2Z4+Z7);

[0087] G x=(Z7+2Z8+Z9)-(Z1+2Z2+Z3);

[0088] |G|=|G x |+G y |

[0089] Step S140: Based on each crack image sample and its first label image and second label image, train the pre-constructed crack detection and recognition model to obtain the trained crack detection and recognition model.

[0090] The crack detection and recognition model is based on channel convolution operations, which divides the Unet++ decoding network into a backbone segmentation network and an edge prediction branch network. The backbone segmentation network is used to extract all features of each crack image sample, and the edge prediction branch network is used to extract the edge crack features of the first label image.

[0091] like Figure 3 As shown, this application uses Unet++ as the main framework, and divides the original Unet++ segmentation network into two branches using channel convolution. One branch continues the segmentation of the main crack region; the other branch utilizes the aggregation context function of the Unet++ network decoder to construct an end-to-end edge prediction branch network. The two branch networks together form a multi-task learning system, enabling the entire network to focus on crack edge information while learning the main features, thereby improving the detection and identification accuracy of subtle crack defects.

[0092] Among them, the Unet++ network is a semantic segmentation model proposed based on the original Unet network. Because the skip connection layer used in the decoder network of the Unet++ network can aggregate features of different scales, it can form a highly flexible feature fusion framework. The crack segmentation backbone network based on the Unet++ network is used to extract multi-scale features of cracks, thereby improving the ability to perceive crack defects.

[0093] To improve the detection performance of crack detection and recognition models for minor crack defects, based on pixel segmentation of the crack backbone region, an edge feature-assisted detection branch network is constructed by further utilizing the crack edge features unique to crack defects. This branch network is then integrated with the backbone network into a single deep learning model.

[0094] The crack detection network Unet++ mainly consists of an encoder and a decoder. In the encoder, the network obtains feature maps of different resolutions through convolution. In the decoder, the network fuses its own features with the encoder's feature maps via skip connections, constructing a pyramid with multi-resolution features. A crack edge prediction branch network is built on top of the Unet++ network's decoder.

[0095] Because deep learning-based concrete crack detection and recognition models are prone to edge feature loss during feature encoding due to multiple downsampling operations, they suffer from missed detection of minute cracks and low accuracy in crack width perception. This performance is not significant in training sample images with wider cracks, but requires special attention for training samples with narrower cracks. Therefore, to achieve differentiated training, weight parameters for different training sample images are established based on the different crack widths, thereby enabling the crack detection and recognition model to learn more effectively during training.

[0096] However, considering the significant randomness of crack occurrence, it is difficult to directly statistically calculate the crack width parameter. Therefore, this application employs an indirect crack width evaluation operator and designs a dynamic regularized parameter training weight calculation method accordingly.

[0097] The training weights of the crack image samples corresponding to the first label image are determined based on the crack area in the first label image and the crack edge area in the second label image.

[0098] Specifically, let Y be the ground truth image of the first labeled image of a crack image sample, which is manually labeled. Using the Sobel edge detection operator, the ground truth image of the second labeled image of the crack edge of the crack image sample can be obtained as Y′. The crack area Q in the ground truth image Y of the first labeled image and the crack edge area P in the ground truth image Y′ of the second labeled image can be calculated sequentially. The training weights of the crack image sample can be calculated according to the following formula:

[0099] λ = 0.1 × P / Q;

[0100] Generally speaking, the area of ​​the crack edge is smaller than the crack area; the finer the crack, the closer the value is to 0.1, and vice versa; this can indirectly measure the crack width, promote the formation of the entire network model to learn the differences of crack samples with different widths, and thus more accurately establish a crack detection and recognition model.

[0101] In order to constrain the pre-built crack detection and identification model, it is necessary to construct the backbone loss function of the backbone segmentation network and the prediction branch loss function of each layer of the edge prediction branch network.

[0102] First, based on the ground truth map of the first labeled image and the segmentation results of the third image, the backbone loss function of the backbone segmentation network is constructed.

[0103] The third image is the image obtained by passing the crack image sample corresponding to the first label image through the backbone segmentation network.

[0104] Specifically, the backbone loss function of the backbone segmentation network is constructed based on the Dice correlation coefficient between the number of pixels in the ground truth map of the first labeled image and the number of pixels in the segmentation result of the third image.

[0105] Among them, the backbone loss function L Dice for:

[0106]

[0107] X * Y represents the segmentation result of the third image, and Y represents the ground truth image of the first label image. |X * | and |Y| represent the number of pixels in the ground truth image of the first label image and the number of pixels in the segmentation result of the third image, respectively. |X * ∩Y| represents the number of pixels where the ground truth image of the first labeled image intersects with the segmentation result of the third image, and ε is a tiny value set to prevent the denominator from being 0 in the Dice correlation coefficient calculation.

[0108] The prediction branch loss function for each layer of the prediction branch network is constructed based on the cross-entropy between the ground truth map of the second labeled image and the fourth image, as well as the training weights of the crack image samples; wherein, the fourth image is the feature map obtained by passing the crack image sample corresponding to the first labeled image through each layer of the prediction branch network.

[0109] Specifically, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate two sub-decoding networks: a backbone segmentation network and an edge prediction branch network. To facilitate loss function calculation with the original resolution image labels, the feature maps corresponding to all sub-decoding networks are upsampled to the original image resolution.

[0110] The prediction branch loss function L for each layer of the prediction branch network b for:

[0111]

[0112] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nijThis is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0113] To account for the differences in information fed back from the feature maps of each decoding network layer, a feature map weight parameter W = {w1, w2, w3, w4, w5} is introduced. In addition, nn.Parameter is set in the edge prediction branch network for the adaptive weight of each feature map layer.

[0114] The prediction branch loss function L for each layer of the prediction branch network b for:

[0115]

[0116] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer, w k The weights for each feature map layer are... Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nij This is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0117] The loss function L of the crack detection and identification model is:

[0118] L = L Dice +L b ;

[0119] During the training process of the crack detection and recognition model, the loss function L will become smaller and smaller until the model converges, thus obtaining the trained crack detection and recognition model.

[0120] The loss functions of the backbone segmentation network and the edge prediction branch network are jointly trained. The joint training framework uses the regularized training weight parameters of each training sample to constrain the loss function of the edge prediction branch network, thereby forming a differential perception learning of crack image samples of different widths, thus improving the comprehensive perception accuracy of the algorithm for crack diseases.

[0121] Step S150: Based on the trained crack detection and recognition model, perform crack detection and recognition on the image to be detected.

[0122] Once the crack detection and recognition model is trained, it can be used to detect and recognize cracks in the image to be detected.

[0123] When the image to be detected is input into the trained crack detection and recognition model, it outputs a labeled ground truth image of the crack obtained after the image is processed by the backbone segmentation network, and a labeled crack edge image obtained after the image is processed by the edge prediction branch network. At this point, depending on the needs of the scenario, both types of output images can be used simultaneously, or only the labeled ground truth image of the crack obtained after the backbone segmentation network can be used to detect and recognize cracks.

[0124] Of course, depending on the needs of the scenario, the labeled ground truth image of the crack obtained after passing through the backbone segmentation network and the labeled edge image of the crack obtained after passing through the edge prediction branch network can be processed to obtain a fused result, which makes it easier for users to detect cracks.

[0125] By selecting a certain number of images of fine cracks on concrete surfaces to test the trained crack detection and recognition model, it is shown that after introducing the edge prediction branch network, this application achieves better crack detection and recognition results than the original Unet and Unet++ networks.

[0126] The present invention provides a concrete crack detection and identification method, apparatus, device, and storage medium. First, multiple crack image samples are acquired from a pre-set sample library. Then, each crack image sample is manually labeled to obtain a first-labeled image containing a first label. Next, edge detection is performed on the first-labeled image to obtain a second-labeled image containing a second label. Next, a pre-constructed crack detection and identification model is trained based on each crack image sample and its first and second-labeled images to obtain a trained crack detection and identification model. Finally, crack detection and identification are performed on the image to be detected based on the trained crack detection and identification model. The crack detection and identification model constructed in this invention, in addition to having a backbone segmentation network for extracting all features of each crack image sample, also has an edge prediction branch network for extracting edge crack features of the manually labeled crack image samples. This allows the detection and identification model constructed in this invention to not only learn all crack features but also focus on the edge features of the cracks, thereby improving the accuracy of the crack detection and identification model in detecting and identifying minute cracks.

[0127] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0128] Based on the concrete crack detection and identification method provided in the above embodiments, the present invention also provides a specific implementation of a concrete crack detection and identification device applied to the concrete crack detection and identification method. Please refer to the following embodiments.

[0129] like Figure 4 As shown, a concrete crack detection and identification device 400 is provided, the device comprising:

[0130] The acquisition module 410 is used to acquire multiple crack image samples from a preset sample library;

[0131] The labeling module 420 is used to manually label each crack image sample to obtain a first label image containing a first label, wherein the first label indicates the crack in the image;

[0132] Edge detection module 430 is used to perform edge detection on the first label image to obtain a second label image containing a second mark, wherein the second mark indicates the edge of the crack in the image;

[0133] The model training module 440 is used to train the pre-built crack detection and recognition model based on each crack image sample and its first label image and second label image to obtain the trained crack detection and recognition model. The crack detection and recognition model is based on channel convolution operation, which divides the Unet++ decoding network into a backbone segmentation network and an edge prediction branch network. The backbone segmentation network is used to extract all features of each crack image sample, and the edge prediction branch network is used to extract the edge crack features of the first label image.

[0134] The detection and recognition module 450 is used to detect and recognize cracks in the image to be detected based on the trained crack detection and recognition model.

[0135] In one possible implementation, the model training module 440 is used to determine the training weights of the crack image samples corresponding to the first label image based on the crack area in the first label image and the edge area of ​​the crack in the second label image.

[0136] The pre-built crack detection and recognition model is trained based on each crack image sample, its first label image, its second label image, and the training weights of each crack image sample.

[0137] In one possible implementation, the training weight λ of the crack image sample corresponding to the first labeled image is:

[0138] λ = 0.1 × P / Q;

[0139] Where P is the edge area of ​​the crack in the second label image, and Q is the crack area of ​​the crack in the first label image.

[0140] In one possible implementation, the model training module 440 is used to construct the backbone loss function of the backbone segmentation network based on the ground truth map of the first label image and the segmentation result of the third image; wherein, the third image is the image obtained after the crack image sample corresponding to the first label image passes through the backbone segmentation network;

[0141] The prediction branch loss function of each prediction branch network is constructed based on the cross-entropy of the ground truth map of the second label image and the fourth image, as well as the training weights of the crack image samples; wherein, the fourth image is the feature map obtained by passing the crack image sample corresponding to the first label image through each prediction branch network.

[0142] The pre-built crack detection and recognition model is trained based on the backbone loss function, the prediction branch loss function, crack image samples, the first label image, and the second label image.

[0143] In one possible implementation, the model training module 440 is used to construct the backbone loss function of the backbone segmentation network based on the Dice correlation coefficient between the number of pixels in the ground truth map of the first labeled image and the number of pixels in the segmentation result of the third image.

[0144] Among them, the backbone loss function L Dice for:

[0145]

[0146] X * Y represents the segmentation result of the third image, and Y represents the ground truth image of the first label image. |X * | and |Y| represent the number of pixels in the ground truth image of the first label image and the number of pixels in the segmentation result of the third image, respectively. |X * ∩Y| represents the number of pixels where the ground truth image of the first labeled image intersects with the segmentation result of the third image.

[0147] In one possible implementation, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network.

[0148] The prediction branch loss function L for each layer of the prediction branch network b for:

[0149]

[0150] Where, λ nLet be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nij This is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0151] In one possible implementation, Unet++ is a 5-layer network. The feature pyramid of the Unet++ decoding network outputs 5 layers of feature maps. Each feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network. The edge prediction branch network also has nn.Parameter for the adaptive weight of each feature map.

[0152] The prediction branch loss function L for each layer of the prediction branch network b for:

[0153]

[0154] Where, λ n Let be the training weights for the nth image of the crack image samples. The feature map of the k-th layer is obtained by passing the n-th image from the crack image samples through the k-th layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer, w k The weights of each feature map layer are W = {w1, w2, w3, w4, w5}. Let y′ be the pixel prediction value at each location of the feature map at the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network. nij This is the ground truth map for each position in the second label image matrix of the nth image in the crack image sample.

[0155] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 5As shown, the electronic device 5 in this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, it implements the steps in the various concrete crack detection and identification method embodiments described above, for example... Figure 1 Steps 110 to 150 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module in the above-described device embodiments, for example... Figure 4 The functions of modules 410 to 450 are shown.

[0156] For example, the computer program 52 can be divided into one or more modules, which are stored in the memory 51 and executed by the processor 50 to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 52 in the electronic device 5. For example, the computer program 52 can be divided into... Figure 4 Modules 410 to 450 are shown.

[0157] The electronic device 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0158] The processor 50 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] The memory 51 can be an internal storage unit of the electronic device 5, such as a hard disk or memory. The memory 51 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 51 can include both internal and external storage units of the electronic device 5. The memory 51 is used to store the computer program and other programs and data required by the electronic device. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0163] In the embodiments provided by this invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0166] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various concrete crack detection and identification method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0167] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting and identifying concrete cracks, characterized in that, include: Obtain multiple crack image samples from a preset sample library; Each crack image sample is manually labeled to obtain a first labeled image containing a first label, wherein the first label indicates the crack in the image; Edge detection is performed on the first label image to obtain a second label image containing a second marker, wherein the second marker indicates the edge of the crack in the image; the edge of the crack includes two attributes: direction and magnitude, and the second label image is obtained by detecting the ground truth map of the crack edge of the first label image based on the Sobel edge detection operator; Based on each crack image sample and its first label image and second label image, a pre-constructed crack detection and recognition model is trained to obtain a trained crack detection and recognition model; wherein, the crack detection and recognition model is based on channel convolution operation to divide the Unet++ decoding network into a backbone segmentation network and an edge prediction branch network. The backbone segmentation network is used to extract all features of each crack image sample, and the edge prediction branch network is used to extract edge crack features of the first label image. Based on the trained crack detection and recognition model, crack detection and recognition are performed on the image to be detected; The step of training a pre-built crack detection and recognition model based on each crack image sample and its first label image and second label image to obtain a trained crack detection and recognition model further includes: Based on the crack area in the first label image and the edge area of ​​the crack in the second label image, the training weights of the crack image samples corresponding to the first label image are determined. These training weights are used to characterize the crack width of the crack image samples corresponding to the first label image. Based on the Dice correlation coefficient between the number of pixels in the ground truth map of the first labeled image and the number of pixels in the segmentation result of the third image, the backbone loss function of the backbone segmentation network is constructed; wherein, the third image is the image obtained after the crack image sample corresponding to the first labeled image passes through the backbone segmentation network; The prediction branch loss function of each layer of the edge prediction branch network is constructed based on the cross-entropy of the ground truth map of the second label image and the fourth image, as well as the training weights of the crack image samples; wherein, the fourth image is the feature map obtained by passing the crack image sample corresponding to the first label image through each layer of the prediction branch network; wherein, the feature pyramid output of the Unet++ decoding network outputs 5 layers of feature maps, and each layer of feature map is processed by channel convolution to generate 2 sub-decoding networks, namely the backbone segmentation network and the edge prediction branch network, and the edge prediction branch network is further provided with nn.Parameter for the weight adaptation of each layer of feature map; The pre-built crack detection and recognition model is trained based on the main loss function, the prediction branch loss function, the crack image samples, the first labeled image, and the second labeled image; Wherein, the training weight λ of the crack image sample corresponding to the first label image is: λ=0.1×P / Q； Where P is the edge area of ​​the crack in the second label image, and Q is the crack area of ​​the crack in the first label image.

2. The detection and identification method as described in claim 1, characterized in that, The backbone loss function for: ； Y represents the segmentation result of the third image, and Y represents the ground truth image of the first label image. and These are the number of pixels in the ground truth image of the first labeled image and the number of pixels in the segmentation result of the third image, respectively. The number of pixels where the ground truth map of the first labeled image intersects with the segmentation result of the third image.

3. The detection and identification method as described in claim 1, characterized in that, Unet++ is a 5-layer network; The prediction branch loss function L of each layer of the edge prediction branch network. b for: ； Where, λ n The training weights for the nth image of the crack image samples are... The feature map of the kth layer is obtained by passing the nth image in the crack image sample through the kth layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer. For each position of the pixel prediction value in the feature map of the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network, it is defined as: The ground truth map for each position of the second label image matrix of the nth image in the crack image sample.

4. The detection and identification method as described in claim 1, characterized in that, Unet++ is a 5-layer network. The prediction branch loss function L of each layer of the edge prediction branch network. b for: Where, λ n The training weights for the nth image of the crack image samples are... The feature map of the kth layer is obtained by passing the nth image in the crack image sample through the kth layer prediction branch network. Let i and j be the second label image matrix corresponding to the nth image of the crack image sample, respectively. The rows and columns in the array, 0≤i≤M, 0≤j≤N, 1≤k≤5, where n is a positive integer. w k The weights of each feature map layer are W={ w 1, w 2, w 3, w 4, w 5}, For each position of the pixel prediction value in the feature map of the k-th layer obtained by passing the n-th image in the crack image sample through the k-th layer prediction branch network, it is defined as: The ground truth map for each position of the second label image matrix of the nth image in the crack image sample.

5. A concrete crack detection and identification device, characterized in that, include: The acquisition module is used to acquire multiple crack image samples from a preset sample library; A labeling module is used to manually label each crack image sample to obtain a first labeled image containing a first label, wherein the first label indicates the crack in the image; An edge detection module is used to perform edge detection on the first label image to obtain a second label image containing a second marker, wherein the second marker indicates the edge of a crack in the image; The model training module is used to train a pre-built crack detection and recognition model based on each crack image sample and its first label image and second label image to obtain a trained crack detection and recognition model. The crack detection and recognition model is based on channel convolution operation, which divides the Unet++ decoding network into a backbone segmentation network and an edge prediction branch network. The backbone segmentation network is used to extract all features of each crack image sample, and the edge prediction branch network is used to extract edge crack features of the first label image. The detection and recognition module is used to detect and recognize cracks in the image to be detected based on the trained crack detection and recognition model. The model training module is also used to determine the training weights of the crack image samples corresponding to the first label image based on the crack area in the first label image and the edge area of ​​the crack in the second label image. Based on the ground truth map of the first labeled image and the segmentation result of the third image, the backbone loss function of the backbone segmentation network is constructed; wherein, the third image is the image obtained after the crack image sample corresponding to the first labeled image passes through the backbone segmentation network; Based on the cross-entropy of the ground truth map of the second labeled image and the fourth image, and the training weights of the crack image samples, a prediction branch loss function is constructed for each layer of the edge prediction branch network; wherein, the fourth image is the feature map obtained by passing the crack image sample corresponding to the first labeled image through each layer of the prediction branch network. The pre-built crack detection and recognition model is trained based on the main loss function, the prediction branch loss function, the crack image samples, the first labeled image, and the second labeled image; Wherein, the training weight λ of the crack image sample corresponding to the first label image is: λ=0.1×P / Q； Where P is the edge area of ​​the crack in the second label image, and Q is the crack area of ​​the crack in the first label image.

6. An electronic device, characterized in that, The method includes a memory and a processor, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images