A two-stage crack detection method based on attention mechanism

Abstract

The application discloses a two-stage crack detection method based on an attention mechanism, and belongs to the technical field of disease identification and computer vision. A platform carrying a lightweight target segmentation algorithm module combined with an attention mechanism is used to shoot a contaminated area of an infrastructure surface that may have cracks, and the area is processed into a picture as input of the lightweight target segmentation algorithm module of the attention mechanism. Features of the area that may have cracks are extracted through the algorithm, and other pixel interference is excluded. To some extent, resource waste caused by the algorithm detecting areas that cannot have cracks in the image is avoided, and false alarm rate is reduced. Subsequently, the processed sub-picture is sent to a computer carrying a refined crack detection algorithm. The algorithm extracts information such as the position and size of cracks in the sub-picture through a high-level network structure and a multi-task learning framework, and higher crack detection accuracy is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of disease identification and computer vision, and specifically relates to a two-stage crack detection method based on an attention mechanism. BACKGROUND

[0002] Cracks are one of the typical diseases commonly existing in many infrastructures at present, and their forms are various, including surface cracks, through cracks and diagonal cracks. The features of these cracks presented in two-dimensional images are not obvious, and lack of diverse geometric information, so the detection of cracks existing in infrastructures has always been a difficulty in engineering. At the same time, the existence of cracks will bring unpredictable risks to the service life and safety of related facilities, and may cause serious safety accidents and great economic losses. In different fields, such as building structure engineering, geological engineering, oil and gas exploration and development, bridge disease maintenance and industrial facility maintenance, crack detection plays a crucial role.

[0003] Current crack detection methods mainly include model-based methods and deep learning-based methods. The model-based method mainly relies on modeling and analysis of known crack forms, and is suitable for specific scenarios and targets. However, due to the diversity and complexity of crack forms, the application range of this method is limited. The deep learning-based method learns the features and patterns of cracks by training a large number of crack image data, and has stronger generalization ability and higher detection accuracy. However, current deep learning detection methods generally set only one stage to complete the crack detection task, i.e. using a single algorithm to perform feature extraction and prediction operations on each region of the target image. This method ensures that each possible crack region in the image is detected to some extent, but also causes waste of computing resources. At the same time, in terms of infrastructure, since cracks in the real world generally exist in a few fixed types of facilities, the algorithm may identify some contaminants in regions where cracks are unlikely to appear as cracks, resulting in a high false alarm rate of the detection result. SUMMARY

[0004] In view of the problem of excessive waste of computing resources and low detection accuracy of traditional deep learning algorithms in performing crack detection tasks of infrastructures, the present application proposes a two-stage crack detection method based on an attention mechanism according to the certainty of possible crack regions, combined with an attention mechanism. The two-stage detection method is adopted, the target is segmented in the first stage, and the spatial attention mechanism is used to overcome the phenomena of translation, rotation and distortion of the region to be detected in the picture, so as to accurately carry out the segmentation of the region to be detected. In the second stage, high-precision crack detection is further realized in the segmented region, the region where cracks may exist is identified and preferentially processed, and the calculation and misjudgment of irrelevant regions are reduced, thereby reducing the waste of computing resources in the crack detection task and improving the accuracy of crack detection.

[0005] The two-stage crack detection method based on the attention mechanism comprises the following specific steps:

[0006] Step one: train a lightweight convolutional neural network using crack region data to obtain a semantic segmentation model for segmenting crack regions from images. Meanwhile, train an advanced convolutional neural network using the crack region dataset to obtain a target detection model for detecting cracks.

[0007] Step two: configure the trained semantic segmentation model and the target detection model in the lightweight target segmentation algorithm module and the fine crack detection algorithm module based on the attention mechanism.

[0008] Step three: take pictures of infrastructure-related areas that may have cracks to collect inspection images.

[0009] Step four: determine whether the shooting task is completed. If not, return to step three; if yes, proceed to step five.

[0010] Step five: export the collected inspection images to the lightweight target segmentation algorithm module combined with the attention mechanism for storage. Further, perform image segmentation on the stored inspection images through the lightweight target segmentation algorithm module based on the attention mechanism to obtain region images that may contain cracks, specifically:

[0011] (1) correct the image through the spatial attention mechanism.

[0012] (2) extract features from the input corrected image using the semantic segmentation model, and perform weighted processing on the extracted features. Through the attention mechanism, assign weights to each region.

[0013] (3) perform fine segmentation based on the weighted feature image to obtain several regions of interest; and set the pixel values of all pixel points outside the region to zero.

[0014] (4) crop and delete the zeroed region in the segmented image, crop and fill the remaining region and the region of interest image, and obtain a sub-image.

[0015] Step six: transfer the obtained sub-image to the fine crack detection algorithm module for storage.

[0016] Step seven: the input several subgraphs are detected one by one through the fine crack detection algorithm module to obtain the predicted subgraph after each subgraph is detected by the fine crack detection algorithm, wherein the coordinates and sizes of each recognized crack detection frame are contained, and the specific steps are as follows: the fine crack detection algorithm module extracts features of the subgraph through a target detection model; a multi-task learning framework is used to decompose the feature extraction task into multiple subtasks, for each subtask, a high-level convolutional neural network branch is used for processing, and multi-task learning is performed on the basis of shared feature extraction.

[0017] Step eight: check whether all subgraphs have been detected, if not, return to step seven: if yes, group the subgraphs to be spliced according to the source of the original image, then splice the images, rely on the coordinates of the subgraph in the original image and the coordinates and size of the crack detection frame in the subgraph, and map the crack recognition result to the original image position, and each subgraph spliced together must come from the same original image, to obtain a predicted image.

[0018] The advantages of the present application are:

[0019] 1. The two-stage crack detection method based on the attention mechanism of the present application divides the traditional single-stage crack detection process into two stages, first finds out the region of interest in the image where the crack may exist, and then performs high-precision crack detection on the region, thereby reducing the false alarm rate of crack detection and improving the detection accuracy.

[0020] 2. Compared with the traditional crack detection algorithm, the two-stage crack detection method based on the attention mechanism can automatically focus on the key region where the crack may exist, to a certain extent, avoiding the waste of resources caused by detecting the region where the crack is impossible in the image, and reducing the false alarm rate.

[0021] 3. Compared with the traditional crack detection algorithm, the fine crack detection algorithm used in the two-stage crack detection method based on the attention mechanism realizes higher crack detection accuracy through a high-level network structure and a multi-task learning framework.

[0022] 4. Compared with the traditional crack detection method, the two-stage crack detection method based on the attention mechanism reduces unnecessary calculation, reduces the false alarm rate of crack detection results, and improves the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flowchart of the two-stage crack detection method based on the attention mechanism of the present application;

[0024] Figure 2 The flowchart of the first stage segmentation algorithm;

[0025] Figure 3 is a flow chart of the second stage detection algorithm. DETAILED DESCRIPTION

[0026] The application will be further described in detail below with reference to the accompanying drawings.

[0027] To implement the two-stage crack detection method based on the attention mechanism of the present application, the following are required:

[0028] 1) A platform for carrying a lightweight target segmentation algorithm module combined with an attention mechanism, such as: a drone or a device for monitoring wall cracks near a building; at the same time, the platform also carries a reliable data transmission device.

[0029] 2) A camera mounted on the platform in 1) responsible for shooting the contaminated area of the infrastructure surface that may have cracks, and processing it into a picture; the picture taken by the camera is used as the input of the lightweight target segmentation algorithm module of the attention mechanism. The camera should also include a picture storage module.

[0030] 3) A high-performance computer for carrying a refined crack detection algorithm; the computer should be equipped with a GPU and a storage device.

[0031] Based on the above-mentioned equipment, the two-stage crack detection method based on the attention mechanism of the present application, as shown in Figure 1 , the specific steps are as follows:

[0032] Step 1: Train a lightweight convolutional neural network MobileNet using crack area data to obtain a semantic segmentation model for segmenting crack areas from images; at the same time, train a high-level convolutional neural network ResNet using the crack area dataset to obtain a target detection model for detecting cracks.

[0033] During the model training process, in order to effectively apply weight decay regularization, thereby improving the optimization performance and better controlling the weight decay, the present application selects AdamW (Adam+weight decate) as the optimizer, the initial learning rate is 0.002, the periodic learning rate is 0.01, the momentum is 0.937, and the weight decay is 0.0005. Each training process contains 100 epochs, and the batch is set to 32.

[0034] The above-mentioned crack area data is composed of infrastructure images and their corresponding label files. Among them, the infrastructure images are the structure surface area images of various types of infrastructure that may contain cracks; the label file mainly records the label value of the area that may contain cracks corresponding to each image.

[0035] Step two: configure the trained semantic segmentation model into the lightweight target segmentation algorithm based on attention mechanism (hereinafter referred to as the first stage segmentation algorithm) and the fine crack detection algorithm (hereinafter referred to as the second stage detection algorithm).

[0036] Step three: take pictures of the infrastructure-related area that may have cracks by the camera, and collect inspection images.

[0037] Step four: determine whether the unmanned aerial vehicle shooting task is completed. If not, return to step three; if yes, proceed to step five.

[0038] Step five: automatically export the collected inspection images to the storage area of the lightweight target segmentation algorithm module combined with attention mechanism.

[0039] Further, the first stage detection algorithm carried by the mobile platform uniformly performs image segmentation on the stored inspection images to obtain images of the region that may contain cracks (hereinafter referred to as the region of interest).

[0040] First, correct the image by spatial attention mechanism to remove translation, rotation, distortion and other interference.

[0041] (2) Use the lightweight convolutional neural network MobileNet (i.e. the semantic segmentation model trained in step 1) to extract features from the input corrected image. These features include not only the direct visual features of the crack, but also the context information around it. At the same time, the algorithm carries the Self-Attention in Transformer, which weights the extracted features. Through the attention mechanism, the algorithm learns which areas are more important for crack detection, and thus gives these areas greater weight.

[0042] The above-mentioned lightweight algorithm framework is not limited to using MobileNet, but also can be U-Net, FCN (Fully Convolutional Networks) and the like. At the same time, the attention mechanism carried by the algorithm is not limited to using Self-Attention, and the fine segmentation is not limited to using CRF.

[0043] (3) Based on the weighted features, use Conditional Random Field (CRF) to perform fine segmentation on the image to obtain several regions of interest. In this process, the threshold R is set according to prior knowledge crack When the confidence of the region in the image that may contain cracks (given by the result of MobileNet semantic segmentation, i.e. the confidence of the region of interest and its prediction in the segmentation result) is greater than the threshold, that is, At this time, the pixel values of all pixel points outside this region are set to zero. Wherein, object is the label of different regions segmented in the image, img(i) is the i-th image photographed by the unmanned aerial vehicle in time sequence.

[0044] (4) After post-processing such as noise removal and boundary optimization on the segmented image, the zeroed region in the image is cropped and deleted, the remaining region and the image of the region of interest are cropped to an image with a resolution of 100x100, and a padding operation is performed to obtain a sub-image. The size of the above cropped sub-image is related to the data processing capacity of the downstream algorithm to some extent, so the size of the sub-image is not limited to 100x100, and any size of the cropped sub-image is applicable to the two-stage crack detection method based on the attention mechanism.

[0045] In summary, through the lightweight network and the attention mechanism, the algorithm greatly reduces the parameter quantity, reduces the model size, and improves the deployment efficiency.

[0046] In order to accurately align the subsequent fine crack detection algorithm to the original image, the positions of each sub-image in the original image need to be recorded for use in post-processing. Considering that the sizes of all sub-images are the same, special points (sub-image top-left corner points or sub-image center points) are used to represent the positions of each sub-image in the original image.

[0047] Step six: the unmanned aerial vehicle transmits the sub-image to the ground station through a reliable link, and the ground station stores the received sub-image in the processing area.

[0048] Step seven: input several sub-images one by one through the fine crack detection algorithm module to obtain the detection result on the sub-image, and then step eight is implemented.

[0049] The sub-image to be detected is input into the second stage detection algorithm for processing to obtain a predicted sub-image after the fine crack detection algorithm is detected, which contains the coordinates and sizes of each identified crack detection frame; the flow chart of the second stage detection algorithm is as shown in Figure 3 , specifically:

[0050] The algorithm uses a high-level convolutional neural network ResNet (i.e. the target detection model obtained by step 1) to extract features from the subgraph. The present application uses a multi-task learning framework to decompose the crack detection task (i.e. the feature extraction task) into multiple subtasks, such as crack positioning, length and width estimation, etc. For each subtask, an independent high-level convolutional neural network ResNet branch is used for processing, and multi-task learning is performed based on shared feature extraction. The above deep learning algorithm includes but is not limited to using ResNet, and deep learning algorithms such as Faster R-CNN, CenterNet, DETR are also applicable to the two-stage crack detection method based on the attention mechanism described in the present application.

[0051] Step eight: check whether all subgraphs have been detected, if not, return to step seven: if yes, group the subgraphs to be spliced according to the source of the original image, then perform image splicing, rely on the coordinates of the subgraph in the original image and the coordinates and size of the crack detection box in the subgraph, etc. Information, map the crack recognition result to the original image position, and splice the subgraphs together. Each subgraph must come from the same original image, and the predicted image is obtained. The predicted image is in the form of: in the original image, all detected cracks are framed with a prediction box, and the corresponding crack prediction confidence, size and other information are displayed.

[0052] The method described in the present application emphasizes but is not limited to crack detection in general infrastructure areas. Any entity to be detected and its constituent areas can use the method described in the present application for crack detection, and are applicable to the two-stage crack detection method based on the attention mechanism described in the present application.

Claims

1. A two-stage crack detection method based on attention mechanism, characterized by: Step one: training a lightweight convolutional neural network using crack region data to obtain a semantic segmentation model for segmenting crack regions from images; simultaneously training an advanced convolutional neural network using a crack region dataset to obtain a target detection model for detecting cracks; Step two: configuring the trained semantic segmentation model and target detection model in the lightweight target segmentation algorithm module based on attention mechanism and the fine crack detection algorithm module; Step three: taking pictures of infrastructure-related areas that may have cracks to collect inspection images; Step four: determining whether the shooting task is complete, and if not, returning to step three; if so, proceeding to step five; Step five: exporting the collected inspection images to the lightweight target segmentation algorithm module combined with attention mechanism for storage; further, uniformly segmenting the stored inspection images through the lightweight target segmentation algorithm module based on attention mechanism to obtain region images that may contain cracks, specifically: (1) correcting the image through a spatial attention mechanism; (2) extracting features from the corrected image input using the semantic segmentation model and weighting the extracted features, assigning weights to each region through the attention mechanism; (3) performing fine segmentation based on the weighted feature image to obtain several regions of interest; and setting the pixel values of all pixel points outside the region to zero; (4) cropping and deleting the zeroed region in the segmented image, cropping and padding the remaining region and the region of interest image to obtain a sub-image; Step six: transmitting the obtained sub-image to the fine crack detection algorithm module for storage; Step seven: inputting several sub-images one by one through the fine crack detection algorithm module to obtain a predicted sub-image for each sub-image after fine crack detection algorithm detection, which contains the coordinates and dimensions of each recognized crack detection box, specifically: the fine crack detection algorithm module extracts features from the sub-image through the target detection model; adopts a multi-task learning framework to decompose the feature extraction task into multiple sub-tasks, processes each sub-task using an advanced convolutional neural network branch, and performs multi-task learning based on shared feature extraction; Step eight: checking whether all sub-images have been detected, if not, returning to step seven; if so, grouping the sub-images to be spliced according to the source of the original image, then performing image splicing, mapping the crack recognition results to the original image location relying on the coordinates of the sub-image in the original image and the coordinates and dimensions of the crack detection box in the sub-image, and splicing the sub-images together to obtain a predicted image.

2. The two-stage crack detection method based on attention mechanism according to claim 1, wherein: After step five, express the position of each sub-image in the original image using the coordinates of the top-left corner of the sub-image or the center point of the sub-image in the original image. 3.The two-stage crack detection method based on attention mechanism of claim 1, wherein: In the model training process of step 1, AdamW is selected as the optimizer, the initial learning rate is 0.002, the periodic learning rate is 0.01, the momentum is 0.937, and the weight decay is 0.0005; each training process contains 100 epochs, and the batch is set to 32.

4. The two-stage crack detection method based on attention mechanism of claim 1, wherein: The crack area data used in step 1 is composed of infrastructure images and their corresponding label files; among them, the infrastructure images are the structure surface area images of various types of infrastructure that may contain cracks; the label file records the label value of the area that may contain cracks corresponding to each image.

5. The two-stage crack detection method based on attention mechanism according to claim 1, wherein: The predicted image obtained in step eight is in the form of: in the original image, all detected cracks are framed with a prediction box, and the corresponding crack information is displayed.

6. The two-stage crack detection method based on attention mechanism of claim 1, wherein: In step five (3), the threshold is set according to prior knowledge When the confidence of the region in the image that can contain a crack is greater than the threshold, the pixel values of all pixel points outside this region are set to zero.

7. The two-stage crack detection method based on attention mechanism of claim 1, wherein: The platform on which the lightweight target segmentation algorithm module combined with the attention mechanism is mounted has a device responsible for shooting the contaminated area of the infrastructure surface that may have cracks and processing it into a picture; the fine crack detection algorithm module is mounted in a high-performance computer.

Images