Crack image segmentation method based on multi-window high-low frequency visual transformer

Abstract

The present invention discloses a crack image segmentation method based on a multi-window high- and low-frequency visual transformer. By using semantic segmentation to detect cracks at the pixel level, an end-to-end network structure composed of an encoder and a decoder is improved. After network inference, a crack mask is directly output, and the training and inference efficiency is high. At the encoder stage, a visual transformer is used to extract high and low frequency features in parallel. The visual transformer has many advantages that convolution does not have, such as input adaptation, long-distance and high-order spatial interaction capabilities. It can capture high and low frequencies at the same time and can better capture small crack features. The multi-window mechanism can capture information on cracks of different scales, ensuring the robustness of the model. And improve the detection accuracy of the present invention. The method of the present invention can improve the crack detection accuracy while improving the overall computational efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and more particularly relates to a crack image segmentation method based on a multi-window high-low frequency visual converter. Background Art

[0002] Cracks are the most common type of structural damage in urban infrastructure. However, most countries and regions still rely on manual inspection to detect cracks in walls and pavements. This is undoubtedly time-consuming and labor-intensive, and there are many blind spots in the structure that are difficult for humans to reach, making it easy for missed detections. With the advancement of digital image processing technology, many excellent crack detection algorithms have been proposed. Furthermore, with the current surge in interest in deep learning, the use of deep neural networks for crack detection is increasingly being employed. Automating crack detection can save significant manpower and resources, while also offering higher accuracy than manual inspection.

[0003] Currently, crack detection methods can be roughly divided into two categories: one using traditional digital image processing techniques, and the other leveraging emerging deep learning technologies. Traditional methods have developed many excellent algorithms, such as edge detection algorithms such as the fast Haar transform, the Canny operator, the fast Fourier transform, and the Sobel operator. These have all been applied to crack detection. Furthermore, crack detection models based on Gabor filters have also achieved great success. Deep learning methods, on the other hand, require building a deep neural network for crack detection, which is currently the mainstream approach. Anchor-based object detection algorithms can be used to locate crack regions in images. Crack images can also be classified to identify those with cracks. Crack images can also be segmented for pixel-level detection. Currently, more people prefer using segmentation methods for crack detection. For example, DeepCrack is a segmentation network model specifically designed for crack detection. It outputs a prediction result at each layer, enabling multi-scale prediction. Cracklab, a modified version of DeepLabv3+, produces a network for crack segmentation that uses depthwise separable convolutions and dilated convolutions to reduce the number of parameters and expand the receptive field. And by pruning the convolution layers with weak correlation, the efficiency is improved. Figure 1 As shown, the dilated convolution is Figure 2 shown.

[0004] Although many crack segmentation algorithms have been proposed, they still have many shortcomings. Traditional crack detection methods are significantly affected by environmental factors such as noise and illumination, resulting in poor robustness and low detection precision and recall. However, using detection algorithms to locate crack regions, due to the characteristics of cracks, will generate many small, overlapping, redundant frames. This undoubtedly increases the computational workload and cannot accurately locate the crack region, resulting in only a single crack region. Segmentation algorithms are currently the most suitable method for crack detection, and many have been proposed. While there are many general segmentation algorithms, few specifically target cracks. For example, the DeepCrack segmentation algorithm performs predictions at each layer, resulting in significant computational time and inefficiency during training and inference. While algorithms such as the Cracklab algorithm reduce the number of convolutional layers to improve efficiency, this also reduces accuracy. Furthermore, simply stacking convolutional layers infinitely expands the receptive field, which is undoubtedly disadvantageous for detecting small cracks.

[0005] Based on this, it is necessary to design a new crack image segmentation method to improve crack detection accuracy and computational efficiency. Summary of the Invention

[0006] (1) Technical issues

[0007] To address the problems of low computational efficiency and insufficient precision in existing crack detection algorithms, the present invention provides a crack image segmentation method based on a multi-window high- and low-frequency visual transformer. This method uses an improved new visual transformer to achieve direct image input, better retaining the spatial position information of the image. The image is then divided into high-frequency and low-frequency parts for parallel processing. Multiple windows are used to calculate self-attention in the high-frequency part, and finally a fusion feature of the high- and low-frequency parts is obtained. The method is combined with a depthwise separable convolution and a depthwise separable atrous pyramid pooling module to replace the fully connected layer, thereby improving the computational efficiency of the parameters while improving the crack detection accuracy.

[0008] (2) Technical solution

[0009] The present invention provides a crack image segmentation method based on a multi-window high-low frequency visual converter, the method comprising the following steps:

[0010] Step 1: Use two serially connected depth-wise separable convolutional blocks to preliminarily learn and extract the input crack image features, and then use the obtained features Figure X The input is processed into a four-layer cascaded feature extraction backbone, where each layer includes downsampling and a multi-window high- and low-frequency self-attention visual transformer.

[0011] Step 2: Input features Figure X Divide the channel in proportion α, and the high-frequency branch will get (1-α)N h , the low-frequency branch will get αN h , N h is the number of channels;

[0012] Step 3: In the high-frequency branch, the feature map is further divided in the channel direction according to different proportions β1, β2, and β3 to obtain feature maps of three different parts X11, X12, and X13. Each part of the feature map will generate a set of query K, key Q, and value V matrices.

[0013] K1i=f ki (X1i), Q1i=f qi (X1i), V1i=f vi (X1i)

[0014] Function f xi () represents the depthwise separable convolution operation, which replaces the conventional linear layer to obtain three matrices to significantly reduce the number of parameters and calculations; and divides the feature map according to different window sizes γ1, γ2, and γ3, and then calculates the scale point multiplication self-attention separately for each window. The calculation formula is as follows:

[0015]

[0016] Where Dhi is the dimension of the hidden layer, Softmax is the normalized exponential function, i is an integer from 1 to 3, and SA11, SA12, and SA13 are obtained by calculation. Then, the three values ​​of SA11, SA12, and SA13 are concatenated in the channel direction, and high-frequency features are obtained through a projected linear layer and batch normalization.

[0017] Step 4: In the low-frequency branch, the input features are downsampled and pooled, and the low-frequency features in the low-frequency component are obtained through the projection linear layer and batch normalization;

[0018] Step 5: Connect the high-frequency features of step 3 and the low-frequency features of step 4 in the channel direction to obtain a fused feature map, which is then sent to the feature extraction backbone of the next layer;

[0019] Step 6: Based on the Atrous Pyramid Pool (ASPP), a depthwise separable Atrous Pyramid Pool (DSPP) is designed as a feature enhancement module. The last layer output of the feature extraction network is used as input. Through five parallel branches, including a pointwise convolution and four depthwise separable convolution blocks with different sampling rates, the five parts of the results at different scales are connected in the channel dimension and weighted summed using an inverted depthwise separable convolution block to obtain multi-scale features.

[0020] Step 7: The number of decoder layers corresponds to the number of layers of the feature extraction backbone in the encoder. A skip connection strategy is adopted at each decoder layer to connect to the feature extraction backbone corresponding to the encoder. Each decoder layer includes a transposed convolution for upsampling and a convolution block for image restoration, and is processed by two depthwise separable convolution blocks. Finally, a segmentation head composed of a depthwise separable convolution block is used to obtain the crack mask.

[0021] Furthermore, the depthwise separable convolution block includes 3×3 depthwise separable convolution units, batch normalization processing, and HardWish activation function connected in series.

[0022] Furthermore, the ratios α of the four-layer multi-window high- and low-frequency self-attention visual transformers are set to 0.4, 0.5, 0.3, and 0.2 respectively, and the window sizes of each layer are set to 2, 4, and 8; the values ​​of β1, β2, and β3 of each layer are set to 0.3, 0.3, and 0.4.

[0023] Furthermore, the sampling rates of the depthwise separable atrous pyramid pooling in step 6 are set to: 1, 3, 5, 7.

[0024] Furthermore, the method for obtaining the low-frequency component in step 4 is global maximum pooling.

[0025] Furthermore, the feedforward network in the multi-window high-low frequency self-attention visual transformer specifically includes a 1×1 convolution, a HardWish activation function, a 3×3 depth-separable convolution unit, a HardWish activation function and a 1×1 convolution connected in series.

[0026] Furthermore, the convolution block for restoring the image specifically includes an inverted depthwise separable convolution block 1, a depthwise separable convolution block 1, a depthwise separable convolution block 2 and an inverted depthwise separable convolution block 2 which are sequentially connected in series.

[0027] Furthermore, the crack image segmentation method further includes the following steps 8:

[0028] Use BCEWithLogits loss and Dice loss to optimize the network and obtain a probability map for each point. BCEWithLogits loss is a loss function for binary classification. The calculation formula is as follows:

[0029] BCE Loss(x,y)=-(1-y)log(1-x)-ylog(x)

[0030] x represents the predicted value, and y represents the true value. In order to better mine foreground information and highlight the crack area, Dice loss is added:

[0031]

[0032] x represents the predicted value, y represents the true value; and the cosine annealing strategy with warm restart is used to dynamically adjust the learning rate, and the initial learning rate is set to 0.001.

[0033] In another aspect, the present invention further discloses a crack image segmentation system based on a multi-window high-low frequency visual converter, comprising:

[0034] at least one processor; and

[0035] at least one memory in communication with the processor, wherein:

[0036] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute any of the above-mentioned crack image segmentation methods based on multi-window high-low frequency visual converters.

[0037] In another aspect, the present invention also discloses a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the crack image segmentation method based on the multi-window high-low frequency visual converter as described in any one of the above items.

[0038] (3) Beneficial effects

[0039] (1) First, the present invention designs a new multi-window high-low frequency attention mechanism to replace the multi-headed self-attention mechanism, and designs a new visual transformer based on this. First, the visual transformer of the present invention realizes the direct input of the image, which can better retain the spatial position information of the image. Then the image is divided into high-frequency and low-frequency parts for parallel processing, and in the high-frequency part, the present invention uses multiple windows to calculate self-attention. Finally, the fusion features of the high-frequency and low-frequency parts will be obtained. In addition, the present invention uses depthwise separable convolution to replace all the original fully connected layers, which can greatly reduce the parameters and computational complexity.

[0040] (2) Secondly, the present invention designs a depth-wise separable dilated pyramid pooling module for use at the end of the encoder to obtain multi-scale feature maps, which further enhances the features. The module consists of five parallel branches, namely one point-by-point convolution and four depth-wise separable dilated convolutions with different sampling rates. Finally, the five results of different scales are concatenated in the channel dimension and weighted summed using flipped depth-wise separable convolution to obtain multi-scale features.

[0041] (3) In addition, the present invention designs an end-to-end crack segmentation model. The encoder gradually extracts image features, and then restores the image through the decoder, and finally obtains the image mask. The encoder mainly consists of two depth-wise separable convolutions, the visual transformer designed by the present invention, and the depth-wise separable dilated pyramid pooling module designed by the present invention. The decoder consists of a series of depth-wise separable convolution blocks with residual structures. In addition, the present invention adds skip connections to introduce the feature maps of the corresponding layers of the encoder into the decoder to better restore the image.

[0042] (4) The network model designed by the present invention can achieve end-to-end optimization. When training the entire segmentation network, the present invention adopts Dice loss plus BCEWithLogits loss. This can better mine foreground crack information and improve segmentation performance. In addition, a cosine annealing strategy with hot restart is adopted to dynamically adjust the learning rate, alleviate the local minimum problem, and make the trained network more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a diagram of the depthwise separable convolution process in the prior art.

[0044] Figure 2 Schematic diagram of the hole convolution in the prior art.

[0045] Figure 3 The figure is an overall flow chart of the method proposed in the present invention.

[0046] Figure 4 Figure (a) is a diagram showing the composition of depth-separable convolution blocks, and Figure (b) is a diagram showing the composition of convolution blocks for restoring the image proposed by the present invention.

[0047] Figure 5 Schematic diagram of a visual converter in the prior art.

[0048] Figure 6 Schematic diagram of the improved multi-window high-low frequency self-attention visual transformer designed for the present invention.

[0049] Figure 7 Schematic diagram of the multi-window high- and low-frequency attention mechanism structure improved by the present invention.

[0050] Figure 8 (a) is a schematic diagram of the Atrous Pyramid Pooling (ASPP), (b) Figure 8 Schematic diagram of the depth-separable dilute pyramid pooling designed for this invention.

[0051] Figure 9 Performance comparison chart for parameter selection of high and low frequency ratio α;

[0052] Figure 10Performance comparison chart for parameter selection of sampling rate ratio for depthwise separable dilute pyramid pooling;

[0053] Figure 11 This is a performance comparison chart of global average pooling and maximum pooling in the decoding process of the present invention;

[0054] Figure 12 This is a performance comparison chart of the method of the present invention and other methods on CrackForest.

[0055] Figure 13 : is a performance comparison chart of the method of the present invention and other methods on DeepCrack.

[0056] Figure 14 The figure is a performance comparison chart of the method of the present invention and other methods on Crack500.

[0057] Figure 15 This is the segmentation effect diagram of the present invention. Each column, from left to right, is the original image, the predicted effect diagram and the true label. DETAILED DESCRIPTION

[0058] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0059] In the prior art, such as Figure 1 The depthwise separable convolution in

[15] completely decouples the ordinary convolution in terms of channels and space, first performing channel-wise convolution (DW) and then point-wise convolution (PW). The inverted depthwise separable convolution performs PW first and then DW. Figure 2 The red box of the dilated convolution represents the receptive field area. Only the blue squares inside the receptive field are involved in the calculation, the white square positions are assigned 0, and D is the sampling rate.

[0060] The present invention is based on the prior art Figure 3 The encoder and decoder parts shown have been improved:

[0061] Encoder part: such as Figure 3 As shown in the figure, in the encoder part, the present invention uses a pyramid-structured feature extraction network to progressively extract image features, and can obtain feature maps of different scales after crack segmentation. Since the initial input image is too large, directly using a visual transformer to extract image features will result in a large amount of calculation and parameter quantity. Therefore, the present invention uses two series-connected depth-separable convolution blocks (Separable Conv) to perform preliminary learning and extraction of image features, reducing the image size without losing too much image information. The structure of the depth-separable convolution block can be specifically shown as follows: Figure 4As shown in Figure (a), the depthwise separable convolution block includes 3×3 depthwise separable convolution units, batch normalization processing and HardWish activation function connected in series. Then, after passing through the feature extraction backbone composed of downsampling and the visual transformer designed by the present invention, there are preferably four layers in total, each layer including downsampling and visual transformer. In each layer, high-frequency features and low-frequency features are extracted respectively by the visual transformer, and then the two parts of features are fused and sent to the next layer. In addition, the visual transformer designed by the present invention realizes the direct input of the image, and the spatial position information is greatly retained, which is beneficial for image-related tasks. At the same time, depthwise separable convolution is used to replace all fully connected layers, which greatly reduces the number of parameters. At the end of the last layer, the present invention is inspired by the atrous spatial pyramid pooling (ASPP) and also designs a depthwise separable atrous pyramid pooling structure for feature enhancement to obtain a multi-scale feature map.

[0062] Decoder part: According to the multi-scale feature map finally obtained in the encoder, it is input into the decoder to gradually restore the image information and finally obtain the crack mask. The number of decoder layers corresponds to the number of encoder layers, and the number of layers can be preferably four. Figure 3 As shown, in each layer of the decoder, the present invention also adopts a skip connection strategy, merging the output of the corresponding layer of the encoder segment before the input of each layer of the decoder to retain the image information as much as possible. Each layer includes a transposed convolution for upsampling and a convolution block for restoring the image. Figure 4 As shown in Figure (b), the convolution block for restoring the image can be composed of several depthwise separable convolutions with a residual structure, specifically including an inverted depthwise separable convolution block 1, a depthwise separable convolution block 1, a depthwise separable convolution block 2, and an inverted depthwise separable convolution block 2, which are sequentially connected in series to decode the image information. Finally, the present invention obtains a mask through a segmentation head using a depthwise separable convolution.

[0063] In order to improve crack detection accuracy and computational efficiency, based on the above encoder and decoder, the present invention specifically proposes a crack image segmentation method based on a multi-window high-low frequency visual transformer, which specifically includes the following steps:

[0064] 1. Depthwise Separable Convolution Preprocessing

[0065] Step 1: Use two serially connected depth-wise separable convolutional blocks to preliminarily learn and extract the input crack image features, and then use the obtained features Figure X The input is processed into a four-layer cascaded feature extraction backbone, where each layer’s feature extraction backbone includes downsampling and multi-window high- and low-frequency self-attention visual transformers.

[0066] Because the initial input image is typically too large, directly extracting features using a visual transformer would require a significant amount of computation and parameters. Therefore, after inputting the image, the present invention first compresses the image. Using a depthwise separable convolutional block, local features of the image can be extracted, minimizing information loss caused by compression.

[0067] For the multi-window high-low frequency self-attention visual transformer, please refer to the detailed description of steps 2-5 below and Figure 6 , which belongs to the prior art Figure 5 The improvement of the visual transformer in

[15] mainly focuses on the improvement of the multi-headed self-attention mechanism in the visual transformer. In addition, the feedforward network is also improved.

[0068] Furthermore, the depthwise separable convolution block (Separable Conv) includes a series of 3×3 depthwise separable convolution units, batch normalization processing, and HardWish activation function.

[0069] 2. Multi-window high and low frequency attention calculation

[0070] Step 2: Input features Figure X Divide the channel in proportion α, and the high-frequency branch will get (1-α)N h , the low-frequency branch will get αN h , N h is the number of channels.

[0071] It is worth mentioning that Figure 7 As shown, 0<α<1, and the number of low-frequency branches αN h Can be rounded down, the number of high-frequency branches is (1-α)N h It can be rounded up, and the ratio α of the high- and low-frequency self-attention visual transformers in each layer of multi-windows can take different values.

[0072] Step 3: In the high-frequency branch, the feature map is further divided in the channel direction according to different proportions β1, β2, and β3 to obtain feature maps of three different parts X11, X12, and X13. Each part of the feature map will generate a set of query K, key Q, and value V matrices.

[0073] K1i=f ki (X1i), Q1i=f qi (X1i), V1i=f vi (X1i)

[0074] Function f xi() represents the depthwise separable convolution operation, which replaces the conventional linear layer to obtain three matrices to significantly reduce the number of parameters and calculations; and divides the feature map according to different window sizes γ1, γ2, and γ3, and then calculates the scale point multiplication self-attention separately for each window. The calculation formula is as follows:

[0075]

[0076] Wherein, Dhi is the dimension of the hidden layer, Softmax is the normalized exponential function, i takes a value of 1 to 3, and the present invention obtains SA11, SA12 and SA13 through calculation, and then connects the three values ​​of SA11, SA12 and SA13 in the channel direction, and obtains high-frequency features through a projected linear layer and batch normalization.

[0077] Step 4: In the low-frequency branch, the input features are downsampled and pooled, and the low-frequency features in the low-frequency component are obtained through the projection linear layer and batch normalization.

[0078] Furthermore, the present invention compares maximum pooling and average pooling. Since the present invention believes that average pooling will blur the edge details of the image, the effect will be inferior to maximum pooling, and subsequent examples also verify the present invention's conjecture. The present invention obtains the matrix of keys K and values ​​V on the feature map after maximum pooling, and the query matrix Q still needs to be obtained from the original input. Because the scaled dot multiplication self-attention calculation is performed, the present invention cannot change the size of the feature map. The result of the calculation is also passed through the projected linear layer and batch normalization to obtain low-frequency features.

[0079] Step 5: Connect the high-frequency features of step 3 and the low-frequency features of step 4 in the channel direction to obtain a fused feature map, which is then sent to the feature extraction backbone of the next layer.

[0080] Furthermore, the present invention also improves the feedforward network in the visual transformer, such as Figure 6 As shown in Figure 1, the feedforward network specifically includes 1×1 convolution (i.e., point-by-point convolution), HardWish activation function, 3×3 depth-separable convolution unit, HardWish activation function and 1×1 convolution connected in series to improve data processing speed.

[0081] 3. Depth-wise Separable Hollow Pyramid Pooling

[0082] Step 6: Based on the atrous pyramid pool (ASPP), a depthwise separable atrous pyramid pool (DSP) is designed as a feature enhancement module. The last layer output of the feature extraction network is used as input. Through five parallel branches, including a point-by-point convolution and four depthwise separable convolution blocks with different sampling rates, the five parts of the results of different scales are connected in the channel dimension and weighted summed using an inverted depthwise separable convolution block to obtain multi-scale features.

[0083] Further, such as Figure 8 As shown, the original ASPP algorithm only has one global average pooling branch. However, this method, according to the present invention, blurs the edge details of the cracks, making it extremely detrimental to crack segmentation. Therefore, this branch is discarded. Furthermore, the depthwise separable convolution used in this invention uses fewer parameters and computational complexity than conventional convolution, resulting in faster execution. The present invention then concatenates the five results and uses an inverted depthwise separable convolution to reduce the number of channels.

[0084] 4. Decoder side restores image information

[0085] Step 7: The number of decoder layers corresponds to the number of layers of the feature extraction backbone in the encoder. A skip connection strategy is adopted at each decoder layer to connect to the feature extraction backbone corresponding to the encoder. Each decoder layer includes a transposed convolution for upsampling and a convolution block for image restoration, and is processed by two depthwise separable convolution blocks. Finally, a segmentation head composed of a depthwise separable convolution block is used to obtain the crack mask.

[0086] Further, such as Figure 4 As shown in Figure (b), the convolutional block for restoring the image can be composed of several depthwise separable convolutions with a residual structure. Specifically, it includes an inverted depthwise separable convolution block 1, a depthwise separable convolution block 1, a depthwise separable convolution block 2, and an inverted depthwise separable convolution block 2, which are connected in series, thereby quickly decoding the image information. In each layer of the decoder, two depthwise separable convolutions with a residual structure and two inverted depthwise separable convolutions are used to restore the image. When the input of the previous layer arrives, it is connected to the output of the corresponding layer on the decoder side. Finally, the crack mask is obtained through the segmentation head.

[0087] 5. Calculation of crack category probability

[0088] Step 8: Use BCEWithLogits loss and Dice loss to optimize the network and obtain a probability map for each point. BCEWithLogits loss is a loss function for binary classification and is calculated as follows:

[0089] BCE Loss(x,y)=-(1-y)log(1-x)-ylog(x)

[0090] x represents the predicted value, and y represents the true value. In order to better mine foreground information and highlight the crack area, Dice loss is added:

[0091]

[0092] x represents the predicted value, y represents the true value; and the cosine annealing strategy with warm restart is used to dynamically adjust the learning rate, and the initial learning rate is set to 0.001.

[0093] It should be pointed out that step 8 is not a necessary step in the encoder and decoder. It is only to make the trained network more robust.

[0094] It can be seen from the above steps 1-8 that the present invention uses semantic segmentation to detect cracks at the pixel level, improves an end-to-end network structure composed of an encoder and a decoder, and directly outputs a crack mask after network inference, with high training and inference efficiency. In the encoder stage, the present invention uses a visual transformer to extract high and low frequency features in parallel. The visual transformer has many advantages that convolution does not have, such as input adaptation, long-distance and high-order spatial interaction capabilities. And it captures high and low frequencies at the same time, which can better capture small crack features. The multi-window mechanism can capture information on cracks of different scales and ensure the robustness of the model. This improves the detection accuracy of the present invention. For the original visual transformer, see [1]. Figure 5 , the visual converter of the present invention see Figure 6 .

[0095] To illustrate the beneficial effects of the method of the present invention, the following is also combined with Figure 9-15 The experimental effect diagram and Example 1 describe the method of the present invention and its advantages in detail:

[0096] Example 1

[0097] Example 1 is developed entirely using the pytorch framework, and the relevant configuration description is as follows: operating system Ubuntu 20.04, CPU Intel (R) Xeon (R) Platinum 8350C, GPU RTX3090, RAM 43G, Python version 3.8.0, Pytorch version 1.10.0, and CUDA version 11.3.

[0098] The entire training process uses a mini-batch training strategy, training on 3 images at a time for a total of 50 training cycles. After the entire network is trained, the parameters of each component of the network model are saved. During the testing phase, the trained parameters are imported and tested using the test set images.

[0099] Related evaluation indicators:

[0100] The performance indicators used in this experiment to evaluate the model are precision, recall, F1 score, and mean intersection over union (mIOU). Precision is the probability of correctly predicting a pixel class. In this paper, precision is only for crack classes, and its calculation formula is:

[0101]

[0102] The recall rate is the probability that a certain category of pixels is predicted correctly. In this paper, the recall rate is only for crack categories, and its calculation formula is:

[0103]

[0104] The F1 score is a performance indicator that balances precision and recall. Its calculation formula is:

[0105]

[0106] The intersection-over-union ratio is the ratio of the intersection of the true value and the predicted value of the pixel to the union of the true value and the predicted value of the pixel, while the average intersection-over-union ratio is calculated for each class separately and then the average of the IOU of all classes is calculated. The calculation formula is:

[0107]

[0108] Where k represents the number of images. The mIOU values ​​for all images are calculated and then averaged. TP is the number of positive samples predicted as positive, TN is the number of negative samples predicted as negative, FP is the number of negative samples predicted as positive, and FN is the number of positive samples predicted as negative.

[0109] Then the time of multi-window attention was tested: the present invention uses depthwise separable convolution to replace all linear layers (i.e., using linear functions to obtain variables), thereby improving the overall computational efficiency by about 25%.

[0110] See also Figure 9-11 It can be seen that the present invention also conducts comparative experiments to find the best parameter configuration, and all parameter comparative experiments are performed on the DeepCrack dataset.

[0111] 1. First, we search for the ratio α of high and low frequencies in each layer from top to bottom during the encoder stage. The inventor believes that since the lower the number of layers, the more image compression there is, the more feature maps should not be allocated to low frequencies, as this will cause serious loss of inverted image information. Therefore, the present invention chooses to allocate more to low-frequency branches in the upper layer. Specific experiments have verified the inventor's conjecture, see Figure 9 shown.

[0112] 2. In the second aspect, the present invention also compares the sampling rate ratio of the depth-separable hole pyramid pool. The specific experiment is as follows Figure 10 shown.

[0113] 3. Thirdly, the present invention also compares the effects of using different methods to obtain low-frequency components within the low-frequency branch. Since the inventor believes that the global average pooling method will blur the details of the crack edge, which is not conducive to segmentation, the specific experiments are as follows: Figure 11 shown.

[0114] The ratios α adopted in the final experiment are: 0.4, 0.5, 0.3, 0.2; the window sizes of each layer are set to 2, 4, 8; each layer of β1, β2, β3 is set to 0.3, 0.3, 0.4; the sampling rate of the depth-wise separable dilute pyramid pool is set to: 1, 3, 5, 7; and the final method of obtaining the low-frequency component is global maximum pooling.

[0115] Also, see Figure 12-14 It can be seen that the method proposed in this invention has been verified on three authoritative public datasets: CrackForest, DeepCrack and Crack500. Figure 12 The performance comparison of the method of the present invention and other methods on CrackForest is shown. Figure 13 The performance comparison between the proposed method and other methods on the DeepCrack dataset is shown. Figure 14 The performance comparison of the method of the present invention and other methods on Crack500 is shown. The crack segmentation applied by the method proposed in the present invention can be optimized end-to-end. The present invention can achieve good results on the CrackForest, DeepCrack and Crack500 datasets. In particular, the present invention achieves the best performance on the DeepCrack dataset, which proves the superior performance of the method proposed in the present invention in crack detection. In addition, Figure 15 The excellent segmentation results of the method of the present invention are demonstrated.

[0116] It can be seen that compared with the prior art, the image segmentation method of the present invention proposes a crack image segmentation method based on a multi-window high-low frequency visual transformer, which improves the visual transformer to make it more suitable for crack segmentation. By dividing the image into two parts, high frequency and low frequency, which are processed in parallel, rich texture features and global contextual semantic features are obtained, and then the two parts of features are fused. This greatly improves the accuracy and recall rate of segmenting small cracks. At the same time, the features are enhanced by using a deep separable pyramid pool. Multi-scale features are obtained through parallel 5-part convolution operations. And in the encoder stage, jump connections are used to better restore the image. The loss function of the present invention uses Dice loss plus BCEWithLogits loss, which can better mine foreground objects.

[0117] It is worth mentioning that the segmentation method of the present invention can be converted into software program instructions, which can be implemented by running a software analysis system including a processor and a memory, or can be implemented by computer instructions stored in a non-transitory computer-readable storage medium.

[0118] Finally, the method of the present invention is only a preferred embodiment and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A crack image segmentation method based on a multi-window high-low frequency visual transformer, characterized in that: The crack image segmentation method comprises the following steps: Step 1: Two depth-wise separable convolutional blocks are used in series to perform preliminary learning and extraction of the input crack image features. The resulting feature map X is then fed into a four-layer feature extraction backbone. Each layer of the backbone includes downsampling and a multi-window high- and low-frequency self-attention visual transformer. Step 2: Divide the input feature map X in the channel direction according to the ratio α, and the high-frequency branch will get (1-α)N h , the low-frequency branch will get αN h , N h is the number of channels; Step 3: In the high-frequency branch, the feature map is further divided in the channel direction according to different proportions β1, β2, and β3 to obtain feature maps of three different parts X11, X12, and X13. Each part of the feature map will generate a set of query K, key Q, and value V matrices. K1i=f ki (X1i),Q1i=f qi (X1i),V1i=f vi (X1i) Function f xi () represents the depthwise separable convolution operation, which replaces the conventional linear layer to obtain three matrices to significantly reduce the number of parameters and calculations; and divides the feature map according to different window sizes γ1, γ2, and γ3, and then calculates the scale point multiplication self-attention separately for each window. The calculation formula is as follows: Where Dhi is the dimension of the hidden layer, Softmax is the normalized exponential function, i is an integer from 1 to 3, and SA11, SA12, and SA13 are obtained by calculation. Then, the three values ​​of SA11, SA12, and SA13 are concatenated in the channel direction, and high-frequency features are obtained through a projected linear layer and batch normalization. Step 4: In the low-frequency branch, the input features are downsampled and pooled, and the low-frequency features in the low-frequency component are obtained through the projection linear layer and batch normalization; Step 5: Connect the high-frequency features of step 3 and the low-frequency features of step 4 in the channel direction to obtain a fused feature map, which is then sent to the feature extraction backbone of the next layer; Step 6: Based on the Atrous Pyramid Pool (ASPP), a depthwise separable Atrous Pyramid Pool (DSPP) is designed as a feature enhancement module. The last layer output of the feature extraction network is used as input. Through five parallel branches, including a pointwise convolution and four depthwise separable convolution blocks with different sampling rates, the five parts of the results at different scales are connected in the channel dimension and weighted summed using an inverted depthwise separable convolution block to obtain multi-scale features. Step 7: The number of decoder layers corresponds to the number of layers of the feature extraction backbone in the encoder. A skip connection strategy is adopted at each decoder layer to connect to the feature extraction backbone corresponding to the encoder. Each decoder layer includes a transposed convolution for upsampling and a convolution block for image restoration, and is processed by two depthwise separable convolution blocks. Finally, a segmentation head composed of a depthwise separable convolution block is used to obtain the crack mask.

2. The crack image segmentation method based on multi-window high-low frequency visual converter according to claim 1 is characterized in that: The depth-wise separable convolution block includes 3×3 depth-wise separable convolution units, batch normalization processing, and a HardWish activation function connected in series.

3. The crack image segmentation method based on multi-window high-low frequency visual converter according to claim 1 is characterized in that: The ratios α of the four-layer multi-window high- and low-frequency self-attention visual transformers are set to 0.4, 0.5, 0.3, and 0.2 respectively, and the window sizes of each layer are set to 2, 4, and 8; β1, β2, and β3 are set to 0.3, 0.3, and 0.4 for each layer.

4. The crack image segmentation method based on multi-window high-low frequency visual converter according to claim 3 is characterized in that: The sampling rates of depth-wise separable atrous pyramid pooling in step 6 are set to: 1, 3, 5, 7.

5. The crack image segmentation method based on multi-window high-low frequency visual converter according to claim 4 is characterized in that: The method for obtaining the low-frequency component in step 4 is global maximum pooling.

6. The crack image segmentation method based on multi-window high-low frequency visual converter according to claim 1 is characterized in that: The feedforward network in the multi-window high-low frequency self-attention visual transformer specifically includes a 1×1 convolution, a HardWish activation function, a 3×3 depth-separable convolution unit, a HardWish activation function and a 1×1 convolution connected in series.

7. The crack image segmentation method based on multi-window high-low frequency visual converter according to claim 1 is characterized in that: The convolution block for restoring the image specifically includes an inverted depthwise separable convolution block 1, a depthwise separable convolution block 1, a depthwise separable convolution block 2 and an inverted depthwise separable convolution block 2 which are sequentially connected in series.

8. The crack image segmentation method based on a multi-window high-low frequency visual converter according to any one of claims 1 to 7, characterized in that: The crack image segmentation method further comprises the following steps: Step 8: Use BCEWithLogits loss and Dice loss to optimize the network and obtain a probability map for each point. BCEWithLogits loss is a loss function for binary classification and is calculated as follows: BCE Loss(x,y)=-(1-y)log(1-x)-ylog(x) x represents the predicted value, and y represents the true value. In order to better mine foreground information and highlight the crack area, Dice loss is added: x represents the predicted value, y represents the true value; and the cosine annealing strategy with warm restart is used to dynamically adjust the learning rate, and the initial learning rate is set to 0.

001.

9. A crack image segmentation system based on a multi-window high-low frequency visual converter, characterized in that: include: at least one processor; as well as at least one memory in communication with the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the crack image segmentation method based on a multi-window high-low frequency visual converter according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the crack image segmentation method based on a multi-window high-low frequency visual transformer according to any one of claims 1 to 8.

Images