Semantic segmentation method and system for crack image based on depth separable convolution

Abstract

The invention discloses a crack image semantic segmentation method and system based on depth separable convolution. The method comprises the steps of 1, constructing a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and a decoder in the U-Net architecture is constructed by adopting depth separable convolution; 2, training the crack image semantic segmentation network to obtain a crack image semantic segmentation network model; and 3, inputting a to-be-detected crack image into the crack image semantic segmentation network model to obtain a semantic segmentation result. According to the method, the crack width and trend can be accurately identified by constructing the crack image semantic segmentation network model, and the building and the road can be conveniently maintained and evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a crack image semantic segmentation method and system based on deep separable convolution. Background Art

[0002] Cracks may appear on the surface of concrete buildings and roads due to natural factors, improper construction and environmental impact. Cracks not only destroy the beauty of the building and affect the visual effect, but also pose a huge safety hazard. If they are not discovered and repaired in time, the cracks will further intensify, not only requiring more maintenance costs, but also may damage the building structure, thus threatening people's lives. Being able to accurately detect the location, width and direction of cracks is crucial to assessing the health and maintenance of concrete buildings and roads.

[0003] At present, in most regions and countries in the world, crack detection and repair are mainly carried out manually by professionals. Traditional crack identification methods still rely mainly on manual work, through field inspections, handheld flaw detectors and some ultrasonic detection devices. For concrete building surfaces and road surfaces in real life, the area that needs to be inspected is very large, and the manual inspection method has the disadvantages of being time-consuming, expensive and highly subjective.

[0004] With the development of computer vision technology, its application in many industries such as agriculture, medical treatment, transportation and manufacturing has achieved excellent results. Therefore, the current use of image acquisition methods such as drones and monitoring equipment, combined with computer vision technology, to achieve efficient and accurate crack detection has become a research hotspot. Computer vision technology uses digital image processing and pattern recognition methods to analyze the image data of building structures to achieve automated damage detection and quantitative analysis. The existing crack image detection methods based on computer vision technology mainly include classification-based crack detection methods, target-based crack detection methods and semantic segmentation-based crack detection methods.

[0005] The image classification method is to identify whether there are cracks in the image or to classify the crack levels, and then perform binary or multi-classification processing on the image according to whether there are cracks or the crack levels. The image classification method has a good effect on automatically identifying crack damage, but this method lacks accuracy. It can only identify the existence of cracks and other damage, but cannot accurately identify the damage location and damage degree.

[0006] The target detection method can identify the location of the damaged target in the image. Compared with the image classification, the target detection method is more accurate in identifying cracks, but there is less specific information about the cracks, and there are still certain limitations in the evaluation and repair of cracks.

[0007] Semantic segmentation technology analyzes images at the pixel level and separates the foreground and background. It can not only identify the target but also segment the boundary of the target, thereby retaining the semantic information of the image.

[0008] These computer vision-based crack detection algorithms are efficient and safe methods for detecting cracks in concrete structures and roads. However, there are still some limitations: (1) The crack distribution background is complex, making it difficult to accurately identify cracks. (2) There is no fixed pattern in the shape of cracks, the crack distribution is uneven, and the width varies greatly, making it difficult to grasp all the information about the cracks. (3) Most semantic segmentation models have complex structures, require a large number of calculation parameters, and consume a lot of computing resources. Summary of the invention

[0009] In order to at least partially solve the problems of difficulty in accurately identifying cracks, difficulty in grasping all the information of cracks, and large consumption of computing resources by semantic segmentation models, the present invention provides a method and system for semantic segmentation of crack images based on deep separable convolution, by proposing a semantic segmentation network (SDU-Net) for crack images. A lightweight decoder is constructed through deep separable convolution, thereby reducing the complexity of the model and better retaining the high-level features extracted by the encoder, and three modules are designed to improve the performance of the model. First, a feature enhancement module (DCM) integrating an attention mechanism (CBAM) and an excitation module (cSE) is constructed to further enhance and optimize the intermediate features extracted by the encoder. Secondly, an adjacent layer information fusion module (NIF) is constructed to enrich the semantic information of the extracted features. Finally, a feature refinement module (FRM) is constructed by multi-layer convolution jump connection to finally refine the features extracted by the model.

[0010] In order to achieve the above object, the technical solution of the present invention is:

[0011] The first aspect of the present invention proposes a crack image semantic segmentation method based on deep separable convolution, comprising:

[0012] Step 1: construct a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and the decoder in the U-Net architecture is constructed using deep separable convolution, which is convenient for crack image semantic segmentation;

[0013] Step 2: training the crack image semantic segmentation network to obtain a crack image semantic segmentation network model to ensure the accuracy of semantic segmentation;

[0014] Step 3: Input the crack image to be detected into the crack image semantic segmentation network model to obtain the semantic segmentation result.

[0015] Furthermore, in the crack image semantic segmentation network, an adjacent layer information fusion module, an excitation module and a feature enhancement module are provided between the encoder and the decoder, and a feature refinement module is provided after the decoder to facilitate feature extraction and enhancement;

[0016] The multiple encoding layers in the encoder perform layer-by-layer feature extraction on the input crack image. The adjacent layer information fusion module fuses the feature maps output by adjacent encoding layers in the encoder and transmits them to the corresponding decoding layers in the decoder. The excitation module weights the channel information of the output of the first encoding layer in the encoder and outputs the weighted feature map to the last decoding layer in the decoder. The feature enhancement module enhances the features of the output of the last encoding layer in the encoder and outputs the enhanced feature map to the first decoding layer in the decoder. The feature refinement module refines the features of the output of the last decoding layer in the decoder to improve the accuracy of semantic segmentation.

[0017] Furthermore, in the step three, before inputting the crack image to be detected into the crack image semantic segmentation network model, it also includes: preprocessing the crack image to be detected; the preprocessing includes at least one of randomly adjusting the light intensity, horizontally flipping the image, and standardizing the image data, so as to enhance the image features and improve the robustness of the model.

[0018] Furthermore, the decoder in the U-Net architecture is constructed using depth-separable convolution, specifically including:

[0019] Each decoding layer in the decoder includes a depthwise separable convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence; wherein the depthwise separable convolution layer includes a depthwise convolution and a pointwise convolution connected in sequence, so as to reduce the amount of parameters while maintaining its performance.

[0020] Furthermore, the excitation module is expressed by the following formula:

[0021]

[0022] Among them, σ is the sigmoid activation function, MLP is the multi-layer perceptron, GLobalAvgPool is the global average pooling, is the point-by-point multiplication of the matrix, F is the input feature map, F cSE It is the feature map output by the excitation module.

[0023] Furthermore, the adjacent layer information fusion module is expressed by the following formula:

[0024]

[0025] Among them, F′ is the feature map output by the adjacent layer information fusion module, Fa is the feature map output by the a-th encoding layer, F b is the feature map output by the b-th encoding layer, a+1=b, ω is 1×1 convolution, ReLU activation function and downsampling, It is a 1×1 convolution and ReLU activation function.

[0026] Furthermore, the feature enhancement module is expressed by the following formula:

[0027]

[0028] Among them, F CBAM is the output feature map of the attention mechanism, φ is 1×1 convolution, batch normalization and ReLU activation function, θ is 1×1 dilated convolution with a dilation rate of 3 and ReLU activation function, It is the feature map output by the feature enhancement module;

[0029] The output feature map of the attention mechanism is expressed as follows:

[0030]

[0031] Among them, F SA is the feature map of the channel output, F CA It is the feature map of spatial output.

[0032] Further, the feature refinement module includes a first convolution block, a second convolution block, a third convolution block and a fourth convolution block;

[0033] The features extracted by the first convolution block are input to the second convolution block, the features extracted by the second convolution block are input to the third convolution block, the features extracted by the third convolution block are fused with the features extracted by the first convolution block and input to the fourth convolution block, the input of the first convolution block is fused with the features extracted by the fourth convolution block to obtain the semantic segmentation result. This connection method is used to avoid the loss of feature information.

[0034] Furthermore, in the process of training the crack image semantic segmentation network, the adopted loss function is expressed by the following formula:

[0035] L loss =L BCE +L DICE +L SSIM

[0036] Among them, L loss is the loss function, L BCE is the binary cross entropy, L DICE is the dice coefficient function, L SSIM is the structural similarity loss function.

[0037] The second aspect of the present invention proposes a crack image semantic segmentation system based on deep separable convolution, comprising:

[0038] A crack image semantic segmentation network module is used to construct a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and the decoder in the U-Net architecture is constructed by using a depth-separable convolution, which is convenient for crack image semantic segmentation;

[0039] A training module is used to train the crack image semantic segmentation network to obtain a crack image semantic segmentation network model to ensure the accuracy of semantic segmentation;

[0040] The detection module is used to input the crack image to be detected into the crack image semantic segmentation network model to obtain the semantic segmentation result.

[0041] Beneficial effects of the present invention:

[0042] (1) The present invention proposes a crack image semantic segmentation network model, which introduces deep separable convolution to construct a lightweight decoder. The decoder maintains the spatial structure in the feature map while reducing model calculation parameters and saving computing resources.

[0043] (2) The present invention designs three modules to improve the accuracy of crack segmentation. The adjacent layer information fusion module (NIF) can fuse information at different levels to obtain richer semantic information. A feature enhancement module (DCM) is designed and placed at the transition between the encoder and the decoder. The main function of this module is to enhance and optimize the features extracted by the encoder. A feature refinement module (FRM) using jump connections is also designed. It acts at the end of the model to further refine the features extracted by the model, which can improve the integrity of the cracks. Jump connections are used to avoid information loss, solving the problem that the model is difficult to accurately identify cracks and difficult to grasp all the information about the cracks.

[0044] (3) The present invention has carried out a large number of comparative experiments and ablation experiments on three datasets, DeepCrack, crack500 and CCSS. The experimental results show that the crack image semantic segmentation network model proposed in the present invention can achieve the best results and prove the effectiveness of the proposed deep separable convolutional decoder and three modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a crack image semantic segmentation method based on deep separable convolution is provided in an embodiment of the present invention.

[0046] Figure 2 A schematic diagram of a decoder provided in an embodiment of the present invention.

[0047] Figure 3 A schematic diagram of a crack image semantic segmentation network model provided by an embodiment of the present invention.

[0048] Figure 4 A schematic diagram of an excitation module provided in an embodiment of the present invention.

[0049] Figure 5 A schematic diagram of a feature enhancement module provided in an embodiment of the present invention.

[0050] Figure 6 A schematic diagram of an attention mechanism provided for an embodiment of the present invention.

[0051] Figure 7 A schematic diagram of an adjacent layer information fusion module provided in an embodiment of the present invention.

[0052] Figure 8 A schematic diagram of a feature refinement module provided in an embodiment of the present invention.

[0053] Fig. 9 A schematic diagram of a PR curve for a DeepCrack dataset test provided in an embodiment of the present invention.

[0054] Fig.10 A schematic diagram of a PR curve tested using the crack500 dataset provided in an embodiment of the present invention.

[0055] Fig.11 A schematic diagram of a PR curve for a CCSS dataset test provided in an embodiment of the present invention.

[0056] Fig.12 A schematic diagram of experimental test results provided for an embodiment of the present invention.

[0057] Fig.13 An architecture diagram of a crack image semantic segmentation system based on deep separable convolution provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Example 1

[0060] like Figure 1 As shown, a crack image semantic segmentation method based on deep separable convolution includes the following steps:

[0061] S101: constructing a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and the decoder in the U-Net architecture is constructed by adopting a depth-separable convolution.

[0062] Specifically, a new decoder is constructed using deep separable convolution in the crack image semantic segmentation network model, such as Figure 2 As shown in the figure, the left side is the traditional decoder structure of U-Net, and the right side is the modified decoder. The traditional U-Net model consists of a symmetrical encoder and decoder. The encoder is responsible for extracting image features, while the decoder maps these features back to the original image size. The traditional U-Net decoder consists of double convolution and upsampling operations, which will lose some feature information of the crack image and may limit the expression ability and generalization performance of the model in complex crack scenes. Therefore, the embodiment of the present invention introduces deep separable convolution to construct a new decoder. The new decoder includes multiple decoding layers (in this embodiment, four decoding layers are set), each decoding layer includes a deep separable convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence, wherein the deep separable convolution layer includes deep convolution and point-by-point convolution connected in sequence. The deep convolution first independently convolves each channel of the input data to generate a corresponding number of output channels, and then the point-by-point convolution mixes these output channels into the final output feature map through a 1x1 convolution kernel.

[0063] For example, there is a feature map with an input shape of H×W×C, and the feature extraction output channel is N. For the traditional convolution with a convolution kernel size of k, the number of parameters is k×k×C×N. In the depthwise separable convolution, the depthwise convolution performs independent convolution on each channel, the convolution kernel is k, the parameters of each channel are k×k×1, and there are C channels in total, so the number of parameters of the depthwise convolution is k×k×C. The pointwise convolution kernel size is 1, the number of input channels is C, and the number of output channels is N, so the number of pointwise convolution parameters is 1×1×C×N, and the total number of parameters of the depthwise separable convolution is k×k×C+C×N. Therefore, the decoder can maintain the model while reducing the number of parameters. Compared with the traditional U-Net decoder structure, it not only has better performance, but also can improve the operation efficiency of the model.

[0064] S102: Training the crack image semantic segmentation network to obtain a crack image semantic segmentation network model.

[0065] S103: Inputting the crack image to be detected into the crack image semantic segmentation network model to obtain a semantic segmentation result.

[0066] It is understandable that in practical applications, whether in the model training process or in the actual crack image testing process, the format of the original crack image may not be suitable for the pre-built network model. At the same time, in order to improve the robustness of the model, the image generally needs to be pre-processed before it is input into the network model. In the data preprocessing process, the embodiment of the present invention adopts a variety of techniques to enhance the characteristics of the image and improve the robustness of the model. These include methods such as randomly adjusting the light intensity, horizontal flipping and standardization of the image, and the comprehensive application of these techniques contributes to the accuracy and robustness under real environmental conditions.

[0067] The present invention adopts deep separable convolution to construct a new decoder in the crack image semantic segmentation network model, which is convenient for retaining context information while reducing the training cost of the crack image semantic segmentation network model, and uses the loss function to train the model to improve the accuracy of the model's semantic segmentation.

[0068] Example 2

[0069] Based on the above embodiments, Figure 3 As shown, an embodiment of the present invention provides a specific structure of a crack image semantic segmentation network model, specifically including: an adjacent layer information fusion module (NIF), an excitation module (cSE) and a feature enhancement module (DCM) are arranged between the encoder and the decoder, and a feature refinement module (FRM) is arranged after the decoder.

[0070] The multiple encoding layers in the encoder perform layer-by-layer feature extraction on the input crack image. The adjacent layer information fusion module fuses the feature maps output by adjacent encoding layers in the encoder and transmits them to the corresponding decoding layers in the decoder. The excitation module weights the channel information of the output of the first encoding layer in the encoder, and outputs the weighted feature map to the last decoding layer in the decoder. The feature enhancement module enhances the features of the output of the last encoding layer in the encoder and outputs the enhanced feature map to the first decoding layer in the decoder. The feature refinement module refines the features of the output of the last decoding layer in the decoder.

[0071] The structure of the incentive module is as follows: Figure 4 As shown in Figure 1, the excitation module can adaptively learn the correlation between feature channels. This module generates the attention weight of each channel through two steps: compression and excitation. Specifically, the first step is global average pooling to convert the feature map of each channel into a scalar. The second step is to learn and adjust the weight of each channel using a fully connected two-layer network to generate the attention weight of each channel. The excitation module is expressed as follows:

[0072]

[0073] Among them, σ is the sigmoid activation function, MLP is the multi-layer perceptron, GLobalAvgPool is the global average pooling, is the point-by-point multiplication of the matrix, F is the input feature map, F cSE It is the feature map output by the excitation module.

[0074] The DCM module structure is as follows Figure 5 As shown in the figure, this module combines the attention mechanism (CBAM) and the channel squeeze and excitation module (cSE). The channel attention mechanism of CBAM can weight different channels of the feature map, so that important feature channels are strengthened and irrelevant channels are suppressed. CSE adjusts the weight of each channel by learning global information to enhance the response of useful features. In many deep learning models, an end-to-end model with encoder and decoder is usually used. In the image segmentation task, the encoder's task is to extract high-level features. Most cracks do not have a fixed shape, and there are complex backgrounds in the location, which makes it difficult to extract crack features. The DCM module enhances and optimizes the high-level features extracted by the encoder to accurately distinguish cracks from the background and improve the accuracy of crack segmentation.

[0075] DCM combines the features of CBAM and cSE in a splicing manner. This module makes comprehensive use of two different attention mechanisms to further improve the discriminability and effectiveness of features. After the fused features are passed through 1×1 convolution, batch normalization and activation function ReLU, the CBAM optimized features and the fused features are added again as feature supplements, and finally the features are subjected to 1×1 dilated convolution with a dilation rate of 3. The mathematical expression of the above enhancement steps is as follows:

[0076]

[0077] Among them, F CBAM is the output feature map of the attention mechanism, F cSE is the feature map output by the excitation module, φ is a 1×1 convolution, batch normalization and ReLU activation function, θ is a 1×1 dilated convolution with a dilation rate of 3 and a ReLU activation function, It is the feature map output by the feature enhancement module.

[0078] CBAM is used to learn the correlation between the spatial and channel features of an image, which enables the model to understand and process complex visual information more effectively. CBAM is combined with U2-Net to improve the segmentation accuracy of the model. CBAM is a combination of channel attention and spatial attention. The attention mechanism of CBAM is as follows: Figure 6 (a) Channel attention mechanism structure Figure 6 (b) Spatial attention mechanism Figure 6(c) As shown. The channel attention module learns the correlation between features from the channel dimension of the feature map and grasps the important information in the channel. The channel attention mechanism converts the feature map of each channel into a scalar value through global average pooling, and then inputs these scalar values ​​into the fully connected network for learning and weight adjustment. The spatial attention module aims to improve the model's attention to spatial information by learning the correlation between different positions in the feature map. Through the maximum pooling operation, the most significant features are extracted from the feature map of each channel, and then these features are input into the fully connected network for learning and weight adjustment to enhance useful spatial information and suppress irrelevant spatial information. The calculation process of CBAM is as follows:

[0079]

[0080] F CA =σ(MLP(GlobalAvgPool(F))+MLP(GlobalMaxPool(F)))

[0081] F SA =σ(CONV 7×7 (Cat(GlobalAvgPool(F),GlobalMaxPool(F))))

[0082] Among them, F SA is the feature map of the channel output, F CA is the feature map of spatial output, GlobalMaxPool is the global maximum pooling, CONV 7×7 It is a 7×7 convolution.

[0083] Preferably, the DCM module uses dilated convolution, which has better performance in processing global information and maintaining resolution than traditional convolution. The crack segmentation task requires the model to be able to effectively capture the details and overall shape of the cracks, and to perform fine-grained pixel-level predictions on the image, it is also necessary to maintain high resolution of the feature map. Referencing dilated convolution in the DCM module can increase the receptive field of the model, so that it can better capture the global information of the cracks. In addition, dilated convolution can maintain the resolution of the feature map while increasing the receptive field, avoiding the problem of resolution degradation caused by multiple pooling in traditional convolution, which is beneficial to improving the accuracy of segmentation. The calculation formula for the dilated convolution receptive field is as follows:

[0084] r out =r in +(k-1)×(d-1)

[0085] Among them, r out is the receptive field size of a neuron on the output feature map relative to the input feature map, r inis the receptive field size of a neuron on the input feature map, k is the size of the convolution kernel, and d is the dilation rate.

[0086] The NIF module structure is as follows Figure 7 As shown in the figure, the NIF module can retain more detailed information of the original image during the decoding process, avoiding information loss and helping to improve the accuracy of segmentation. Different levels in the semantic segmentation model have different semantic information. In order to retain as much semantic information as possible, the NIF module is designed for feature fusion of adjacent levels. This module first processes the high-level features through 1×1 convolution, activation function ReLU and maximum pooling with a step size of 2, adjusts the high-level features and adjacent low-level features to feature maps of the same size, concatenates the high-level features with the low-level features, then processes the dilated convolution and activation function ReLU, and optimizes the concatenated features through the cSE attention mechanism, and finally adds the concatenated features to the low-level features. The adjacent layer information fusion module is expressed as follows:

[0087]

[0088] Among them, F′ is the feature map output by the adjacent layer information fusion module, F a is the feature map output by the a-th encoding layer, F b is the feature map output by the b-th encoding layer, a+1=b, ω is 1×1 convolution, ReLU activation function and downsampling, It is a 1×1 convolution and ReLU activation function.

[0089] In actual crack scenarios, cracks often have complex textures and subtle extensions. In order to make crack segmentation more accurate, the FRM module is proposed to further refine the rough features extracted by the model. Its structure is as follows: Figure 8 As shown. This module uses 4 layers of 3×3 convolution with ReLU function for coarse feature extraction. Specifically, the FRM module includes the first convolution block, the second convolution block, the third convolution block and the fourth convolution block. The first convolution block, the second convolution block, the third convolution block and the fourth convolution block all include convolution layers and ReLU activation function layers. The features extracted by the first convolution block are input to the second convolution block, the features extracted by the second convolution block are input to the third convolution block, the features extracted by the third convolution block are fused with the features extracted by the first convolution block and then input to the fourth convolution block. The input of the first convolution block is fused with the features extracted by the fourth convolution block to obtain the semantic segmentation result. This jump connection method is used to prevent the loss of feature information.

[0090] Example 3

[0091] Based on the above embodiments, the embodiment of the present invention provides a loss function for training a model, which is as follows:

[0092] In the binary crack segmentation task, class imbalance is a common and important problem, because cracks are usually small and sparse features on the road surface or structure surface, while non-crack areas occupy most of the image. This imbalance will affect the training and performance of the model, causing the model to tend to pay more attention to non-crack areas and ignore crack areas. Therefore, the embodiment of the present invention uses a combined loss function when training the model.

[0093] In order to effectively capture the characteristic information of cracks at each scale, a loss function that is a superposition of three loss functions, namely binary cross entropy, Dice coefficient function and structural similarity loss function (SSIM) is used. The Dice loss function is a loss function commonly used in image segmentation tasks. Its main function is to measure the similarity or overlap between the predicted segmentation results and the true segmentation results. It is usually used to evaluate the performance of the segmentation model and guide model training. The SSIM loss function accurately quantifies cracks of different widths by predicting the spatial location of the cracks. This measurement method does not require pixel-by-pixel correspondence and can explain slight geometric differences when comparing crack structures.

[0094] The loss function is expressed as follows:

[0095] L loss =L BCE +L DICE +L SSIM

[0096]

[0097] Among them, L loss is the loss function, L BCE is the binary cross entropy, L DICE is the dice coefficient function, L SSIM is the structural similarity loss function, h×w is the size of the image, y (x,y) is the ground truth, y (x,y) ∈[0,1], is the predicted probability graph of the network, (x, y) is the coordinate of the image pixel. When (x, y) is set to the foreground (crack), it is assigned a value of 1. Otherwise, when it is set to the background (non-crack), it is assigned a value of 0. ε is the smoothing factor. In order to prevent the denominator from being 0, ε is taken as 1. μ is the mean and σ is the variance. for and (x,y) The covariance of k1 is the decimal for calculating stability, and its value is 0.012; k2 is the decimal for calculating stability, and its value is 0.032.

[0098] Example 4

[0099] On the basis of the above embodiments, the embodiments of the present invention provide an experimental test process of a crack image semantic segmentation network model, and a large number of comparative experiments and ablation experiments are carried out on three datasets: DeepCrack, crack500 and CCSS.

[0100] DeepCrack: The DeepCrack dataset contains 537 images, 300 of which are used for training and 237 for testing, with a resolution of 544×384 pixels. The dataset has rich image diversity, including concrete and asphalt road cracks with different widths and various environmental conditions. This diversity can help the algorithm better adapt to real-world application scenarios.

[0101] Crack500: The Crack500 dataset contains 1896 images for training and 1124 images for testing, with a resolution of 640 × 360 pixels. The shapes and widths of the cracks in this dataset are diverse, and the background has many complex patterns and spots.

[0102] CCSS: The CCSS dataset contains 721 images for training and 226 images for testing, with image resolution of 512×512 pixels. The dataset consists of a public concrete crack segmentation dataset and the CCSS-DATA dataset. The public concrete crack segmentation dataset contains 458 images, of which 186 are 2448×3264 pixels and 272 are 4032×3024 pixels. The CCSS-DATA dataset consists of 670 RGB images and their corresponding semantic segmentation labels. The dataset was collected in different seasons, the images have different lighting conditions, and have complex background conditions such as snow and grass.

[0103] The software tools of this experiment use Python 3.10.13 (programming language), PyTorch 2.1.0 (open source deep learning framework for machine learning and deep learning), and CUDA 12.2 (unified computing device architecture) for experiments. The hardware equipment of the experiment uses Intel Core i5-12400F CPU, 16GB RAM and NVIDIA GeForce RTX 4060 GPU with 8GB memory.

[0104] The training images were randomly cropped to 320×320 pixels to increase the training speed. The network was optimized using the Adaptive Moment Estimation (Adam) optimizer. The initial learning rate was set to 0.001, and cosine annealing was used to smoothly adjust the learning rate during training. The batch size was 2, and a total of 50 epochs were trained.

[0105] This experiment uses precision (Pr), recall (Re), F1 score (F1) and mean intersection / union (MIoU) as evaluation indicators. These indicators are widely used to judge the performance of network segmentation, and the calculation formula is as follows:

[0106]

[0107] Among them, TP is the pixel where the model correctly predicts cracks, FP is the pixel where the model incorrectly predicts cracks, FN is the pixel where the model incorrectly predicts non-cracks, TN is the pixel where the model correctly predicts non-cracks, Pr is precision, Re is recall, F1 is F1 score, and MioU is the average intersection / union ratio.

[0108] The proposed model is tested on DeepCrack, crack500 and CCSS datasets, and 9 models are used to compare with the proposed model. The models used for comparison are FCN, SegNet, DeepCrack, U-Net, PYAttUNet, Swin-Unet, TransUNet, DeepLabv3+ and U2-Net.

[0109] The PR curve of the DeepCrack dataset test is as follows Fig. 9 As shown in Table 1, SDU-Net achieved the best performance. From Table 1, we can see that the Pr, Re, MIoU and F1 values ​​of DSU-Net are 92.04%, 82.75%, 88.11% and 87.15% respectively, which are 1.45% and 1.7% higher than the suboptimal U2-Net model MIoU and F1 respectively. Fig.12 As shown, Fig.12 (a)(b) are the crack detection results selected in DeepCrack. It can be seen that DSU-Net has a more accurate grasp of the crack boundary and a more complete crack segmentation.

[0110] Table 1 Performance comparison on the Deepcrack dataset

[0111]

[0112] The crack500 dataset has complex images and crack identification is difficult. The PR curve of the crack500 dataset test is shown in the figure below. Fig.10As shown, SDU-Net still has the best performance at the stable position of the PR curve. From Table 2, it can be seen that the Pr, Re, MIoU and F1 values ​​of DSU-Net are 66.98%, 71.98%, 74.81% and 69.39% respectively, which are 1.26% and 2.17% higher than the suboptimal U2-Net model MIoU and F1 respectively. The recognition effects of FCN and TransUNet are the worst, and the MIoU and F1 are 8.31% and 8.99% higher than the FCN and Swin-UNets models respectively, and 13.66% and 14.82% higher. Fig.12 As shown, Fig.12 (c) and (d) are the crack detection results selected from crack500. It can be seen that DSU-Net is more complete and clearer in the segmentation of cracks, while FCN and Swin-UNets have more missing results.

[0113] Table 2 Performance comparison on the crack500 dataset

[0114]

[0115]

[0116] like Fig.11 As shown in Table 3, SDU-Net still has the best performance in the CCSS dataset test results. From Table 3, we can see that the Pr, Re, MIoU and F1 values ​​of DSU-Net are 81.68%, 76.19%, 82.43% and 78.84% respectively, which are 1.19% and 1.76% higher than the suboptimal U2-Net model MIoU and F1 respectively. Fig.12 As shown, Fig.12 (e) (f) There are many small cracks and images with complex backgrounds in the CCSS dataset. From the segmentation effect, DSU-Net has a very good effect on identifying small cracks and suppressing noise. It can also identify connection gaps that are very similar to cracks.

[0117] Table 3 Performance comparison on the CCSS dataset

[0118]

[0119] In order to improve the accuracy of segmentation, most semantic segmentation models have complex model structures. Floating point operations (FLOPs), parameter counts (Params), frames per second (FPS), and inference time (IPT) are used as metrics. The calculation formulas for FLOPs and Params are as follows:

[0120] FLOPs = 2HW (C in K 2+1)C out

[0121] Params=(C in K 2 +1)C out

[0122] Among them, H is the height of the input feature map, W is the width of the input feature map, and C in is the number of channels of the input feature map, C out is the number of output channels, K is the kernel size, FLOPs is the floating point operation, and Params is the parameter count.

[0123] The model is tested using a 512×512×3 tensor, and the model evaluation results are shown in Table 4. According to the evaluation results, DSU-Net has relatively few FLOPs, only higher than Swin-Unet and DeepLabv3+ models. DSU-Net has the smallest Params, with the least number of parameters and the best crack segmentation accuracy. The inference time of DSU-Net is in the middle among the comparison models.

[0124] Table 4 Model calculation cost

[0125]

[0126] In DSU-Net, a decoder is constructed using deep separable convolution. The effect of the deep separable convolution decoder (DSCD) is tested on three datasets. As shown in Table 5, the U-Net model and the model after replacing the decoder are tested on the three datasets during the ablation experiment. After the deep separable convolution decoder is used, the tests on the three datasets are improved, proving the effectiveness of the deep separable convolution decoder. On the DeepCrack dataset, Pr, Re, MIoU and F1 are improved by 1.6%, 1.35%, 1.3% and 1.52% respectively. On the crack500 dataset, Re, MIoU and F1 are improved by 3.34%, 0.14% and 1.13% respectively, but Pr is reduced by 0.97%. On the CCSS dataset, Pr, Re, MIoU and F1 are improved by 1.49%, 0.28%, 0.72% and 0.82% respectively. Although the deep separable convolution decoder will not improve the test results of a certain dataset, the overall performance is improved.

[0127] Table 5 Effect of depth-separable convolutional decoder

[0128]

[0129] Three modules are designed in DSU-Net to improve the segmentation performance of cracks. In order to explore the impact of each module on the overall performance of the crack segmentation model, specific parts of the model are gradually eliminated and their impact on the crack segmentation task is observed. The DeepCrack dataset is used to test the effect of each module. The test results are shown in Table 6. The three designed modules can improve the segmentation accuracy of the model, but the segmentation accuracy when DCM and FRM are used at the same time is slightly lower than that of the two modules used separately. After adding the NIF module, the information enrichment will make up for the information loss caused by the simultaneous use of DCM and FRM.

[0130] Table 6 Performance improvement effects of each module

[0131]

[0132] In summary, concrete and road cracks usually have different shapes and exist in a complex background. This paper proposes a semantic segmentation method for crack images based on deep separable convolution, which aims to accurately and comprehensively identify concrete and road cracks. DSU-Net uses deep separable convolution to construct a decoder, which reduces the number of model parameters and can better reduce the feature loss of cracks. In order to improve the accuracy of crack segmentation, three functional modules are constructed. First, it is a DCM module for enhancing intermediate features, which effectively enhances the extracted features. Secondly, the features of adjacent levels are fused through NIF, aiming to enrich the extraction of semantic features and help to segment cracks more accurately. Finally, FRM is used to refine and optimize the features extracted by the model in depth, which further improves the accuracy and robustness of the segmentation results. Through ablation experiments, the effectiveness of several functions is demonstrated using experimental data. Extensive experimental verification is carried out on three different datasets, including DeepCrack, Crack500 and CCSS. The test results of the three datasets are improved by 1.29% and 1.89% on average in MIoU and F1 respectively compared with the suboptimal model.

[0133] The present invention is of great significance for concrete construction and road maintenance. Accurate crack identification can help evaluate the structural integrity of buildings and roads, detect and repair cracks in a timely manner, and help improve the safety and durability of buildings and roads. Future research directions can further optimize the efficiency of the model and promote the method to other related fields to meet the needs of practical engineering applications. In the future, the model can be further improved to enable more accurate identification of potential building damage such as tiny cracks, so that the further evolution of cracks can be predicted.

[0134] Example 5

[0135] Corresponding to the above method, such as Fig.13As shown, a crack image semantic segmentation system based on deep separable convolution includes:

[0136] The crack image semantic segmentation network module is used to construct a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and the decoder in the U-Net architecture is constructed by using deep separable convolution.

[0137] The training module is used to train the crack image semantic segmentation network to obtain a crack image semantic segmentation network model.

[0138] The detection module is used to input the crack image to be detected into the crack image semantic segmentation network model to obtain the semantic segmentation result.

[0139] It should be noted that a crack image semantic segmentation system based on deep separable convolution provided in an embodiment of the present invention is to implement a crack image semantic segmentation method based on deep separable convolution in the above embodiment. Its specific functions can be referred to the above method embodiments and will not be repeated here.

[0140] In summary, the present invention proposes a crack image semantic segmentation network model, which introduces deep separable convolution to construct a lightweight decoder. The decoder maintains the spatial structure in the feature map while reducing model calculation parameters and saving computing resources. The present invention designs three modules to improve the segmentation accuracy of cracks. The adjacent layer information fusion module (NIF) can fuse information at different levels to obtain richer semantic information. A feature enhancement module (DCM) is designed and placed at the transition position between the encoder and the decoder. The main function of this module is to enhance and optimize the features extracted by the encoder. A feature refinement module (FRM) using jump connections is also designed, which acts at the end of the model to further refine the features extracted by the model, which can improve the integrity of the cracks, and uses jump connections to avoid information loss, solving the problem that the model is difficult to accurately identify cracks and difficult to grasp all the information of the cracks. The present invention has carried out a large number of comparative experiments and ablation experiments on three data sets: DeepCrack, crack500 and CCSS. The experimental results show that the crack image semantic segmentation network model proposed by the present invention can achieve the best results, and proves the effectiveness of the proposed deep separable convolution decoder and three modules.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A crack image semantic segmentation method based on deep separable convolution, characterized in that: include: Step 1: construct a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and the decoder in the U-Net architecture is constructed by using a depth-separable convolution; Step 2: training the crack image semantic segmentation network to obtain a crack image semantic segmentation network model; Step 3: Input the crack image to be detected into the crack image semantic segmentation network model to obtain the semantic segmentation result.

2. The crack image semantic segmentation method based on depthwise separable convolution according to claim 1, characterized in that: In the crack image semantic segmentation network, an adjacent layer information fusion module, an excitation module and a feature enhancement module are further arranged between the encoder and the decoder, and a feature refinement module is further arranged after the decoder; The multiple encoding layers in the encoder perform layer-by-layer feature extraction on the input crack image. The adjacent layer information fusion module fuses the feature maps output by adjacent encoding layers in the encoder and transmits them to the corresponding decoding layers in the decoder. The excitation module weights the channel information of the output of the first encoding layer in the encoder, and outputs the weighted feature map to the last decoding layer in the decoder. The feature enhancement module enhances the features of the output of the last encoding layer in the encoder and outputs the enhanced feature map to the first decoding layer in the decoder. The feature refinement module refines the features of the output of the last decoding layer in the decoder.

3. The crack image semantic segmentation method based on depthwise separable convolution according to claim 1, characterized in that: In the step three, before inputting the crack image to be detected into the crack image semantic segmentation network model, it also includes: preprocessing the crack image to be detected; the preprocessing includes at least one of randomly adjusting the light intensity, horizontally flipping the image, and standardizing the image data.

4. The crack image semantic segmentation method based on depthwise separable convolution according to claim 2, characterized in that: The decoder in the U-Net architecture is constructed using depthwise separable convolution, specifically including: Each decoding layer in the decoder includes a depthwise separable convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence; wherein the depthwise separable convolution layer includes a depthwise convolution and a pointwise convolution connected in sequence.

5. The crack image semantic segmentation method based on depthwise separable convolution according to claim 2, characterized in that: The excitation module is expressed by the following formula: Among them, σ is the sigmoid activation function, MLP is the multi-layer perceptron, GLobalAvgPool is the global average pooling, is the point-by-point multiplication of the matrix, F is the input feature map, F cSE It is the feature map output by the excitation module.

6. The crack image semantic segmentation method based on depthwise separable convolution according to claim 5, characterized in that: The adjacent layer information fusion module is expressed by the following formula: Among them, F′ is the feature map output by the adjacent layer information fusion module, F a is the feature map output by the a-th encoding layer, F b is the feature map output by the b-th encoding layer, a+1=b, ω is 1×1 convolution, ReLU activation function and downsampling, It is a 1×1 convolution and ReLU activation function.

7. The crack image semantic segmentation method based on depthwise separable convolution according to claim 5, characterized in that: The feature enhancement module is expressed by the following formula: Among them, F CBAM is the output feature map of the attention mechanism, φ is 1×1 convolution, batch normalization and ReLU activation function, θ is 1×1 dilated convolution with a dilation rate of 3 and ReLU activation function, It is the feature map output by the feature enhancement module; The output feature map of the attention mechanism is expressed as follows: Among them, F SA is the feature map of the channel output, F CA It is the feature map of spatial output.

8. The crack image semantic segmentation method based on depthwise separable convolution according to claim 2, characterized in that: The feature refinement module includes a first convolution block, a second convolution block, a third convolution block and a fourth convolution block; The features extracted by the first convolution block are input to the second convolution block, the features extracted by the second convolution block are input to the third convolution block, the features extracted by the third convolution block are fused with the features extracted by the first convolution block and input to the fourth convolution block, the input of the first convolution block is fused with the features extracted by the fourth convolution block to obtain the semantic segmentation result.

9. The crack image semantic segmentation method based on depthwise separable convolution according to claim 1, characterized in that: In the process of training the crack image semantic segmentation network, the adopted loss function is expressed by the following formula: L loss =L BCE +L DICE +L SSIM Among them, L loss is the loss function, L BCE is the binary cross entropy, L DICE is the dice coefficient function, L SSIM is the structural similarity loss function.

10. A crack image semantic segmentation system based on deep separable convolution, characterized in that: include: A crack image semantic segmentation network module, used to construct a crack image semantic segmentation network; the crack image semantic segmentation network adopts a U-Net architecture, and the decoder in the U-Net architecture is constructed by using a depth-separable convolution; A training module, used for training the crack image semantic segmentation network to obtain a crack image semantic segmentation network model; The detection module is used to input the crack image to be detected into the crack image semantic segmentation network model to obtain the semantic segmentation result.

Images