An all-view feature fusion crack segmentation method based on improved Unet

Abstract

The application discloses a kind of full field feature fusion crack segmentation methods based on improved Unet, belong to image segmentation technical field. Including: obtaining image dataset, and pre-processing;Improved network NxUnet based on Unet network is constructed, and improved serpentine dynamic convolution layer is used instead of basic convolution layer;Residual CBAM module is added in the skip connection of each layer of encoder and decoder;Multi-feature fusion attention module HUBCS is introduced to process the feature map of each layer of the encoder, and the result is sent into each corresponding layer of the decoder;The improved network NxUnet is trained using the pre-processed image dataset, and the trained improved network NxUnet is obtained.The application improves Unet network model by full field feature fusion with residual CBAM module in self-defined HUBCS module, better captures and utilizes global and local information in image. The precision of segmentation is improved, and the generalization ability is also enhanced.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a crack segmentation method based on full-view feature fusion using an improved Unet. Background Technology

[0002] In an industrial context, crack segmentation is a crucial task. The accuracy and reliability of crack detection directly affect the urgency of subsequent remedial measures and the estimated cost of remediation.

[0003] Traditional inspection methods typically rely on manual inspection, which is not only time-consuming and labor-intensive but also susceptible to human error. By segmenting specific regions in an image, structures or defects of interest, such as cracks, corrosion, and wear, can be identified and separated, thus assisting engineers in inspection and maintenance. Efficient and accurate image segmentation has a wide range of applications in industrial image processing, including but not limited to non-destructive testing (NDT), structural health monitoring, quality control, and preventative maintenance. With the rapid development of computer vision and deep learning technologies, automated image segmentation technology provides a more efficient and accurate solution for crack detection.

[0004] Among existing image segmentation algorithms, UNet is a commonly used and high-performing deep learning model. Through its encoder-decoder structure, UNet effectively captures the contextual information and detailed features of images, thus achieving significant results in many industrial image segmentation tasks. Since its inception, the UNet model has become one of the mainstream methods in industrial image segmentation and has undergone numerous improvements and optimizations, resulting in many variants and enhancements, such as the introduction of attention mechanisms, residual connections, and dense connections. These improved versions have enhanced segmentation performance to some extent, but still suffer from limited generalization ability, especially when facing different types of cracks and complex backgrounds. Meanwhile, there are also inventions that introduce serpentine dynamic convolution modules, but due to the large computational cost, high memory overhead, and long training time of the original serpentine dynamic convolution module, it is not very suitable for many scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a full-view feature fusion method to improve the Unet network model, thereby enhancing its segmentation performance and generalization ability. This method, through more comprehensive and detailed image processing, better captures and utilizes global and local information within the image. This not only improves segmentation accuracy but also enhances the model's generalization ability across different datasets and application scenarios.

[0006] To achieve the above objectives, this invention proposes a full-view feature fusion crack segmentation method based on an improved Unet, comprising the following steps:

[0007] (1) Obtain the image dataset and perform preprocessing;

[0008] (2) Construct an improved network NxUnet based on the Unet network, and replace the basic convolutional layer in the UNet network with an improved snake-shaped dynamic convolutional layer; add a residual CBAM module to the skip connection of each layer of the encoder and decoder; introduce a multi-feature fusion attention module HUBCS to process the feature maps of each layer of the encoder, fuse the feature maps of each layer of the encoder, extract features from the feature maps through channel attention mechanism and spatial attention mechanism, and perform dot product operation between the feature maps of each layer of the decoder and the extracted features, and finally send the result to the corresponding layer of the decoder.

[0009] (3) The improved network NxUnet is trained using the preprocessed image dataset;

[0010] (4) Obtain the crack image to be segmented, and use the trained improved network NxUnet to generate the segmentation result image.

[0011] Furthermore, the preprocessing includes:

[0012] (1.1) Adjust the size of each image in the image dataset to 224 pixels × 224 pixels;

[0013] (1.2) Convert the image dataset with adjusted image size into tensor format and perform image brightness and contrast enhancement;

[0014] (1.3) Divide the image dataset after data conversion into a training set and a test set in a 9:1 ratio.

[0015] Furthermore, the NxUnet network consists of two parts: an encoder and a decoder.

[0016] The encoder is downsampled sequentially by 5 layers of the improved serpentine dynamic convolutional layer; the decoder is upsampled sequentially by 5 layers of the improved serpentine dynamic convolutional layer; the output of each layer of the encoder is skipped through the residual CBAM module and then sent to the corresponding layer of the decoder.

[0017] The feature maps output from the first four layers of the encoder and the feature maps input from the corresponding layers of the decoder are also simultaneously fed into the HUBCS module, and their outputs are respectively fed into the corresponding layers of the decoder.

[0018] The output of the last upsampled layer is fed into a module consisting of a BatchNorm regularization layer, a ReLU activation function, and a 3×3 convolutional layer, and then passes through a Sigmoid activation function to obtain the final segmentation result.

[0019] Furthermore, the improved serpentine dynamic convolutional layer includes DSConv, BatchNorm, Dropout functions, and Leak ReLU activation function connected in sequence.

[0020] Furthermore, the HUBCS includes a Concat module, a channel attention module, and a spatial attention module;

[0021] The feature maps output from the first four layers of the encoder are merged by the Concat module and then channel-adjusted by a 1×1 convolutional block to obtain a completely new feature map.

[0022] Then it is divided into 4 branches. Each branch first passes through the channel attention module, and its output is multiplied by the feature map input to the corresponding layer of the decoder. The output of the multiplication is then passed through the spatial attention module, and then multiplied by the feature map input to the corresponding layer of the decoder. The output of the spatial attention module is then concatenated with the feature map output by the channel attention module through a 1×1 convolution block and then output to the corresponding layer of the decoder.

[0023] Furthermore, the residual CBAM module combines channel attention and spatial attention and introduces a residual mechanism.

[0024] Furthermore, the loss function used during the training of the improved network NxUnet is:

[0025] JDice=0.25×Jaccard+0.75×SoftDiceLoss

[0026]

[0027] Where: JDice represents the loss function; y is the binary mask of the real image; ∑ is the binary mask for the predicted image; ∈ is the smoothing factor; ∑ i This represents the summation of all pixels in the mask.

[0028] Furthermore, the improved network NxUnet uses the Dice coefficient and MIOU coefficient as evaluation metrics:

[0029]

[0030] Where TP represents a true negative, FN represents a true negative, and FN represents a false negative.

[0031] The beneficial effects of this invention are:

[0032] 1. Through a custom HUBCS module, we implemented an innovative method for processing feature maps at each layer. This module not only fuses feature maps but also precisely selects and enhances the most informative channels and features through a weighted attention mechanism. This approach not only improves the model's representational power but also effectively prevents information loss. In particular, we introduced residual connections, a mechanism that not only ensures the integrity of the original data but also combines it with processed feature data to further enrich and detail the model's understanding of the data. This comprehensive approach not only makes the model more adaptable to complex data structures but also improves its performance and robustness across various tasks.

[0033] 2. By replacing the basic convolutional blocks in the traditional U-Net network with improved serpentine dynamic convolution, the number of model parameters is reduced while improving the model's generalization ability. This convolution method can dynamically adjust the shape and parameters of the convolution kernel, more flexibly adapting to changes in input features, thereby improving feature extraction efficiency and model adaptability.

[0034] 3. Residual CBAM modules are added to the skip connections of each layer to extract key information. The residual CBAM module combines channel attention and spatial attention mechanisms, adaptively adjusting the weights of feature maps to highlight important information and suppress irrelevant features, thereby enhancing the correlation between features at different scales and the effectiveness of information transfer. To prevent information loss, residual connections are added after the CBAM module to obtain the original information. These improvements not only enhance the model's performance in handling complex image tasks but also further optimize the feature fusion process, ensuring that the model maintains high computational efficiency while possessing stronger robustness and accuracy. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the crack segmentation method based on the improved Unet full-view feature fusion according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of a classic Unet network structure according to an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the improved network NxUnet structure according to an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of the basic convolutional layer structure in the classic Unet network according to an embodiment of the present invention.

[0039] Figure 5 This is a schematic diagram of the improved serpentine dynamic convolutional layer structure in the improved network NxUnet of this invention.

[0040] Figure 6 This is a schematic diagram of the multi-feature fusion attention module HUBCS in the improved network NxUnet according to an embodiment of the present invention.

[0041] Figure 7 This is a schematic diagram of the test results of an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further explained below with reference to the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, this embodiment of the invention provides a crack segmentation method based on improved Unet with full-view feature fusion, comprising the following steps:

[0044] S101. Collect image datasets and perform preprocessing.

[0045] This invention utilizes Khanhha's image dataset. The images in the collected dataset require preprocessing to suit the model's needs. Specifically, this includes:

[0046] (1) Adjust the size of each image in the image dataset to 224 pixels × 224 pixels;

[0047] (2) Convert the image dataset with adjusted image size into tensor format and perform image brightness and contrast enhancement;

[0048] (3) Divide the converted image dataset into a training set and a test set in a 9:1 ratio.

[0049] S102. Construct an improved network NxUnet based on the UNet network, using an improved serpentine dynamic convolutional layer to replace the basic convolutional layer in the UNet network; add a residual CBAM module to the skip connections of each layer of the encoder and decoder; introduce a multi-feature fusion attention module HUBCS to process the feature maps of each layer of the encoder, fuse the feature maps of each layer of the encoder, extract features from the feature maps through channel attention mechanism and spatial attention mechanism, perform dot product operation between the feature maps of each layer of the decoder and the extracted features, and finally send the result to the corresponding layers of the decoder.

[0050] Classic Unet networks, such as Figure 2As shown, the Unet model includes an encoder-decoder structure. This structure effectively captures the contextual information and detailed features of an image. However, with the increasing resolution of industrial images and the growing demands for detection, existing Unet models face several challenges in processing high-resolution industrial images. The main problems include high computational complexity, large memory footprint, and slow segmentation speed.

[0051] This invention constructs an improved Unet network, NxUnet, as described in the embodiments of the present invention. Figure 3 As shown, it is based on the classic UNet network and still adopts an encoder-decoder structure. The basic convolutional layers are replaced by serpentine dynamic convolutional layers. The encoder performs downsampling sequentially using 5 layers of improved serpentine dynamic convolutional layers; the decoder performs upsampling sequentially using 5 layers of improved serpentine dynamic convolutional layers. A residual CBAM module is added to the skip connections in each layer of the encoder and decoder to facilitate skip connections. A multi-feature fusion attention module (HUBCS) is introduced, simultaneously feeding the feature maps output from the first four layers of the encoder and the feature maps input to the corresponding layers of the decoder into the HUBCS module. Its output is then fed into the corresponding layers of the decoder. The output of the final upsampled layer is fed into a module consisting of a BatchNorm regularization layer, a ReLU activation function, and a 3×3 convolutional layer, and then passes through a Sigmoid activation function to obtain the segmentation result.

[0052] like Figure 5 As shown, the improved serpentine dynamic convolutional layer consists of DSConv, BatchNorm, Dropout, and Leak ReLU activation functions connected in sequence. Through dynamic deformation of the convolution, the region of interest is focused more on the crack region, thus obtaining more useful feature information.

[0053] The improved serpentine dynamic convolutional layer has been changed from the original two-layer structure of the basic convolutional layer to a single-layer structure. The structure of the basic convolutional layer is as follows: Figure 4 As shown in the diagram. Experiments revealed that while the segmentation accuracy didn't significantly increase with the two-layer structure, the model training time increased by half. The ReLU activation function was replaced with Leaky ReLU, which inherits the advantages of ReLU and allows a small non-zero gradient when the input is less than zero. This helps maintain gradient flow in the network and avoid gradient vanishing, even for negative inputs. A Dropout function was then added to prevent overfitting due to increased complexity. This improved serpentine dynamic convolution dynamically adjusts the shape and parameters of the convolution kernel based on changes in input features, thus capturing feature information more flexibly. This not only improves the expressive power of the convolutional layer but also more effectively extracts details and global information from the image. Compared to fixed convolution kernels, serpentine dynamic convolution can adapt to different input features, enhancing the model's adaptability and robustness.

[0054] The Residual CBAM attention module combines channel attention and spatial attention, and incorporates a residual mechanism to avoid ignoring other useful features during weighted feature combination. Adding the Residual CBAM attention module to the skip connections in each layer of the encoder and decoder effectively enhances the correlation between features at different scales, improving the model's ability to capture important features. It not only optimizes information flow but also strengthens feature transfer between the encoder and decoder, making multi-scale feature fusion more effective.

[0055] like Figure 6 As shown, HUBCS includes a Concat module, a channel attention module, and a spatial attention module.

[0056] The feature maps output from the first four layers of the encoder are merged using a Concat module, and then channel-adjusted using a 1×1 convolutional block to obtain entirely new feature maps. These new feature maps contain complete information from each layer of the encoder, thus improving the completeness and richness of the information. The model is then divided into four branches. Each branch first extracts channel features using a Channel Attention Mechanism to ensure that the importance of different channels is fully captured. Its output is then multiplied by the feature map input to the corresponding layer of the decoder. The result of this multiplication is then used to extract spatial features using a Spatial Attention Mechanism, enabling the model to better understand the spatial relationships in the feature maps. This result is then multiplied by the feature map input to the corresponding layer of the decoder. To retain some original information, the feature map output from the Channel Attention Mechanism is then residually concatenated with the output of a 1×1 convolutional block before being output to the corresponding layer of the decoder.

[0057] S103. Train the improved network NxUnet using the preprocessed image dataset.

[0058] The NxUnet network is trained using the training set and validated using the validation set.

[0059] Set the training parameters as follows: training epochs = 200, batch size = 2, initial learning rate (lr) = 3e-3, and use Adam optimizer to update weight parameters.

[0060] The custom loss function used during training is JDice, which is a hybrid of the Jaccard loss function and the SoftDiceLoss loss function. Its mathematical expression is as follows:

[0061] JDice=0.25×Jaccard+0.75×SoftDiceLoss

[0062]

[0063] Where: y is the binary mask of the real image; ∑ is the binary mask for the predicted image; ∈ is the smoothing factor; ∑ i This represents the summation of all pixels in the mask.

[0064] The improved network NxUnet was evaluated using the Dice coefficient and MIOU coefficient.

[0065]

[0066] Where TP represents a true negative, FN represents a true negative, and FN represents a false negative.

[0067] The NxUnet network model is used for iterative training to extract image feature information. Through multiple rounds of training and optimization, the network parameters and structure are continuously adjusted to improve the model's recognition accuracy and generalization ability. During training, the model continuously learns and extracts key features from the image. Through effective feature extraction and fusion, it can more accurately identify and classify different types of pixels. After training, cross-validation and a test set are used to evaluate the model's performance, and the model with the best segmentation results is selected as the final output. In this way, not only is the accuracy of image segmentation improved, but the model's applicability in different scenarios is also enhanced.

[0068] The table below shows the results of different network models on Khanhha's dataset. Figure 7 A schematic diagram of the NxUnet network results is provided.

[0069] Table 1

[0070] Unet-vgg16 0.468 0.603 UNet_Resnet_101 0.386 0.518 KBPN+PSPNet 0.524 0.654 CSBSR 0.552 0.681 KBPN+U-Net 0.455 0.598 NxUnet(our) 0.583 0.733

[0071] As can be seen from the table above, the NxUnet network achieved excellent results in both the Dice coefficient and the MIOU coefficient.

[0072] S104. Obtain the crack image to be segmented, and use the trained improved network NxUnet to generate the segmentation result image.

[0073] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and various changes or modifications can be made to these embodiments without departing from the nature and spirit of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A crack segmentation method based on full-view feature fusion using an improved Unet, characterized in that, Includes the following steps: (1) Obtain the image dataset and perform preprocessing; (2) Construct an improved network NxUnet based on the Unet network, and replace the basic convolutional layer in the UNet network with an improved snake dynamic convolutional layer; add a residual CBAM module to the skip connection of each layer of the encoder and decoder; introduce a multi-feature fusion attention module HUBCS to process the feature maps of each layer of the encoder, fuse the feature maps of each layer of the encoder, extract features from the feature maps through channel attention mechanism and spatial attention mechanism, and perform dot product operation between the feature maps of each layer of the decoder and the extracted features, and finally send the result to the corresponding layer of the decoder. The NxUnet network consists of two parts: an encoder and a decoder. The encoder is downsampled sequentially by five layers of the improved serpentine dynamic convolutional layers; the decoder is upsampled sequentially by five layers of the improved serpentine dynamic convolutional layers. The output of each layer of the encoder is fed into the corresponding layer of the decoder after being skipped through the residual CBAM module. The feature maps output from the first four layers of the encoder and the feature maps input from the corresponding layers of the decoder are also simultaneously fed into the HUBCS module, and their outputs are respectively fed into the corresponding layers of the decoder. The output of the last upsampled layer is fed into a module consisting of a BatchNorm regularization layer, a ReLU activation function, and a 3×3 convolutional layer, and then passes through a Sigmoid activation function to obtain the final segmentation result. The improved serpentine dynamic convolutional layer comprises DSConv, BatchNorm, Dropout functions, and LeakReLU activation function connected in sequence; The HUBCS includes a Concat module, a channel attention module, and a spatial attention module; The feature maps output from the first four layers of the encoder are merged by the Concat module and then channel-adjusted by a 1×1 convolutional block to obtain a completely new feature map. Then it is divided into 4 branches. Each branch first passes through the channel attention module, and its output is multiplied by the feature map of the corresponding layer input of the decoder before being output. The result after dot product is passed through the spatial attention module, then dot productd with the feature map input to the corresponding layer of the decoder, and then residually concatenated with the output of the feature map output by the channel attention module after passing through a 1×1 convolution block before being output to the corresponding layer of the decoder. The residual CBAM module combines channel attention and spatial attention, and introduces a residual mechanism; (3) The improved network NxUnet is trained using the preprocessed image dataset; (4) Obtain the crack image to be segmented and use the trained improved network NxUnet to generate the segmentation result image.

2. The crack segmentation method based on improved Unet with full-view feature fusion according to claim 1, characterized in that, The preprocessing includes: (1.1) Adjust the size of each image in the image dataset to 224 pixels × 224 pixels; (1.2) Convert the image dataset with adjusted image size into tensor format and perform image contrast enhancement; (1.3) Divide the image dataset after data conversion into a training set and a test set in a 9:1 ratio.

3. The crack segmentation method based on improved Unet with full-view feature fusion according to claim 1, characterized in that, The loss function used during the training of the improved network NxUnet is: JDice = 0.25 Jaccard + 0.75 SoftDiceLoss Where: JDice represents the loss function; y is the binary mask of the real image; For predicting the binary mask of the image; ∑ is the smoothing factor; i This represents the summation of all pixels in the mask.

4. The crack segmentation method based on improved Unet with full-view feature fusion according to claim 1, characterized in that, The improved network NxUnet uses the Dice coefficient and MIOU coefficient as evaluation metrics: Where TP represents a true negative, FN represents a true negative, and FN represents a false negative.

Images