A network architecture for crack image segmentation and its training method and segmentation method
Through the network architecture of self-attention calculation and dynamic weight adjustment, the problem of complex background and category imbalance in crack image segmentation is solved, and high-precision narrow and long crack segmentation effect is achieved.
Patent Information
- Application Number
- CN202310485486.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2025-08-15
- Estimated Expiration
- 2043-04-28
AI Technical Summary
The existing crack image segmentation technology is difficult to accurately locate narrow and long cracks in complex backgrounds, and is affected by the limitations of traditional standard convolution and category imbalance problems, resulting in insufficient model learning ability.
The network architecture including encoder, bottleneck layer and decoder is adopted, and self-attention calculation and bar pooling are combined with transformer module to enhance global semantic information acquisition, and the sensitivity to pixel points in the crack area is enhanced through Dice and the weight cross entropy mixed loss function, and the weight is dynamically adjusted to promote model convergence.
It improves the accuracy and robustness of crack image segmentation, effectively deals with category imbalance problem, and improves the model's narrow and long crack characterization ability and segmentation accuracy in complex backgrounds.
Smart Images
Figure CN116542991B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a network architecture for crack image segmentation, a training method thereof, and a segmentation method. Background Art
[0002] Image segmentation is one of the three fundamental tasks in computer vision, with widespread applications in medicine, biometrics, and other fields. Image segmentation aims to detect objects of interest and assign corresponding semantic labels to each pixel. In recent years, with the continuous development of convolutional neural networks, their performance has continued to improve. However, the network model architecture has also become increasingly large and layered, requiring increasingly high computing power and training datasets. In real-world applications, this is clearly unacceptable, as efficiency and cost are always paramount.
[0003] In recent years, convolutional neural network architectures have become the mainstream architecture for crack image segmentation. Thanks to their powerful feature extraction capabilities and ability to represent high-dimensional data, deep learning-based methods can learn features that are difficult to obtain manually. This not only reduces the cost of manual feature extraction but also significantly improves detection accuracy. In the early days of deep learning, simple convolutions were used to extract features. For example, the DeepCrack network employed multiple convolutional blocks to extract features from the sample under test and then fused feature maps of different scales to achieve segmentation. However, due to the irregularity of cracks and their large variations in shape and size, simple convolutions were no longer sufficient. For example, the fully convolutional network (FCN) pioneered an end-to-end segmentation approach, while the UNet constructed a complete encoder-decoder U-shaped structure and employed skip connections to supplement detailed features. Researchers in this field often improve upon these basic network structures. For example, SegNet, also based on an encoder-decoder architecture, records the maximum pooling index during pooling within its encoder. During upsampling, the maximum pooling index of the corresponding encoder is used to perform the upsampling. EffificientNet also proposed a scaling factor to control the balance between network depth, width, and resolution.
[0004] Although the above methods have made significant progress, they are still limited by the following shortcomings: (1) Cracks are widely present in various life scenes, which also leads to complex backgrounds in crack images. It is difficult to accurately locate cracks in complex backgrounds. (2) Limited by traditional standard convolution, its local features are often confined to square areas. Cracks usually have narrow and long characteristics. Therefore, the features extracted by standard convolution will inevitably be mixed with other interference information. (3) In existing public datasets, there is a serious class imbalance, which leads to the classification results being biased towards large categories. In the binary classification problem of cracks, this directly causes the model's learning ability to be greatly reduced. Therefore, how to solve the class imbalance is still a difficult problem that needs to be solved urgently. Summary of the Invention
[0005] The purpose of the present invention is to provide a network architecture for crack image segmentation and its training method and segmentation method, so as to solve the problems raised in the above background technology and achieve the desired effect.
[0006] The object of the present invention is achieved as follows: a network architecture for crack image segmentation, characterized in that: the network architecture includes a segmentation network model for processing images and a loss function model for training the segmentation network model;
[0007] The segmentation network model includes an encoder, a bottleneck layer and a decoder, the encoder includes multiple first convolution blocks and a downsampling module, and the downsampling module obtains local detail feature maps of different resolutions and low-resolution local high-level semantic feature maps;
[0008] The bottleneck layer includes a transformer module, a strip pooling module, and a second convolutional block. The transformer module performs self-attention calculations to obtain global semantic information and complete the complementarity of local and global features.
[0009] The decoder includes an upsampling module, an attention module, and a third convolution block. The upsampling module upsamples the image, makes up for the lost detail features, and gradually restores the image to its original size.
[0010] The loss function model adopts a Dice and weighted cross entropy mixed loss function to enhance the sensitivity of pixels in the crack area.
[0011] Preferably, the transformer module specifically operates as follows:
[0012] The transformer module accepts a one-dimensional Token sequence and reshapes the input features in order to process 2D features. The picture shows a series of flattened slices The size of each slice is P×P, where N represents the number of slices of the feature map. is the input feature map; C represents the number of channels of the feature map, H and W represent the height and width of the input feature map respectively; for Slices after reshaping;
[0013] In order to preserve the spatial information of the slice, a learnable position encoding E is selected pos Embedded into a slice that is linearly mapped to D dimensions as follows:
[0014]
[0015] in, represents the trainable linear mapping part, represents a learnable positional encoding.
[0016] Preferably, the downsampling module includes a maximum pooling and a first convolution block of 3×3 with a step size of 1 and a padding of 1, and the number of the first convolution blocks is two.
[0017] Preferably, the upsampling module specifically operates as follows:
[0018] Given an input feature map After parallel upsampling, we get Then fuse the two feature maps to get the final output:
[0019]
[0020] Where C represents the number of channels of the feature map, H and W represent the height and width of the input feature map respectively; Represents a point-by-point addition operation.
[0021] Preferably, the attention module specifically operates as follows:
[0022] Given two input feature maps, a skip connection from the encoder One from the decoder upsampling module The two feature maps are convolved by 1×1 to reduce the channel to 1 / 2, and the size remains unchanged, which are and Then, by point-by-point addition, we can obtain After this, X add The spatial attention map is generated by sequentially passing through the ReLU activation function, 1×1 convolution block and Sigmoid activation function.
[0023]
[0024] Among them, σ rReLU activation function, the attention value is between 0-1, and W sp To skip-connect feature maps of the encoder Weighted, we get
[0025]
[0026] Preferably, the second convolution block adopts a 1×1 convolution with a step size of 1 and a padding of 1, and the third convolution block adopts a 3×3 convolution with a step size of 1 and a padding of 1.
[0027] Preferably, a training method for a segmentation network model is characterized in that the method comprises the following steps:
[0028] Step (1): Establish a loss function model. The loss function model adopts a hybrid form of Dice loss and cross entropy loss CE, and sets the necessary parameters of the loss function model and the necessary parameters of the segmentation model. The loss function model is expressed as:
[0029] L=Dice+w·CE
[0030] Among them, w represents the weight of cross entropy loss;
[0031] Step (2): Calculate weights and adjust the loss function model iteration interval N gap ;
[0032]
[0033] Among them, N epoch Indicates the total number of Epochs in training, N max Indicates the maximum number of iterations in training;
[0034] Step (3): The training starts iterating and the loss function value L of the current iteration cycle is calculated according to the input feature map n ;
[0035] Step (4): Set the number of iterations to n and calculate the average loss L of the iterations in the number of iterations mean :
[0036]
[0037] Where n represents the current number of iterations;
[0038] Step (5): Determine the saturation iteration, and perform back propagation on the segmentation model of the segmentation result, and adjust the parameter values of each module in the segmentation model; when the number of iterations is greater than the iteration round, it means that the segmentation network model training tends to be saturated, then proceed to step (6); when the number of iterations is less than the iteration round, repeat step (3);
[0039] Step (6): Determine the current loss function value L n With the average loss L mean The size of L n >L mean , then M down =M down +1; among them, M down Indicates L in an interval n <L mean the number of times;
[0040] When L n <L mean , then M up =M up +1, M up Indicates L in an interval n >L mean the number of times;
[0041] Step (7): Determine M up -M down The size of M up -M down <0, then M up and M down Return to zero, and the initial value of w is taken according to experience;
[0042] When M up -M down >0, judge (M up -M down ) and the size of Thr; when (M up -M down )>Thr, the model is judged to be positively converged, and the cross entropy weight w of the loss function model is updated at the same time:
[0043]
[0044] Wherein, Thr is (M up -M down )The mean of the differences;
[0045] When (M up -M down )<Thr, the model is judged to have negative convergence, then
[0046] Step (8): Repeat steps (4) to (7) until the maximum number of iterations N is reached. max .
[0047] A method for segmenting a crack image, characterized in that the method comprises the following steps:
[0048] Step S1: Input the original image into the encoder, and obtain the first feature map with the original image resolution and 64 channels through the convolution block in the encoder;
[0049] Step S2: Use the downsampling module to perform feature extraction on the original image, and perform feature extraction on the first feature map to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 128;
[0050] Step S3: Use the downsampling module to downsample the second feature map to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 256;
[0051] Step S4: Use a downsampling module to downsample the third feature map to obtain a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 512;
[0052] Step S5: using a downsampling module to downsample the fourth feature map to obtain a fifth feature map with a resolution of 1 / 16 of the original image and a channel number of 1024;
[0053] Step S6: performing a convolution operation on the fifth feature map to obtain a feature map with a resolution of 1 / 16 of the original image and a number of channels of 768, aligning and serializing to obtain a feature sequence, and passing the feature sequence through a continuous self-attention module to obtain a sixth feature map with a resolution of 1 / 16 of the original image and a number of channels of 768;
[0054] Step S7: Convolutionally reduce the sixth feature map to 512 channels, use the strip pooling module to perform two parallel pooling branches vertically and horizontally to obtain the pooled feature map, convolve the feature map, expand the pooled feature map to 1 / 16 of the original resolution in the horizontal and vertical directions, and add and fuse the two feature maps point by point to finally obtain the seventh feature map with 1 / 16 of the original resolution and 512 channels;
[0055] Step S8: The seventh feature map is subjected to hybrid upsampling to obtain an eighth feature map with a resolution of 1 / 8 of the original image and a channel number of 512;
[0056] Step S9: The fourth feature map is connected to the eighth feature map through a jump connection Figure 1 After passing through the attention module, the ninth feature map with a resolution of 1 / 8 of the original image and a channel number of 512 is generated;
[0057] Step S10: The ninth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a tenth feature map with a resolution of 1 / 8 of the original image and a channel number of 256;
[0058] Step S11: The tenth feature map is subjected to hybrid upsampling to obtain an eleventh feature map with a resolution of 1 / 4 of the original image and a channel number of 256;
[0059] Step S12: The third feature map is connected to the eleventh feature map via a jump connection Figure 1 After passing through the attention module, the twelfth feature map with a resolution of 1 / 4 of the original image and a channel number of 256 is generated;
[0060] Step S13: The twelfth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a thirteenth feature map with a resolution of 1 / 4 of the original image and a channel number of 128;
[0061] Step S14: The thirteenth feature map is subjected to hybrid upsampling to obtain a fourteenth feature map with a resolution of 1 / 2 of the original image and a channel number of 128;
[0062] Step S15: The second feature map is connected to the fourteenth feature map through a jump connection Figure 1 After passing through the attention module, the fifteenth feature map with a resolution of 1 / 2 of the original image and a channel number of 128 is generated;
[0063] Step S16: The fifteenth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a sixteenth feature map with a resolution of 1 / 2 of the original and a channel number of 64;
[0064] Step S17: The sixteenth feature map is subjected to hybrid upsampling to obtain a seventeenth feature map having the original image resolution and 64 channels;
[0065] Step S18: The first feature map is connected to the seventeenth feature map through a jump connection Figure 1 After passing through the attention module, the original resolution is obtained, and the number of channels is 64, the eighteenth feature map;
[0066] Step S19: Send the eighteenth feature map into three consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain the crack image segmentation result.
[0067] Compared with the existing technology, the present invention has the following improvements and advantages: 1. By setting up a transformer module, self-attention calculation is performed to obtain global semantic information, complete the complementation of local and global features, and enhance the expression ability of narrow and long cracks through the strip pooling module, while improving the representation ability of the narrow and long characteristics of the segmentation network model and handling the category imbalance problem.
[0068] 2. By setting dynamic weights, the sensitivity of the segmentation network model to the pixels in the crack area is enhanced. When the segmentation network model tends to saturation, the weights are adaptively adjusted dynamically to promote rapid convergence of the model and improve the model segmentation accuracy while ensuring robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 This is the overall network architecture diagram of the present invention.
[0070] Figure 2 This is a diagram of the segmentation network model architecture of the present invention.
[0071] Figure 3 Schematic diagram of the transformer module in the network architecture of the present invention.
[0072] Figure 4 Schematic diagram of the strip pooling module in the network architecture of the present invention.
[0073] Figure 5 Schematic diagram of the hybrid upsampling module at the decoding end in the network architecture of the present invention.
[0074] Figure 6 Schematic diagram of the attention module at the decoding end in the network architecture of the present invention.
[0075] Figure 7 This is a flow chart for calculating the loss function in the present invention.
[0076] Figure 8 This is the segmentation result diagram of the network architecture of the present invention on the public dataset DeepCrack. DETAILED DESCRIPTION
[0077] The present invention is further summarized below with reference to the accompanying drawings.
[0078] See Figure 1-5 , a network architecture for crack image segmentation, the network architecture comprising a segmentation network model for processing images and a loss function model for training the segmentation network model;
[0079] The segmentation network model includes an encoder, a bottleneck layer, and a decoder. The encoder includes multiple first convolution blocks and a downsampling module. The downsampling module obtains local detail feature maps of different resolutions and low-resolution local high-level semantic feature maps.
[0080] The bottleneck layer includes a transformer module, a strip pooling module, and a second convolutional block. The transformer module performs self-attention calculations to obtain global semantic information and complement local and global features.
[0081] The decoder consists of an upsampling module, an attention module, and a third convolutional block. The upsampling module upsamples the image, compensates for the lost detail features, and gradually restores the image to its original size.
[0082] The loss function model adopts a mixed loss function of Dice and weighted cross entropy to enhance the sensitivity of pixels in the crack area.
[0083] Furthermore, the transformer module operates as follows:
[0084] like Figure 3 As shown, the transformer module accepts a one-dimensional Token sequence and reshapes the input features in order to process 2D features. The picture shows a series of flattened slices The size of each slice is P×P, where N represents the number of slices of the feature map. is the input feature map; C represents the number of channels of the feature map, H and W represent the height and width of the input feature map respectively; for Slices after reshaping;
[0085] In order to preserve the spatial information of the slice, a learnable position encoding E is selected pos Embedded into a slice that is linearly mapped to D dimensions as follows:
[0086]
[0087] in, represents the trainable linear mapping part, represents a learnable positional encoding.
[0088] Furthermore, the downsampling module includes a maximum pooling and a first convolution block of 3×3 with a stride of 1 and a padding of 1, and the number of the first convolution blocks is two.
[0089] Furthermore, the upsampling module operates as follows:
[0090] Given an input feature map After parallel upsampling, we get Then fuse the two feature maps to get the final output:
[0091]
[0092] Where C represents the number of channels of the feature map, H and W represent the height and width of the input feature map respectively; Represents a point-by-point addition operation.
[0093] The specific operations of the attention module are as follows:
[0094] Given two input feature maps, a skip connection from the encoder One from the decoder upsampling module The two feature maps are convolved by 1×1 to reduce the channel to 1 / 2, and the size remains unchanged, which are and Then, by point-by-point addition, we can obtain After this, X add The spatial attention map is generated by sequentially passing through the ReLU activation function, 1×1 convolution block and Sigmoid activation function.
[0095]
[0096] Among them, σ r ReLU activation function, the attention value is between 0-1, and W sp To skip-connect feature maps of the encoder Weighted, we get
[0097]
[0098] like Figure 7 As shown, a training method for a segmentation network model includes the following steps:
[0099] Step (1): Establish a loss function model. The loss function model adopts a hybrid form of Dice loss and cross entropy loss CE, and sets the necessary parameters of the loss function model and the necessary parameters of the segmentation model. The loss function model is expressed as:
[0100] L = Dice + w·CE;
[0101] Among them, w represents the weight of cross entropy loss;
[0102] Step (2): Calculate weights and adjust the loss function model iteration interval N gap ;
[0103]
[0104] Among them, N epoch Indicates the total number of Epochs in training, N max Indicates the maximum number of iterations in training;
[0105] Step (3): The training starts iterating and the loss function value L of the current iteration cycle is calculated according to the input feature map n ;
[0106] Step (4): Set the number of iterations to n and calculate the average loss L of the iterations in the number of iterations mean :
[0107]
[0108] Where n represents the current number of iterations;
[0109] Step (5): Determine the saturation iteration, and perform back propagation on the segmentation model of the segmentation result, and adjust the parameter values of each module in the segmentation model; when the number of iterations is greater than the iteration round, it means that the segmentation network model training tends to be saturated, then proceed to step (6); when the number of iterations is less than the iteration round, repeat step (3);
[0110] Step (6): Determine the current loss function value L n With the average loss L mean The size of L n >L mean , then M down =M down +1; among them, M down Indicates L in an interval n <L mean the number of times;
[0111] When L n <L mean , then M up =M up +1, M up Indicates L in an interval n >L mean the number of times;
[0112] Step (7): Determine M up -M down The size of M up -M down <0, then M up and M down Return to zero, and the initial value of w is taken according to experience;
[0113] When M up -M down >0, judge (M up -M down ) and the size of Thr; when (M up -M down )>Thr, the model is judged to be positively converged, and the cross entropy weight w of the loss function model is updated at the same time:
[0114]
[0115] Wherein, Thr is (M up -M down )The mean of the differences;
[0116] When (M up -M down )<Thr, the model is judged to have negative convergence, then
[0117] Step (8): Repeat steps (4) to (7) until the maximum number of iterations N is reached. max .
[0118] The present invention adaptively adjusts the weight parameters to make them as close to the optimal value as possible, and the constantly changing weight values make the model have stronger generalization ability.
[0119] like Figure 2 As shown, a method for segmenting a crack image comprises the following steps:
[0120] Step S1: Input the original image into the encoder, and obtain the first feature map with the original image resolution and 64 channels through the convolution block in the encoder;
[0121] Step S2: Use the downsampling module to perform feature extraction on the original image, and perform feature extraction on the first feature map to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 128;
[0122] Step S3: Use the downsampling module to downsample the second feature map to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 256;
[0123] Step S4: Use a downsampling module to downsample the third feature map to obtain a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 512;
[0124] Step S5: using a downsampling module to downsample the fourth feature map to obtain a fifth feature map with a resolution of 1 / 16 of the original image and a channel number of 1024;
[0125] Step S6: performing a convolution operation on the fifth feature map to obtain a feature map with a resolution of 1 / 16 of the original image and a number of channels of 768, aligning and serializing to obtain a feature sequence, and passing the feature sequence through a continuous self-attention module to obtain a sixth feature map with a resolution of 1 / 16 of the original image and a number of channels of 768;
[0126] Step S7: Convolutionally reduce the sixth feature map to 512 channels, use the strip pooling module to perform two parallel pooling branches vertically and horizontally to obtain the pooled feature map, convolve the feature map, expand the pooled feature map to 1 / 16 of the original resolution in the horizontal and vertical directions, and add and fuse the two feature maps point by point to finally obtain the seventh feature map with 1 / 16 of the original resolution and 512 channels;
[0127] Step S8: The seventh feature map is subjected to hybrid upsampling to obtain an eighth feature map with a resolution of 1 / 8 of the original image and a channel number of 512;
[0128] Step S9: The fourth feature map is connected to the eighth feature map through a jump connection Figure 1 After passing through the attention module, the ninth feature map with a resolution of 1 / 8 of the original image and a channel number of 512 is generated;
[0129] Step S10: The ninth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a tenth feature map with a resolution of 1 / 8 of the original image and a channel number of 256;
[0130] Step S11: The tenth feature map is subjected to hybrid upsampling to obtain an eleventh feature map with a resolution of 1 / 4 of the original image and a channel number of 256;
[0131] Step S12: The third feature map is connected to the eleventh feature map via a jump connection Figure 1 After passing through the attention module, the twelfth feature map with a resolution of 1 / 4 of the original image and a channel number of 256 is generated;
[0132] Step S13: The twelfth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a thirteenth feature map with a resolution of 1 / 4 of the original image and a channel number of 128;
[0133] Step S14: The thirteenth feature map is subjected to hybrid upsampling to obtain a fourteenth feature map with a resolution of 1 / 2 of the original image and a channel number of 128;
[0134] Step S15: The second feature map is connected to the fourteenth feature map through a jump connection Figure 1 After passing through the attention module, the fifteenth feature map with a resolution of 1 / 2 of the original image and a channel number of 128 is generated;
[0135] Step S16: The fifteenth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a sixteenth feature map with a resolution of 1 / 2 of the original and a channel number of 64;
[0136] Step S17: The sixteenth feature map is subjected to hybrid upsampling to obtain a seventeenth feature map having the original image resolution and 64 channels;
[0137] Step S18: The first feature map is connected to the seventeenth feature map through a jump connection Figure 1 After passing through the attention module, the original resolution is obtained, and the number of channels is 64, the eighteenth feature map;
[0138] Step S19: Send the eighteenth feature map into three consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain the crack image segmentation result.
[0139] The proposed network framework efficiently extracts both global semantic information and local details from images, effectively characterizes narrow cracks in complex backgrounds, and can handle class imbalance. Compared to recent mainstream networks, the designed network architecture achieves higher crack segmentation accuracy.
[0140] To verify the accuracy and efficiency of the network designed in this paper, the model was trained and validated on the widely used DeepCrack dataset. DeepCrack contains 537 images of cracks on surfaces from various scenes and their corresponding pixel-level labels. The images have a resolution of 544×384 pixels. 300 of these images were used for training, and the remaining 237 images served as the test set. After training, the UNet, Swin-UNet, TransUNet, and DeepCrack networks were selected as comparison models.
[0141] Table 1 is a comparison table of various models
[0142] Model F1 IoU Pr Re Acc DeepCrack 0.7473 0.6132 0.6342 0.9604 0.9737 U-Net 0.7887 0.6832 0.8852 0.7504 0.9837 Swin-UNet 0.7789 0.6513 0.8691 0.7328 0.9819 TransUNet 0.8338 0.7243 0.9034 0.7902 0.9865 CrackNet 0.8425 0.7406 0.8690 0.8424 0.9872
[0143] The network designed by this invention achieved an F1 score of 84.25% and a Re score of 84.24%, both outperforming other networks. Table 1 shows that compared to UNet, this segmentation network model achieves a 5.38% improvement in F1 and a 9.2% improvement in Re. Furthermore, compared to the pure transformer model Swin-UNet, while processing speed is slightly slower than that of Swin-UNet, which reduces computational complexity from exponential to linear, there is a significant improvement in accuracy.
[0144] From the experimental results, no matter from which perspective, it is fully demonstrated that the network model proposed in this invention is superior in the crack image segmentation task.
[0145] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.
Claims
1. A network architecture for crack image segmentation, characterized by: The network architecture includes a segmentation network model for processing images and a loss function model for training the segmentation network model; The segmentation network model includes an encoder, a bottleneck layer and a decoder, the encoder includes multiple first convolution blocks and a downsampling module, and the downsampling module obtains local detail feature maps of different resolutions and low-resolution local high-level semantic feature maps; The bottleneck layer includes a transformer module, a strip pooling module, and a second convolutional block. The transformer module performs self-attention calculations to obtain global semantic information and complete the complementarity of local and global features. The decoder includes an upsampling module, an attention module, and a third convolution block. The upsampling module upsamples the image, makes up for the lost detail features, and gradually restores the image to its original size. The attention module specifically operates as follows: Given two input feature maps, a skip connection from the encoder One from the decoder upsampling module The two feature maps are convolved by 1×1 to reduce the channel to 1 / 2, and the size remains unchanged, which are and Then, by point-by-point addition, we can obtain After this, X add The spatial attention map is generated by sequentially passing through the ReLU activation function, 1×1 convolution block and Sigmoid activation function. Among them, σ r ReLU activation function, the attention value is between 0-1, and W sp To skip-connect feature maps of the encoder Weighted, we get The loss function model adopts a Dice and weighted cross entropy mixed loss function to enhance the sensitivity of pixels in the crack area.
2. The network architecture for crack image segmentation according to claim 1, characterized in that: The transformer module specifically operates as follows: The transformer module accepts a one-dimensional Token sequence and reshapes the input features in order to process 2D features. The picture shows a series of flattened slices The size of each slice is P×P, where N represents the number of slices of the feature map. is the input feature map; C represents the number of channels of the feature map, H and W represent the height and width of the input feature map respectively; for Slices after reshaping; In order to preserve the spatial information of the slice, a learnable position encoding E is selected pos Embedded into a slice that is linearly mapped to D dimensions as follows: in, represents the trainable linear mapping part, represents a learnable positional encoding.
3. The network architecture for crack image segmentation according to claim 1, characterized in that: The downsampling module includes a maximum pooling and a first convolution block with a 3×3 step size of 1 and a padding of 1, and the number of the first convolution blocks is two.
4. The network architecture for crack image segmentation according to claim 1, characterized in that: The upsampling module specifically operates as follows: Given an input feature map After parallel upsampling, we get Then fuse the two feature maps to get the final output: Where C represents the number of channels of the feature map, H and W represent the height and width of the input feature map respectively; Represents a point-by-point addition operation.
5. The network architecture for crack image segmentation according to claim 1, characterized in that: The second convolution block uses a 1×1 convolution with a stride of 1 and a padding of 1, and the third convolution block uses a 3×3 convolution with a stride of 1 and a padding of 1.
6. A training method for a segmentation network model, characterized by: The training method comprises a network architecture for crack image segmentation according to any one of claims 1 to 5; the method comprises the following steps: Step (1): Establish a loss function model. The loss function model adopts a hybrid form of Dice loss and cross entropy loss CE, and sets the necessary parameters of the loss function model and the necessary parameters of the segmentation model. The loss function model is expressed as: L=Dice+w·CE Among them, w represents the weight of cross entropy loss; Step (2): Calculate weights and adjust the loss function model iteration interval N gap ; Among them, N epoch Indicates the total number of Epochs in training, N max Indicates the maximum number of iterations in training; Step (3): The training starts iterating and the loss function value L of the current iteration cycle is calculated according to the input feature map n ; Step (4): Set the number of iterations to n and calculate the average loss L of the iterations in the number of iterations mean : Where n represents the current number of iterations; Step (5): Determine the saturation iteration, and perform back propagation on the segmentation model of the segmentation result, and adjust the parameter values of each module in the segmentation model; when the number of iterations is greater than the iteration round, it means that the segmentation network model training tends to be saturated, then proceed to step (6); when the number of iterations is less than the iteration round, repeat step (3); Step (6): Determine the current loss function value L n With the average loss L mean The size of L n >L mea ,but Among them, M down Indicates L in an interval n <L mean the number of times; When L n <L mean , then M up =M up +1, M up Indicates L in an interval n >L mean the number of times; Step (7): Determine M up -M down The size of M up -M down <0, then M up and M down Return to zero, and the initial value of w is taken according to experience; When M up -M down >0, judge (M up -M down ) and the size of Thr; when (M up -M down )>Thr, the model is judged to be positively converged, and the cross entropy weight w of the loss function model is updated at the same time: Wherein, Thr is (M up -M down )The mean of the differences; When (M up -M down )<Thr, the model is judged to have negative convergence, then Step (8): Repeat steps (4) to (7) until the maximum number of iterations N is reached. max .
7. A method for segmenting a crack image, characterized in that: The segmentation method comprises a network architecture for crack image segmentation according to any one of claims 1 to 5; the method comprises the following steps: Step S1: Input the original image into the encoder, and obtain the first feature map with the original image resolution and 64 channels through the convolution block in the encoder; Step S2: Use the downsampling module to perform feature extraction on the original image, and perform feature extraction on the first feature map to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 128; Step S3: Use the downsampling module to downsample the second feature map to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 256; Step S4: Use a downsampling module to downsample the third feature map to obtain a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 512; Step S5: using a downsampling module to downsample the fourth feature map to obtain a fifth feature map with a resolution of 1 / 16 of the original image and a channel number of 1024; Step S6: performing a convolution operation on the fifth feature map to obtain a feature map with a resolution of 1 / 16 of the original image and a number of channels of 768, aligning and serializing to obtain a feature sequence, and passing the feature sequence through a continuous self-attention module to obtain a sixth feature map with a resolution of 1 / 16 of the original image and a number of channels of 768; Step S7: Convolutionally reduce the sixth feature map to 512 channels, use the strip pooling module to perform two parallel pooling branches vertically and horizontally to obtain the pooled feature map, convolve the feature map, expand the pooled feature map to 1 / 16 of the original resolution in the horizontal and vertical directions, and add and fuse the two feature maps point by point to finally obtain the seventh feature map with 1 / 16 of the original resolution and 512 channels; Step S8: The seventh feature map is subjected to hybrid upsampling to obtain an eighth feature map with a resolution of 1 / 8 of the original image and a channel number of 512; Step S9: The fourth feature map passes through the attention module together with the eighth feature map via a skip connection to generate a ninth feature map with a resolution of 1 / 8 of the original image and a channel number of 512; Step S10: The ninth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a tenth feature map with a resolution of 1 / 8 of the original image and a channel number of 256; Step S11: The tenth feature map is subjected to hybrid upsampling to obtain an eleventh feature map with a resolution of 1 / 4 of the original image and a channel number of 256; Step S12: The third feature map passes through the attention module together with the eleventh feature map via a skip connection to generate a twelfth feature map with a resolution of 1 / 4 of the original image and a channel number of 256; Step S13: The twelfth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a thirteenth feature map with a resolution of 1 / 4 of the original image and a channel number of 128; Step S14: The thirteenth feature map is subjected to hybrid upsampling to obtain a fourteenth feature map with a resolution of 1 / 2 of the original image and a channel number of 128; Step S15: The second feature map passes through the attention module together with the fourteenth feature map via a skip connection to generate a fifteenth feature map with a resolution of 1 / 2 of the original image and a channel number of 128; Step S16: The fifteenth feature map is fed into two consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain a sixteenth feature map with a resolution of 1 / 2 of the original and a channel number of 64; Step S17: The sixteenth feature map is subjected to hybrid upsampling to obtain a seventeenth feature map having the original image resolution and 64 channels; Step S18: The first feature map passes through the attention module together with the seventeenth feature map through a skip connection to obtain the eighteenth feature map with the original resolution and 64 channels; Step S19: Send the eighteenth feature map into three consecutive 3×3 convolutions with a step size of 1 and a padding of 1 to obtain the crack image segmentation result.
Citation Information
Patent Citations
Image processing method, device and equipment and readable storage medium
CN115330813A
Face attribute recognition method and system based on self-distillation Transform
CN115331295A