Bridge crack image semantic segmentation method based on improved SegFormer

Abstract

The application provides a bridge crack image semantic segmentation method based on an improved SegFormer network. The application constructs a three-way parallel segmentation feature architecture, processes multi-scale features output by an encoder through a multi-scale feature fusion module, a global path and a local path respectively, and obtains three-way segmentation features of multi-scale fusion features, global path features and local path features; wherein the global path enhances crack direction response through a weighted mixed multi-scale feature extraction and direction enhanced attention module, and the local path recovers crack details through layer-by-layer upsampling and low layer feature fusion. Then, through a deep feature fusion module, the three-way features are synergistically fused based on mutually independent non-competitive sigmoid gating weights and residual enhancement methods, and finally, pixel-level segmentation results of bridge crack regions and background regions are output. The application improves the integrity, continuity and boundary definition of the bridge crack segmentation results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and bridge structural health monitoring technology, specifically a semantic segmentation method for bridge crack images based on an improved SegFormer. Background Technology

[0002] As a crucial component of transportation infrastructure, the structural health of bridges directly impacts public transportation safety and service life. Cracks are a common type of apparent damage in bridge structures, primarily manifesting as long, thin, network-like, or bifurcated cracks in the bridge deck, beams, piers, and abutments. Accurately identifying and quantifying these crack areas is of great significance for bridge health monitoring, damage assessment, and maintenance decisions.

[0003] With the rapid development of computer vision and deep learning technologies, image-based automated crack detection methods have gradually become a research hotspot in the field of bridge health monitoring. Among numerous deep learning methods, semantic segmentation technology can classify each pixel in an image and output pixel-level segmentation results between the crack region and the background region, making it suitable for quantitative analysis of parameters such as crack length, width, direction, area, and connectivity. SegFormer is a semantic segmentation network based on the Transformer architecture, consisting of a hierarchical encoder and a lightweight fully convolutional MLP decoder. Its hierarchical encoder can output encoded features at multiple different scales and captures global contextual information through a self-attention mechanism, achieving excellent performance in general scene segmentation tasks. When applying SegFormer to the semantic segmentation task of bridge crack images, the multi-scale features extracted by its encoder include shallow spatial detail information and deep semantic information, theoretically capable of simultaneously taking into account the edge details and global structural features of the crack.

[0004] While SegFormer performs well in general semantic segmentation tasks, its existing architecture still suffers from insufficient multi-scale feature fusion in bridge crack image segmentation scenarios. Specifically, SegFormer's hierarchical encoder outputs multiple stages of features lack an effective information interaction mechanism. The simple concatenation or point-by-point addition of shallow spatial detail features and deep semantic features in the decoder fails to achieve efficient collaborative fusion of multi-scale information. This insufficient multi-scale feature fusion prevents the network from effectively utilizing both edge texture information in shallow features and semantic discriminative information in deep features when processing crack images, thus affecting the completeness and accuracy of the segmentation results. Furthermore, crack targets possess complex morphological features such as elongation, variable orientation, and uneven width, placing higher demands on the information complementarity of multi-scale features. Summary of the Invention

[0005] The purpose of this invention is to propose a semantic segmentation method for bridge crack images based on an improved SegFormer. By designing a multi-scale feature fusion module, the method achieves effective coordination of features at each stage of the encoder. Furthermore, it combines a direction-enhanced attention mechanism and a dual-path decoder architecture to enhance crack segmentation capabilities, thereby improving the completeness, continuity, and accuracy of pixel-level segmentation results for bridge cracks.

[0006] To achieve the above objectives, this invention proposes a semantic segmentation method for bridge crack images based on an improved SegFormer, comprising the following steps: S1: Obtain the surface image of the bridge structure to be segmented; S2: Input the surface image of the bridge structure into a SegFormer-based hierarchical encoder to obtain encoded features at multiple different scales; S3: Construct three-way segmentation features based on multiple coding features at different scales. The three-way segmentation features include multi-scale fusion features, global path features, and local path features. Among them, the multi-scale fusion features are obtained by fusing multiple coding features at different scales by the multi-scale feature fusion module, and serve as the third feature independent of the global path features and local path features. S31: Global path features are extracted from deep semantic features in multiple encoded features at different scales using global paths. S32: Orientation response recalibration via the orientation enhancement attention module; S33: Local path features are obtained by upsampling the local path layer by layer with deep semantic features as the starting feature, and then fusing them with the corresponding low-level coding features; S4: Input global path features, local path features and multi-scale fusion features into the deep feature fusion module. The deep feature fusion module performs collaborative fusion of the three segmentation features based on non-competitive gated weights and residual enhancement methods that correspond to the three segmentation features respectively and are independent of each other, to obtain fused segmentation features. S5: Based on the fused segmentation features, the segmentation head outputs pixel-level segmentation results of the bridge crack area and the background area.

[0007] Furthermore, the SegFormer-based hierarchical encoder outputs multiple encoded features at different scales, including first-stage features (F1), second-stage features (F2), third-stage features (F3), and fourth-stage features (F4), with the spatial resolution of the first-stage features (F1), second-stage features (F2), third-stage features (F3), and fourth-stage features (F4) decreasing sequentially. The first-stage feature (F1) contains crack edge, texture and spatial location information, serving as shallow spatial detail features; the fourth-stage feature (F4) has semantic expressive power, serving as deep semantic features.

[0008] Furthermore, the processing steps of the multi-scale feature fusion module include: Each stage feature is channel-projected, and the number of channels of each stage feature is uniformly projected to a preset uniform number of channels through 1×1 convolution; The second-stage features (F2), third-stage features (F3), and fourth-stage features (F4) after channel projection are upsampled to the same spatial resolution as the first-stage feature (F1); After spatial resolution alignment, scale weights are assigned to the features of each stage, and the features of each stage are concatenated by channels. The scale fusion weights (W1, W2, W3, W4) corresponding to the features of each stage are generated through the channel attention mechanism. The features at each stage are weighted and summed according to the scale fusion weights to obtain the fused multi-scale features. The features are then refined by 3×3 convolution, batch normalization and activation function to output the multi-scale fused features.

[0009] Furthermore, channel attention mechanisms include: The concatenated features are subjected to global average pooling to generate globally pooled features. Global pooling features are processed by convolutional layers to perform channel compression and channel expansion. The scale fusion weights (W1, W2, W3, W4) for each stage of features are generated using the Sigmoid activation function.

[0010] Furthermore, the global path includes a weighted hybrid multi-scale feature extraction module and a direction-enhanced attention module; The weighted hybrid multi-scale feature extraction module takes fourth-stage features (F4) as input and includes multiple parallel dilated convolution branches; Multiple parallel dilated convolution branches use dilated convolution with different dilation rates to convolve the fourth-stage features to extract contextual information under different receptive fields. The outputs of multiple parallel dilated convolution branches are weighted and fused using learnable weights to obtain multi-scale contextual features.

[0011] Furthermore, the dilation rates of the multiple parallel dilated convolutional branches are 1, 2, 4 and 8, respectively; the convolutional branch with a dilation rate of 1 is used to extract local fine-grained contextual information, the convolutional branch with a dilation rate of 2 is used to extract contextual information of medium receptive fields, and the convolutional branches with dilation rates of 4 and 8 are used to extract global semantic information of large receptive fields. The learnable weights are automatically learned and determined during training through backpropagation.

[0012] Furthermore, the orientation-enhanced attention module includes an axial position encoding branch and a diagonal orientation response branch; The axial position encoding branch is used to encode the features input to the orientation enhancement attention module in one dimension along the horizontal and vertical directions, generating axial attention features containing spatial position information. The diagonal direction response branch is used to extract the directional response of the features input to the direction enhancement attention module along the main diagonal direction and the anti-diagonal direction, and generate diagonal direction attention features; The axial attention features and diagonal attention features are multiplied element-wise along the channel dimension to generate the fused direction-enhanced attention weights. The features input to the direction-enhanced attention module are recalibrated using direction-enhanced attention weights to obtain global path features.

[0013] Furthermore, the axial position encoding branch is specifically implemented as a coordinate attention branch, including: A global pooling operation along the horizontal direction pools each channel along the width dimension, generating a horizontally encoded feature with a height of 1 and a width equal to the original width. A global pooling operation along the vertical direction pools each channel along the height dimension, generating a vertically encoded feature with a height equal to the original height and a width of 1. The horizontal and vertical encoded features are spatially concatenated, channel mixing is performed using 1×1 convolution, and axial attention weights are generated by Sigmoid activation. The axial attention weights are then multiplied by the input features to obtain the axial attention features.

[0014] Furthermore, the local path starts with the fourth-stage feature (F4), upsamples the high-level features step by step, and then performs channel concatenation and convolution fusion with the third-stage feature (F3), the second-stage feature (F2), and the first-stage feature (F1) in sequence; Each layer-by-layer upsampling fusion unit includes an upsampling operation, a channel splicing operation, and a convolutional fusion operation; The upsampling operation uses bilinear interpolation to double the spatial resolution of the output features from the previous layer. The channel concatenation operation concatenates the upsampled features with the encoded features of the corresponding stage along the channel dimension. The convolutional fusion operation performs spatial and channel mixing of the spliced ​​features through 3×3 convolution, and refines the features through batch normalization and activation functions.

[0015] Furthermore, the deep feature fusion module's processing steps include: Adjust global path features, local path features, and multi-scale fusion features to the same spatial resolution; The three features adjusted to the same spatial resolution are spliced ​​together to obtain the spliced ​​features. The spliced ​​features are processed by a gated convolutional network to generate three independent Sigmoid gate weights (G1, G2, G3) corresponding to global path features, local path features, and multi-scale fusion features, respectively. The global path features, local path features, and multi-scale fusion features are multiplied by their respective Sigmoid gate weights to obtain three-way gated features. The three gated features are respectively added to their corresponding original features before gated processing to obtain global enhancement features, local enhancement features, and multi-scale enhancement features; The global enhancement features, local enhancement features, and multi-scale enhancement features are processed by channel concatenation, and the features are integrated by fusion convolution to obtain fused segmentation features.

[0016] Compared with existing technologies, the advantages of this invention are as follows: This invention generates independent multi-scale fused features through a multi-scale feature fusion module and enhances the crack direction response through a global path and direction enhancement attention module. It recovers crack edge and fine texture information by upsampling layer by layer through local paths, and performs non-competitive collaborative fusion of the three features through a deep feature fusion module based on mutually independent Sigmoid gate weights and residual enhancement methods. This solves the problems of insufficient multi-scale feature fusion, insufficient crack direction perception, weak local detail recovery, and insufficient coordination between global semantics and local edge information in existing methods, thus improving the completeness, boundary clarity, and stability of the pixel-level segmentation results for bridge cracks. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall process of the semantic segmentation method for bridge crack images based on the improved SegFormer according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the overall network structure according to an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the multi-scale feature fusion module MSFF in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the weighted hybrid multi-scale feature extraction module (WMF) according to an embodiment of the present invention.

[0021] Figure 5 This is a schematic diagram of the structure of the directional enhanced attention module according to an embodiment of the present invention.

[0022] Figure 6 This is a schematic diagram of the Local Path Layer-by-Layer Upsampling and Fusion Module (LFM) according to an embodiment of the present invention.

[0023] Figure 7 This is a schematic diagram of the structure of the Deep Feature Fusion (DFM) module according to an embodiment of the present invention.

[0024] Figure 8 This is a schematic diagram comparing the bridge crack segmentation effect according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0026] This invention provides a semantic segmentation method for bridge crack images based on an improved SegFormer. This method can be used for pixel-level crack identification in images of bridge beams, decks, piers, abutments, and other bridge structural surfaces, and is particularly suitable for the automatic segmentation of slender cracks, diagonal cracks, bending cracks, bifurcation cracks, and low-contrast crack regions. This embodiment improves the completeness, continuity, and boundary clarity of the pixel-level segmentation results for bridge cracks by constructing a three-way parallel segmentation feature architecture and designing independent, non-competitive gating weights and residual enhancement methods for deep feature fusion.

[0027] The overall process of the semantic segmentation method for bridge crack images in this embodiment of the invention is as follows: Figure 1 As shown, it includes the following steps: S1: Obtain the surface image of the bridge structure to be segmented; S2: Input the surface image of the bridge structure into a SegFormer-based hierarchical encoder to obtain encoded features at multiple different scales; S3: Construct three-way segmentation features based on multiple coding features at different scales. The three-way segmentation features include multi-scale fusion features, global path features, and local path features. Among them, the multi-scale fusion features are obtained by fusing multiple coding features at different scales by the multi-scale feature fusion module, and serve as the third feature independent of the global path features and local path features. S31: Global path features are extracted from deep semantic features in multiple encoded features at different scales using global paths. S32: Orientation response recalibration via the orientation enhancement attention module; S33: Local path features are obtained by upsampling the local path layer by layer with deep semantic features as the starting feature, and then fusing them with the corresponding low-level coding features; S4: Input global path features, local path features and multi-scale fusion features into the deep feature fusion module. The deep feature fusion module performs collaborative fusion of the three segmentation features based on non-competitive gated weights and residual enhancement methods that correspond to the three segmentation features respectively and are independent of each other, to obtain fused segmentation features. S5: Based on the fused segmentation features, the segmentation head outputs pixel-level segmentation results of the bridge crack area and the background area.

[0028] like Figure 2 As shown, in this embodiment, the improved SegFormer network structure includes a SegFormer hierarchical encoder, a multi-scale feature fusion module, a global path, a directional attention enhancement module, a local path, a deep feature fusion module, and a segmentation head. The SegFormer hierarchical encoder outputs encoded features in four stages: first-stage feature F1, second-stage feature F2, third-stage feature F3, and fourth-stage feature F4. The decoder section includes a three-branch structure: the first path consists of independent multi-scale fused features generated by the multi-scale feature fusion module; the second path consists of global path features generated by the weighted hybrid multi-scale feature extraction module (WMF) and the directional attention enhancement module; and the third path consists of local path features generated by the layer-by-layer upsampling fusion module (LFM). These three features are ultimately merged into the deep feature fusion module (DFM) for collaborative fusion, and the final segmentation result is output by the segmentation head.

[0029] In step S1, an image of the bridge structure surface to be segmented is acquired. This image can be obtained by manual inspection equipment, drones, vehicle-mounted inspection equipment, or fixed monitoring equipment. In a preferred embodiment, the acquired bridge structure surface image is preprocessed, including at least one of image size normalization, pixel value normalization, and format conversion. During the training phase, synchronous data augmentation can be performed on the bridge structure surface image and its corresponding crack label mask. This data augmentation includes at least one of random horizontal flipping, random vertical flipping, random cropping, brightness adjustment, or contrast adjustment. The crack label mask is used to represent crack regions and background regions, where crack regions are marked as 1 and background regions as 0.

[0030] In step S2, multi-scale coded features are extracted using a SegFormer hierarchical encoder. The preprocessed bridge structure surface image 1 is input into the SegFormer-based hierarchical encoder to obtain coded features at multiple different scales. In a preferred embodiment, the SegFormer-based hierarchical encoder uses a SegFormer-B1 encoder. The hierarchical encoder outputs coded features in four stages: first-stage feature F1, second-stage feature F2, third-stage feature F3, and fourth-stage feature F4. The spatial resolution of the first-stage feature F1, second-stage feature F2, third-stage feature F3, and fourth-stage feature F4 decreases sequentially. In a specific example, if the input image size is H×W, then the spatial resolution of the first-stage feature F1 is H / 4×W / 4, the spatial resolution of the second-stage feature F2 is H / 8×W / 8, the spatial resolution of the third-stage feature F3 is H / 16×W / 16, and the spatial resolution of the fourth-stage feature F4 is H / 32×W / 32. Correspondingly, the number of channels for each stage feature is 64, 128, 320, and 512, respectively. Among them, the first-stage feature F1 contains more crack edge, texture and spatial location information, serving as shallow spatial detail features; the fourth-stage feature F4 has strong semantic expressive ability, serving as deep semantic features.

[0031] In step S3, multi-scale fused features are constructed using a multi-scale feature fusion module. For example... Figure 3 As shown, the multi-scale feature fusion module performs multi-scale fusion processing on the first-stage feature F1, the second-stage feature F2, the third-stage feature F3, and the fourth-stage feature F4 to generate independent multi-scale fused features as the third-path segmentation features. The specific processing of the multi-scale feature fusion module includes four steps: channel projection, spatial resolution alignment, scale weight allocation, and feature refinement.

[0032] In the channel projection step, the multi-scale feature fusion module performs channel projection on the first-stage feature F1, the second-stage feature F2, the third-stage feature F3, and the fourth-stage feature F4, respectively, mapping the features of each stage to a unified number of channels through 1×1 convolutions. In a specific example, the encoded features of the four stages are projected to 256-dimensional channels. Channel projection is achieved through 1×1 convolutions, uniformly projecting the number of channels of the features from each stage to a preset unified number of channels.

[0033] In the spatial resolution alignment step, the second-stage features F2, the third-stage features F3, and the fourth-stage features F4, after channel projection, are upsampled to the same spatial resolution as the first-stage feature F1 to complete the spatial resolution alignment. In a specific example, the three projected features are uniformly upsampled to an H / 4×W / 4 resolution. The upsampling method is implemented using bilinear interpolation, but nearest neighbor interpolation, transposed convolution, or other methods that can achieve spatial resolution alignment can also be used.

[0034] In the scale weight allocation step, scale weights are allocated to the features at each stage after spatial resolution alignment. Specifically, this includes: concatenating the channels of the spatially aligned features at each stage; generating scale fusion weights W1, W2, W3, and W4 corresponding to each stage feature using a channel attention mechanism. The channel attention mechanism involves performing global average pooling on the concatenated features to generate globally pooled features; performing channel compression and expansion on the globally pooled features using convolutional layers; and then generating scale fusion weights for each stage feature using a sigmoid activation function. The features at each stage are then weighted and summed according to the scale fusion weights to obtain the fused multi-scale features.

[0035] In the feature refinement step, 3×3 convolution, batch normalization, and activation functions are used for refinement to output multi-scale fused features. Through this method, the multi-scale fused features can simultaneously contain shallow spatial details, mid-level structural information, and deep semantic information, providing independent multi-scale discriminative information for subsequent deep feature fusion.

[0036] In step S31, global path features are constructed using global paths. For example... Figure 4 As shown, the global path includes a weighted hybrid multi-scale feature extraction module (WMF) and a direction-enhanced attention module. The global path is used to extract multi-scale context and enhance the direction response of deep semantic features, generating global path features with direction awareness.

[0037] The Weighted Hybrid Multi-Scale Feature Extraction (WMF) module takes the fourth-stage feature F4 as input and includes multiple parallel dilated convolution branches. Each parallel dilated convolution branch applies a different dilation rate to the fourth-stage feature F4 to extract contextual information within different receptive fields. In a specific example, the dilation rates d of the four parallel dilated convolution branches are 1, 2, 4, and 8, respectively. The convolution branch with a dilation rate of 1 is used to extract local fine-grained contextual information, the convolution branch with a dilation rate of 2 is used to extract contextual information within a medium receptive field, and the convolution branches with dilation rates of 4 and 8 are used to extract global semantic information within a large receptive field. After the outputs of each branch are processed, they are weighted and fused using learnable weights to obtain multi-scale contextual features. The learnable weights are automatically learned and determined during training through backpropagation to adaptively adjust the fusion ratio of the outputs of each branch.

[0038] After obtaining the multi-scale contextual features, these features are input into the orientation enhancement attention module. The orientation response is then recalibrated through the orientation enhancement attention module to obtain global path features. These global path features are used to enhance the model's ability to globally identify diagonal, curved, and irregularly oriented cracks.

[0039] In step S32, orientation response recalibration is performed using the orientation enhancement attention module. For example... Figure 5 As shown, the orientation enhancement attention module includes an axial position encoding branch and a diagonal orientation response branch. The orientation enhancement attention module is set up by weighting and fusing the outputs of multiple parallel dilated convolution branches to enhance the orientation response of multi-scale contextual features, thus obtaining global path features.

[0040] The axial position encoding branch is used to perform one-dimensional feature encoding on the features input to the orientation-enhanced attention module along the horizontal and vertical directions, generating axial attention features containing spatial position information. Specifically, the axial position encoding branch is implemented as a coordinate attention branch. The coordinate attention branch performs global pooling or one-dimensional encoding on the input features along the horizontal and vertical directions, respectively, to obtain horizontal and vertical encoded features. Specifically, the global pooling operation along the horizontal direction pools each channel along the width dimension, generating a horizontal encoded feature with a height of 1 and a width equal to the original width; the global pooling operation along the vertical direction pools each channel along the height dimension, generating a vertical encoded feature with a height equal to the original height and a width of 1. The horizontal and vertical encoded features are spatially concatenated, channel-wise mixed using a 1×1 convolution, and then axial attention weights are generated using Sigmoid activation. These axial attention weights are multiplied by the input features to obtain the axial attention features.

[0041] The diagonal response branch is used to extract the directional responses of the features input to the directional attention module 5 along the main diagonal and anti-diagonal directions, generating diagonal attention features. The diagonal response branch performs grouped convolutions on the features input to the directional attention module using main diagonal and anti-diagonal convolution kernels, obtaining main diagonal response features and anti-diagonal response features respectively. Specifically, the convolution weights of the main diagonal convolution kernel are distributed along the main diagonal direction, and the convolution weights of the anti-diagonal convolution kernel are distributed along the anti-diagonal direction. These two types of convolution kernels process the input features using depthwise separable grouped convolutions to extract response information of the feature map along the main diagonal and anti-diagonal directions. After channel mixing of the main diagonal and anti-diagonal response features through 1×1 convolution, they are activated using a sigmoid function to generate the diagonal attention features.

[0042] Axial attention features are fused with diagonal attention features. Specifically, the axial and diagonal attention features are multiplied element-wise along the channel dimension to generate fused directional enhancement attention weights. These weights are then used to recalibrate the features input to the directional enhancement attention module, resulting in directionally enhanced global path features. Through this method, the directional enhancement attention module can utilize not only spatial location information in the horizontal and vertical directions but also enhance crack responses along the main diagonal and anti-diagonal directions, thereby effectively improving the identification ability of diagonal cracks, curved cracks, and bifurcated cracks.

[0043] In step S33, local path features are constructed using local paths. For example... Figure 6 As shown, the local path comprises multiple layer-by-layer upsampling fusion units (LFMs). The local path is used to progressively upsample deep semantic features and fuse them with shallow spatial detail features to generate local path features that recover crack edges and fine texture information. The local path starts with the fourth-stage feature F4, progressively upsamples higher-level features, and then sequentially performs channel concatenation and convolution fusion with the third-stage feature F3, the second-stage feature F2, and the first-stage feature F1 to obtain the local path features.

[0044] Each Layer-by-Layer Upsampling Fusion Unit (LFM) includes upsampling, channel concatenation, and convolutional fusion operations. The upsampling operation uses bilinear interpolation to double the spatial resolution of the output features from the previous layer to match the spatial resolution of the encoded features at the corresponding stage. The channel concatenation operation concatenates the upsampled features with the encoded features of the corresponding stage along the channel dimension, forming a concatenated feature with an increased number of channels. The convolutional fusion operation performs spatial and channel mixing of the concatenated features using 3×3 convolutions and refines the features through batch normalization and activation functions.

[0045] The layer-by-layer upsampling fusion process includes the following three steps: First, the fourth-stage feature F4 is upsampled to the spatial resolution of the third-stage feature F3, and then channel-wise concatenated and convolved with the third-stage feature F3 to obtain the first local fusion feature. Next, the first local fusion feature is upsampled to the spatial resolution of the second-stage feature F2, and then channel-wise concatenated and convolved with the second-stage feature F2 to obtain the second local fusion feature. Finally, the second local fusion feature is upsampled to the spatial resolution of the first-stage feature F1, and then channel-wise concatenated and convolved with the first-stage feature F1 to obtain the local path feature. Through this layer-by-layer upsampling fusion method, the local path can gradually restore deep semantic information to a higher spatial resolution and combine it with shallow spatial detail information, thereby effectively enhancing the recovery ability of crack edges, fine textures, and small crack structures.

[0046] In step S4, three-way collaborative fusion is performed through a deep feature fusion module. For example... Figure 7 As shown, global path features, local path features, and multi-scale fusion features are input into the deep feature fusion module to obtain fused segmentation features. The deep feature fusion module is used to perform collaborative fusion processing on global path features, local path features, and multi-scale fusion features to generate fused segmentation features.

[0047] The specific processing steps of the deep feature fusion module include six steps: spatial resolution adjustment, channel stitching, gating weight generation, gating enhancement, residual enhancement, and fusion convolution.

[0048] In the spatial resolution adjustment step, the global path features, local path features, and multi-scale fusion features are adjusted to the same spatial resolution. The spatial resolution of the local path features is H / 4×W / 4, and the spatial resolution of the global path features and multi-scale fusion features also needs to be uniformly adjusted to H / 4×W / 4.

[0049] In the channel stitching step, the three features adjusted to the same spatial resolution are stitched together to obtain the stitched features.

[0050] In the gate weight generation step, the concatenated features are processed by a gated convolutional network to generate three independent Sigmoid gate weights G1, G2, and G3, corresponding to global path features, local path features, and multi-scale fusion features, respectively. The gated convolutional network consists of convolutional layers and Sigmoid activation layers. The number of output channels of the convolutional layers is three times the number of features in the three paths, generating three sets of gate weights. The number of channels in each set of gate weights is consistent with the number of channels in the corresponding feature. Independent gate weights are generated through Sigmoid activation processing.

[0051] In the gating enhancement step, the global path features, local path features, and multi-scale fusion features are multiplied by the corresponding Sigmoid gating weights G1, G2, and G3, respectively, to obtain three-way gating features.

[0052] In the residual enhancement step, the three gated features are respectively added to their corresponding original features before gateding to obtain global enhanced features, local enhanced features, and multi-scale enhanced features. The residual addition operation is used to superimpose the gated enhanced features with the original features to achieve residual enhancement of the features.

[0053] In the fusion convolution step, global enhancement features, local enhancement features, and multi-scale enhancement features are concatenated through channels, and then integrated through fusion convolution to obtain fused segmentation features. The fusion convolution is implemented using a combination of 3×3 convolution, batch normalization, and activation functions to generate the final fused segmentation features.

[0054] In this embodiment, the fusion of the three features adopts a non-competitive collaborative fusion strategy. Unlike the traditional Softmax-based competitive weight allocation method, the three Sigmoid gate weights G1, G2, and G3 generated by the deep feature fusion module 7 are independent and non-competitive. Each feature is enhanced through independent gate weights, rather than inhibited by competitive weights such as Softmax. Therefore, this fusion method enables global path features, local path features, and multi-scale fusion features to participate in segmentation in a complementary and collaborative manner, which helps to avoid the suppression or loss of important crack information during the fusion process, thereby preserving the complete discriminative information of each feature.

[0055] In step S5, the segmentation head outputs pixel-level segmentation results. The fused segmentation features are input into the segmentation head to obtain pixel-level classification results for the bridge crack region and the background region. The segmentation head includes convolutional layers and upsampling layers. The segmentation head outputs a category probability map corresponding to the input image, which includes crack category and background category. A binary crack segmentation mask can be obtained through argmax operation or thresholding.

[0056] like Figure 8 As shown, the output pixel-level segmentation results can be used to display the crack area of ​​the bridge, and can be further used to calculate parameters such as crack length, width, direction, area and connectivity.

[0057] This invention also provides a complete training, evaluation, and inference process based on the BCL dataset.

[0058] In the dataset preparation step, the publicly available BCL (Bridge Crack Library) dataset is used, divided into train / val / test. The dataset class (BCLDataset) implements the following preprocessing pipeline: training set data augmentation: random horizontal flipping, random vertical flipping (image and mask synchronized); uniform image resizing to 256×256; mask binarization (crack=1, background=0); and normalization preprocessing using SegformerImageProcessor.

[0059] In the model building step, the `build_model()` function is called, passing in the parameters: `model_name="nvidia / segformer-b1-finetuned-ade-512-512"`, `num_classes=2`, and `variant="E6"`. This function loads the ImageNet-1K pre-trained SegFormer-B1 encoder weights from HuggingFace and builds a complete model including MSFF, a dual-channel decoder, and CA-Diagonal. Model parameters: encoder channels [layer1:64, layer2:128, layer3:320, layer4:512], decoder uniform channel count 256, and the segmenter outputs 2 classes.

[0060] In the training configuration step, the optimizer used is AdamW, with an initial learning rate of 6×10⁻⁶. -5 The weight decay is 0.01. CosineAnnealingLR is used for learning rate scheduling, with T_max=30 and eta_min=1×10^6. -6 The loss function used was combined loss, with a CrossEntropy weight of 0.3, a Dice weight of 0.7, and a label smoothing factor of 0.05. The batch size was set to 3, the training epochs were 30, and the hardware used was a CUDA GPU.

[0061] In the evaluation and inference steps, after training, the model weights with the highest IoU on the validation set are loaded, and the model is evaluated on the test set. Evaluation metrics include IoU, pixel accuracy, precision, recall, F1-score, and standard deviation. For new images, single-image inference can be performed using the `inference_single()` function, and the results can be visualized.

[0062] In the model deployment step, the trained model is saved in .pth format and can be loaded and used in supported inference environments. During inference, the input image is preprocessed by SegformerImageProcessor and then fed into the model. The output logits are processed by argmax to obtain a binary segmentation map.

[0063] To verify the effectiveness of the method of this invention, experiments were conducted using the BCL bridge crack dataset. In the experiments, the original SegFormer-B1 model was used as the baseline model, and multiple ablation experiment variants were designed to verify the impact of each improved module on the semantic segmentation performance of bridge crack images. The experiments were based on the BCL dataset, containing 1100 test images with a uniform image size of 256×256. All variants were trained on the training set for 30 epochs and then evaluated on the test set.

[0064] Table 1

[0065] This ablation experiment employed a controlled variable approach, maintaining the same data partitioning, training configuration, and evaluation protocol across all variants to ensure that performance differences stemmed solely from changes in model structure. A bidirectional ablation path was also designed: on one hand, modules were progressively added from the simplest baseline to verify the cumulative gain of module stacking; on the other hand, modules were progressively removed from the complete scheme to confirm the necessity of each module.

[0066] Experimental variants include: B1 is the original SegFormer-B1 baseline model; E2 adds standard coordinate attention to the baseline model; E3 adds coordinate attention and diagonal direction response enhancement modules to the baseline model; E4 uses a dual-channel decoder and a multi-scale feature fusion module; E5 uses a dual-channel decoder, a multi-scale feature fusion module, and standard coordinate attention; E6 uses the complete scheme of this invention, namely a dual-channel decoder, a multi-scale feature fusion module, a direction enhancement attention module, and a deep feature fusion module.

[0067] Experimental results show that the original SegFormer-B1 baseline model has an IoU of 0.3794 and an F1-score of 0.4761; the complete scheme E6 of this invention has an IoU of 0.4392 and an F1-score of 0.5282. Compared with the baseline model, the complete scheme of this invention improves IoU by 5.98 percentage points, a relative improvement of 15.8%, and the F1-score by 5.21 percentage points. All indicators are the highest among the variants, verifying the synergistic effect of the three improvements.

[0068] Experimental results show that the present invention can effectively improve the semantic segmentation performance of bridge crack images through three-way segmentation feature construction, directional enhanced attention, multi-scale feature fusion, and Sigmoid-gated residual fusion.

[0069] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A semantic segmentation method for bridge crack images based on an improved SegFormer, characterized in that, Includes the following steps: S1: Obtain the surface image of the bridge structure to be segmented; S2: Input the surface image of the bridge structure into a SegFormer-based hierarchical encoder to obtain multiple encoded features at different scales; S3: Construct a three-way segmentation feature based on the multiple coding features at different scales. The three-way segmentation feature includes a multi-scale fusion feature, a global path feature, and a local path feature. The multi-scale fusion feature is obtained by fusing the multiple coding features at different scales by a multi-scale feature fusion module and serves as a third feature independent of the global path feature and the local path feature. S31: The global path features are extracted from the deep semantic features of the multiple encoded features at different scales by multi-scale context extraction; S32: Orientation response recalibration via the orientation enhancement attention module; S33: The local path features are obtained by upsampling the local path layer by layer with the deep semantic features as the starting features, and then fusing them with the corresponding low-level coding features; S4: Input the global path features, the local path features, and the multi-scale fusion features into the deep feature fusion module. The deep feature fusion module performs collaborative fusion of the three segmentation features based on non-competitive gated weights and residual enhancement methods that correspond to the three segmentation features respectively and are independent of each other, to obtain fused segmentation features. S5: Based on the fusion segmentation features, the segmentation head outputs pixel-level segmentation results of the bridge crack area and the background area.

2. The semantic segmentation method for bridge crack images based on the improved SegFormer as described in claim 1, characterized in that, The SegFormer-based hierarchical encoder outputs multiple coding features of different scales, including first-stage features (F1), second-stage features (F2), third-stage features (F3), and fourth-stage features (F4), with the spatial resolution of the first-stage features (F1), second-stage features (F2), third-stage features (F3), and fourth-stage features (F4) decreasing sequentially. The first stage feature (F1) contains crack edge, texture and spatial location information, serving as shallow spatial detail features; the fourth stage feature (F4) has semantic expressive ability, serving as deep semantic features.

3. The semantic segmentation method for bridge crack images based on the improved SegFormer according to claim 2, characterized in that, The processing steps of the multi-scale feature fusion module include: Each stage feature is channel-projected, and the number of channels of each stage feature is uniformly projected to a preset uniform number of channels through 1×1 convolution; The second-stage feature (F2), the third-stage feature (F3), and the fourth-stage feature (F4) after channel projection are upsampled to the same spatial resolution as the first-stage feature (F1); After spatial resolution alignment, scale weights are assigned to the features of each stage, and the features of each stage are concatenated by channels. The scale fusion weights (W1, W2, W3, W4) corresponding to the features of each stage are generated through the channel attention mechanism. The features at each stage are weighted and summed according to the scale fusion weights to obtain the fused multi-scale features. The features are then refined by 3×3 convolution, batch normalization and activation function to output the multi-scale fused features.

4. The semantic segmentation method for bridge crack images based on the improved SegFormer according to claim 3, characterized in that, The channel attention mechanism includes: The concatenated features are subjected to global average pooling to generate globally pooled features. The global pooling features are processed by channel compression and channel expansion through convolutional layers; The scale fusion weights (W1, W2, W3, W4) for each stage of features are generated using the Sigmoid activation function.

5. The semantic segmentation method for bridge crack images based on the improved SegFormer according to claim 1, characterized in that, The global path includes a weighted hybrid multi-scale feature extraction module and a direction-enhanced attention module; The weighted hybrid multi-scale feature extraction module takes fourth-stage features (F4) as input and includes multiple parallel dilated convolution branches; The multiple parallel dilated convolution branches use dilated convolution with different dilation rates to perform convolution processing on the fourth-stage features in order to extract contextual information under different receptive fields. The outputs of the multiple parallel dilated convolution branches are weighted and fused using learnable weights to obtain multi-scale contextual features.

6. The semantic segmentation method for bridge crack images based on the improved SegFormer according to claim 5, characterized in that, The dilation rates of the multiple parallel dilated convolutional branches are 1, 2, 4 and 8, respectively; the convolutional branch with a dilation rate of 1 is used to extract local fine-grained contextual information, the convolutional branch with a dilation rate of 2 is used to extract contextual information of medium receptive fields, and the convolutional branches with dilation rates of 4 and 8 are used to extract global semantic information of large receptive fields. The learnable weights are automatically learned and determined during the training process through backpropagation.

7. The semantic segmentation method for bridge crack images based on the improved SegFormer as described in claim 5, characterized in that, The direction-enhanced attention module includes an axial position encoding branch and a diagonal direction response branch; The axial position encoding branch is used to perform one-dimensional feature encoding on the features input to the direction enhancement attention module along the horizontal and vertical directions to generate axial attention features containing spatial position information. The diagonal response branch is used to extract the directional response of the features input to the directional enhancement attention module along the main diagonal direction and the anti-diagonal direction, and generate diagonal attention features; The axial attention features and the diagonal attention features are multiplied element-wise along the channel dimension to generate fused direction-enhanced attention weights. The features input to the direction-enhanced attention module are recalibrated using the direction-enhanced attention weights to obtain the global path features.

8. The semantic segmentation method for bridge crack images based on the improved SegFormer according to claim 7, characterized in that, The axial position encoding branch is specifically implemented as a coordinate attention branch, including: A global pooling operation along the horizontal direction pools each channel along the width dimension, generating a horizontally encoded feature with a height of 1 and a width equal to the original width. A global pooling operation along the vertical direction pools each channel along the height dimension, generating a vertically encoded feature with a height equal to the original height and a width of 1. The horizontal and vertical encoded features are spatially concatenated, channel mixing is performed using a 1×1 convolution, and then axial attention weights are generated by Sigmoid activation. The axial attention weights are multiplied by the input features to obtain the axial attention features.

9. The semantic segmentation method for bridge crack images based on the improved SegFormer as described in claim 1, characterized in that, The local path starts with the fourth-stage feature (F4), upsamples the high-level features step by step, and sequentially performs channel splicing and convolution fusion with the third-stage feature (F3), the second-stage feature (F2), and the first-stage feature (F1). Each layer-by-layer upsampling fusion unit includes an upsampling operation, a channel splicing operation, and a convolutional fusion operation; The upsampling operation uses bilinear interpolation to double the spatial resolution of the output features from the previous layer. The channel splicing operation splices the upsampled features with the encoded features of the corresponding stage along the channel dimension. The convolutional fusion operation performs spatial and channel mixing of the spliced ​​features through 3×3 convolution, and refines the features through batch normalization and activation functions.

10. The semantic segmentation method for bridge crack images based on the improved SegFormer as described in claim 1, characterized in that, The processing steps of the deep feature fusion module include: Adjust the global path features, the local path features, and the multi-scale fusion features to the same spatial resolution; The three features adjusted to the same spatial resolution are spliced ​​together to obtain the spliced ​​features. The spliced ​​features are processed by a gated convolutional network to generate three independent Sigmoid gate weights (G1, G2, G3) corresponding to the global path features, the local path features, and the multi-scale fusion features, respectively. The global path feature, the local path feature, and the multi-scale fusion feature are multiplied by their respective Sigmoid gate weights to obtain the three-way gated features; The three gating features are respectively added to their corresponding original features before gating to obtain global enhancement features, local enhancement features, and multi-scale enhancement features; The global enhancement features, the local enhancement features, and the multi-scale enhancement features are processed by channel concatenation, and the features are integrated by fusion convolution to obtain the fused segmentation features.

Images