Crack semantic segmentation method based on global-local feature fusion and edge perception

Abstract

This invention provides a crack semantic segmentation method based on global-local feature fusion and edge awareness, belonging to the field of image segmentation technology. The method includes processing crack images using a TransHRNet deep neural network model to generate crack semantic segmentation results. The TransHRNet deep neural network model includes an encoder and a decoder. The encoder includes an HRM convolutional module for extracting local features and an STM (Swin Transformer) self-attention mechanism module for global context modeling. The decoder includes an AFM attention fusion module. During the training phase of the TransHRNet deep neural network model, an edge-aware supervision method is executed. This invention solves the problems of easy misjudgment and easy loss of edge detail information in existing crack semantic segmentation methods.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, specifically to a crack semantic segmentation method based on global-local feature fusion and edge awareness. Background Technology

[0002] The structural health of critical infrastructure such as roads, bridges, tunnels, and dams is a crucial prerequisite for ensuring the normal operation of society and public safety, and cracks are a core and early indicator for assessing their health. Traditional crack detection mainly relies on manual visual inspection, which is not only labor-intensive, inefficient, and costly, but also highly susceptible to the subjective experience and fatigue of the inspectors, leading to a lack of objectivity and standardization. With the development of computer vision technology, crack detection algorithms based on traditional image processing (such as edge detection and threshold segmentation) have emerged. However, these algorithms are extremely sensitive to changes in lighting, image noise, and interference from background elements such as stains, shadows, and scratches. In complex real-world application scenarios, these algorithms have poor robustness, often resulting in high false positive and false negative rates, making it difficult to meet the accuracy requirements of engineering applications.

[0003] In recent years, deep learning methods, represented by convolutional neural networks (CNNs), have made groundbreaking progress in the field of image segmentation. In particular, fully convolutional networks (FCNs) and U-Net (Convolutional Networks for Biomedical Image Segmentation, a type of convolutional neural network) and their variant architectures have been widely applied to pixel-level semantic segmentation of cracks due to their powerful feature learning capabilities. For example, patent CN119600286A provides a U-Net-based method for semantic segmentation of remote sensing images, which uses an improved U-shaped network. These deep learning methods typically employ an encoder-decoder structure, extracting multi-scale features of the image through layer-by-layer downsampling by the encoder, and then layer-by-layer upsampling by the decoder and fusing the shallow features from the encoder. However, although CNN-based methods have significantly improved the performance of crack detection, their inherent limitations are becoming increasingly apparent, specifically: Firstly, there is a tendency for misjudgment. The core operation of CNNs is convolution, which is limited by the local receptive field. This makes it difficult for pure CNN models to capture the global context and long-range dependencies of cracks. Consequently, they may misclassify background distractions with similar local textures (such as thin scratches or dirt edges) as cracks, resulting in discontinuous segmentation results. Secondly, there is a tendency to lose edge detail information. During the encoder's downsampling process (such as pooling or straddle convolution), the spatial resolution of the image is continuously reduced to extract high-level semantic information, inevitably leading to the loss of detail information, such as the precise spatial location and fine boundaries of subtle cracks. Even though existing technologies compensate for this with skip connections, simple feature stitching or addition cannot fully recover the lost details, still resulting in blurred crack edges and insufficient accuracy in the final predicted cracks.

[0004] In addition, in order to enhance global modeling capabilities, some researchers have tried to introduce the visual Transformer model. However, the computational complexity of the standard Transformer's self-attention mechanism is proportional to the square of the image size. Directly applying it to high-resolution crack detection images would bring a huge computational burden, so it is not suitable for practical deployment.

[0005] Therefore, there is an urgent need for a new semantic segmentation method for cracks that can effectively integrate the local fine features of cracks with global contextual information while maintaining computational efficiency, and enhance the accurate restoration of crack edge details, so as to achieve accurate, complete and efficient detection of cracks in various complex backgrounds. Summary of the Invention

[0006] The purpose of this invention is to provide a crack semantic segmentation method based on global-local feature fusion and edge awareness, which aims to improve the integrity, continuity and edge clarity of crack segmentation, so as to solve the problems of easy misjudgment and easy loss of edge detail information in existing crack semantic segmentation methods mentioned in the background art.

[0007] This invention is achieved using the following technical solution: A crack semantic segmentation method based on global-local feature fusion and edge awareness includes the following steps: Step 1: Obtain the crack image to be processed; Step 2: The crack image is processed using the TransHRNet deep neural network model to generate crack semantic segmentation results; The TransHRNet deep neural network model includes an encoder and a decoder, wherein: The encoder includes: an HRM convolution module for extracting local features, and an STM (Swin Transformer) self-attention mechanism module for performing global context modeling. The decoder includes an AFM attention fusion module, which is used to perform weighted fusion of skip connection features from the encoder and upsampled features from the decoder. During the training phase of the TransHRNet deep neural network model, the following edge-aware supervision method is implemented: an EDB (Edge Detection Branch) is introduced, which uses the features of the shallow layer of the encoder to learn and predict the boundaries of cracks, thereby enhancing the ability of the shallow layer of the encoder to represent the details of crack boundaries.

[0008] In the method provided by this invention, the TransHRNet deep neural network model is an end-to-end deep neural network based on an encoder-decoder architecture. It deeply integrates the advantages of convolutional neural networks (CNN) and SwinTransformer (a hierarchical visual Transformer model based on the shifted window self-attention mechanism) and introduces an attention mechanism for feature fusion, which can achieve high-precision segmentation of cracks and solve the problem of missing global contextual information. In addition, a lightweight EDB edge detection branch is innovatively introduced during the model training stage, which makes the final segmentation result not only semantically accurate, but also with clearer and more continuous boundary contours.

[0009] Furthermore, the edge-aware supervision method specifically includes the following steps: S1: Construct the EDB edge detection branch and input at least two feature maps from the shallow layer of the encoder into the EDB edge detection branch; S2: The EDB edge detection branch processes the feature map to generate an edge prediction map; S3: Calculate the edge-aided loss based on the difference between the edge prediction map and the actual edge labels; S4: The edge-assisted loss and the loss of the main segmentation task are weighted and summed to obtain the total loss function of the model, which is used to guide the update of the weight parameters in the model network.

[0010] Further, step S2 includes the following sub-steps: S2-1: Perform spatial size matching and channel dimension adjustment on at least two of the feature maps; S2-2: At least two feature maps that have undergone spatial size matching and channel dimension adjustment are concatenated and feature extraction is performed through a convolutional layer to generate edge feature representations; S2-3: The edge feature representation is fed into a prediction head, which maps the edge feature representation into a single-channel edge prediction map.

[0011] Furthermore, the expression for the total loss function of the model is as follows:

[0012] In the formula, This represents the total loss function of the model. This represents the loss of the main segmentation task. Indicates edge-assisted loss, This represents the balancing weight factor for edge-assisted loss.

[0013] In the above scheme, an EDB edge detection branch is introduced as an auxiliary supervision task. This branch specifically utilizes the high-resolution features of the shallow layers of the encoder to learn and predict the precise boundaries of cracks, and explicitly supervises the shallow layer parameters of the network through an independent edge loss function (i.e., the model's total loss function). This multi-task learning framework is equivalent to adding an extra training objective of "edge localization" to the model, forcing the encoder to learn a more sensitive and powerful representation ability for edge information at an early stage. These optimized, high-quality edge features can be effectively transferred and permeated throughout the entire network through skip connections and attention fusion mechanisms in the subsequent decoder. Therefore, the EDB edge detection branch can enhance the ability of the shallow layers of the encoder in the model to represent the details of crack boundaries, significantly improving the clarity, continuity, and integrity of crack boundaries. This effectively solves the technical problems of easily broken small cracks and blurred edges that are common in existing technologies, thereby improving the edge prediction accuracy of the main segmentation task. In addition, in practical applications, this step can be removed during the model inference (deployment) stage to avoid additional computational overhead and increased deployment costs.

[0014] Furthermore, the encoder performs multiple encoding stages, divided into a shallow encoder layer and a deep encoder layer; the shallow encoder layer uses an HRM convolution module; the deep encoder layer uses a combined structure including an HRM convolution module and an STM self-attention mechanism module.

[0015] Furthermore, in the shallow layer of the encoder, the HRM convolution module has two paths: a computation path and a residual path. The computation path sequentially includes a first 1x1 convolutional layer, a 3x3 depth convolutional layer, and a second 1x1 convolutional layer. The output of the HRM convolution module is the result of information fusion between the output of the computational path and the output of the residual path.

[0016] Furthermore, the combined structure includes a first HRM convolutional module, an STM self-attention mechanism module, and a second HRM convolutional module arranged in sequence; The steps for encoding the deep layers of the encoder using this combined structure are as follows: A1: The feature map is fed into the first HRM convolutional module, which preprocesses the feature map; A2: The feature map is fed into the STM self-attention mechanism module, which performs global context modeling on the feature map and captures global features. A3: The feature map is fed into the second HRM convolutional module, which fuses the global features output by the STM self-attention mechanism module with the local features output by the shallow layer of the encoder.

[0017] The above scheme designs a hybrid encoder architecture combining U-Net and Transformer, fundamentally solving the problem of missing global contextual information caused by the limited receptive field of traditional U-Net. Specifically, this invention abandons the pure convolutional structure and combines convolutional layers, which are good at extracting local details, with Transformer modules, which are good at capturing global context and long-range dependencies. In the shallow layers of the encoder, the model uses efficient hybrid residual HRM convolutional modules to focus on capturing the fine local texture and edge details of the crack; while in the deep, low-resolution feature maps, a SwinTransformer module (i.e., STM self-attention mechanism module) is innovatively embedded, using its powerful self-attention mechanism to efficiently model the long-range dependencies of the image, thereby understanding the overall trend and continuity of the crack. This hybrid design of "shallow layer for details, deep layer for the whole picture" enables the model to effectively distinguish between real continuous cracks and interference objects with similar local textures in the background (such as stains, scratches, and shadows), thereby greatly reducing the false detection rate and improving the robustness and accuracy of detection under various complex working conditions. The architecture described above is not simply a stacking of CNNs and Transformers, but rather a strategic application of Transformers to deep feature maps at low resolution. This design leverages the powerful global context modeling capabilities of Transformers (understanding the wholeness of cracks) while cleverly avoiding the problem of excessive computational cost on high-resolution images, achieving an efficient combination of fine local perception and macroscopic global understanding.

[0018] Furthermore, a bottleneck layer is provided between the encoder and the decoder; the bottleneck layer includes two HRM convolutional modules.

[0019] In the above scheme, the bottleneck layer acts as a bridge connecting the encoder and the decoder. It can perform a final downsampling and feature extraction on the highest-level feature map output by the deep layer of the encoder to capture the most abstract crack semantic information.

[0020] Furthermore, the decoder includes several decoder modules, each of which includes an AFM attention fusion module and an HRM convolution module; The AFM attention fusion module performs a feature fusion method, including the following steps: T1: At each layer of the decoder, the upsampled features from deeper layers are used as guiding signals to generate a spatial attention map; T2: Multiply the spatial attention map element-wise with the skip connection features from the corresponding stage of the encoder to obtain attention-weighted skip connection features; T3: The attention-weighted skip connection features are concatenated with the upsampled features and fed into the HRM convolution module for feature integration.

[0021] Furthermore, in T1, the spatial attention map generation process is as follows: first, the upsampled features and skip connection features are linearly projected to align the channels; then the projected results are summed and activated by the ReLU function; and then a convolutional layer and a Sigmoid activation function are applied.

[0022] The above scheme abandons the crude approach of simply splicing skip connection features in the traditional U-Net decoder and introduces a novel AFM attention fusion module. Based on this, during the upsampling process of each layer of the decoder, the AFM attention fusion module can use features rich in high-level semantic information from deep layers of the decoder as guiding signals to generate a spatial attention map. This spatial attention map dynamically weights the detailed skip connection features from the encoder, thereby achieving intelligent feature selection, that is, actively enhancing effective features related to cracks and suppressing irrelevant background noise. This mechanism maximizes the efficiency and effectiveness of skip connections, ensuring that the information transmitted to the decoder is purified high-quality detailed information, which in turn enables more accurate recovery of the fine structure of cracks in the final segmentation result, thus significantly improving the clarity of crack edges and the fidelity of pixel-level segmentation.

[0023] The beneficial effects achieved by this invention are: This paper presents a crack semantic segmentation method based on global-local feature fusion and edge awareness, employing the TransHRNet deep neural network model. Compared with existing crack semantic segmentation methods, it has the following advantages: First, it can significantly improve the quality of edge segmentation and solve the problem of blurred edges: by introducing the EDB edge detection branch as an auxiliary supervision task during the training phase, the shallow network of the encoder can be forced to learn a more sensitive and powerful representation ability for crack boundaries, so that the final segmentation result is not only semantically accurate, but also has clearer and more continuous boundary contours.

[0024] Secondly, it can enhance global context understanding and reduce false detection rate: Through the proposed U-Net and Transformer hybrid encoder architecture, it can capture local details in the shallow layer using efficient HRM convolution modules, and embed STM self-attention mechanism modules in the deep layer to model global context and long-distance dependency relationships. This enables the encoder to take into account both the micro-texture and macro-trend of cracks when extracting features, and can effectively distinguish between real continuous cracks and interference objects with similar local textures in the background, thereby improving the robustness and accuracy of detection under various complex working conditions.

[0025] Third, it can improve feature fusion efficiency and accurately restore crack details: By introducing a novel AFM attention fusion module, it can use the deep high-level semantic information of the decoder as a guide to dynamically weight and intelligently filter the skip connection features from the encoder, so that the model can more accurately restore the fine structure of the crack in the final segmentation result, significantly improving the clarity of crack edges and the fidelity of pixel-level segmentation. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the crack experiment portion used to construct the dataset according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the crack image after cropping as described in an embodiment of the present invention; Figure 3 This is a schematic diagram of the module architecture of the EDB edge detection branch according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the overall network architecture of the TransHRNet deep neural network model described in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the HRM convolution module described in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the AFM attention fusion module according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the STM self-attention mechanism module described in an embodiment of the present invention; Figure 8 This is a graph showing the change of the loss function during the training process as described in the embodiments of the present invention; Figure 9This is a schematic diagram of the semantic segmentation results of a partial crack image as described in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] Example 1 This embodiment provides a crack semantic segmentation method based on global-local feature fusion and edge awareness, including the following steps: Step 1: Obtain the crack image to be processed. Specifically: Due to the data-driven nature of neural network models, training relies on a large amount of crack image data. However, most publicly available crack datasets online are watermarked, with only a few usable for training. Therefore, to ensure the model's generalization ability, this method combines some publicly available crack datasets with experimentally acquired crack datasets (comprehensive crack image datasets integrated from actual experimental crack images) for subsequent training and testing tasks. The specific operations include the following: Data Collection: For publicly available crack datasets, data was collected from the internet or databases. For experimentally collected crack datasets, a method of damaging intact concrete slabs was used to obtain cracks of varying widths; please refer to [reference needed]. Figure 1 This study presents six concrete slabs obtained after specific damage in a subset of experiments, containing multiple cracks of varying widths, directions, and depths. After obtaining two datasets, images with clear crack outlines were selected and cropped to obtain a number of crack images (e.g., 500).

[0029] Data annotation: In actual training, the dataset consists of two parts: the original images and manually annotated pixel-level binary ground truth masks corresponding one-to-one with the original images. Therefore, it is necessary to perform fine-grained pixel-level manual annotation on several collected crack images, where crack pixels are marked as 1 and background pixels are marked as 0. After the data annotation is completed, the final dataset is divided into training and test sets in a specific ratio (e.g., 8:2, 400 images for training and 100 images for test).

[0030] Data preprocessing: To balance model training speed, computational resource limitations, and computer hardware performance, all original images and their corresponding masks are uniformly cropped to a set size (e.g., 512x512). Please refer to [reference needed]. Figure 2 (In the figure, 1~20 represent numbers and png represents image format); and before being fed into the model, the pixel values ​​of the image are normalized (e.g., scaled to the [0, 1] range or standardized) to accelerate model convergence.

[0031] Data augmentation: To further improve the robustness and generalization ability of the model, data augmentation can be performed on the training set images. Augmentation techniques used include random horizontal flipping, random vertical flipping, random rotation, and random adjustment of brightness and contrast.

[0032] Step 2: The crack image is processed using the TransHRNet deep neural network model to generate crack semantic segmentation results. Specifically: The TransHRNet deep neural network model proposed in this method is based on an encoder-decoder architecture. Please refer to [link / reference needed]. Figure 4 (In the diagram, input means input and output means output), specifically including a hybrid encoder and a bottleneck layer (corresponding to...) Figure 4 The method includes a Bottleneck and a decoder; furthermore, during the training phase of the TransHRNet deep neural network model, an edge-aware supervision method is executed. Specifically: About hybrid encoders: In crack semantic segmentation, the encoder's role is to extract multi-level, multi-scale features from the input crack image. Traditional U-Net encoders consist of two consecutive 3x3 standard convolutional layers and a ReLU activation function at each stage. While this design is simple, it is prone to gradient vanishing, limiting its feature representation capabilities when dealing with objects with complex textures like cracks. To address this issue, this method constructs a hybrid encoder in the feature extraction stage, aiming to endow it with more powerful feature extraction capabilities.

[0033] In this embodiment, the encoder performs four downsampling stages, divided into shallow and deep encoder layers. Specifically, the first two downsampling stages correspond to the shallow encoder layers, each employing two efficient HRM convolutional modules (corresponding to...). Figure 4 The HRM in the first layer is used to extract local features; the third and fourth layers correspond to the deep layers of the encoder, and each layer uses two HRM convolutional modules and one STM self-attention mechanism module (corresponding to...). Figure 4 The structure combines STM (Structured Micrometer) and its self-attention mechanism module for global context modeling. Specifically: In the shallow layer of the encoder: The HRM convolutional module contains skip connections, allowing information and gradients to flow more smoothly within the network. This enables the network to be built deeper to extract more powerful features, while effectively avoiding the vanishing gradient problem. The core computational layer of the HRM convolutional module replaces standard convolutions with depthwise separable convolutions, thereby significantly reducing computational cost and the number of parameters while maintaining powerful feature extraction capabilities. Consequently, this TransHRNet deep neural network model far surpasses the efficiency of the traditional U-Net.

[0034] The HRM convolutional module, as an efficient hybrid residual module, has two parallel paths: a computation path and a residual path. Therefore, the module's output is a fusion of the outputs from the computation path and the residual path. Please refer to [link / reference]. Figure 5 (In the diagram, Short Path means the shortest path, Main Path means the main path, and HRM1 and HRM2 represent two HRM convolutional modules.) The computation path sequentially includes a first 1x1 convolutional layer, a 3x3 depthwise convolutional layer, and a second 1x1 convolutional layer. The process is as follows: First, a first 1x1 convolutional layer is used for channel dimensionality reduction, significantly reducing the number of channels in the input image to compress features, remove redundant information, and significantly reduce the computational cost of subsequent 3x3 convolutions. Then, a 3x3 depthwise convolutional layer is used for depthwise convolution to achieve spatial feature extraction; notably, each input channel has its own independent 3x3 convolutional kernel, which is only responsible for spatial feature extraction within each channel, especially edges and corners, without any information fusion between channels. After the 3x3 convolutions, a second 1x1 convolutional layer is used for pointwise convolution, weighting and combining the features of all channels to achieve information fusion and interaction between channels. Compared to the standard 3x3 convolution of traditional U-Net, this method decouples the spatial feature extraction and channel feature fusion tasks, significantly reducing computational cost while achieving similar feature extraction results.

[0035] The residual path preserves the original information of the input, directly passing the input to the output and ensuring that it is completely consistent with the final output size of the computational path so that it can be added to the output of the computational path. This residual structure allows information and gradients to flow more smoothly, effectively avoiding the gradient vanishing problem and allowing for the construction of deeper networks. At the end of the HRM convolutional module, the outputs of the computational path and the residual path are fused to ensure that the model learns new features without forgetting the important original information. Finally, a ReLU activation function is used to add non-linearity and output the final result.

[0036] Deep within the encoder: The traditional U-Net encoder is a pure convolutional neural network that gradually expands the receptive field by stacking convolutional layers, but it is still inherently better at capturing local features. Even at the deepest layers of the encoder, its ability to understand the global context of an image is limited, which can easily lead to misjudgments of deceptive local textures (such as stains and scratches). To address this problem, this method innovatively embeds an STM self-attention mechanism module, forming an HRM-STM-HRM combined structure, which includes a first HRM convolutional module, an STM self-attention mechanism module, and a second HRM convolutional module arranged sequentially. Specifically: The steps for deep encoding using this combined structure are as follows: A1: The feature map is fed into the first HRM convolutional module (stride of 2), which performs downsampling and convolutional feature preprocessing on the feature map.

[0037] A2: The preprocessed feature map is fed into the STM self-attention mechanism module based on windowed self-attention. Please refer to... Figure 7 This module, through its efficient self-attention mechanism, performs global context modeling on the feature map with controllable computational costs, in order to effectively capture macroscopic information such as the overall trend of the crack and long-distance dependencies.

[0038] A3: The feature map is fed into the second HRM convolutional module (with a stride of 1), which fuses the global features output by the STM self-attention mechanism module with the local features output by the shallow layer of the encoder.

[0039] In this method, the Swing Transformer is introduced into the deep, low-resolution stages of the encoder. This leverages its global modeling advantages while avoiding the enormous computational overhead of direct application to high-resolution images. The Transformer utilizes a self-attention mechanism to establish long-distance dependencies between any two pixels in an image, possessing powerful global context modeling capabilities. For example, it can understand that "a thin crack is a whole," rather than viewing it as a collection of isolated line fragments. Therefore, the TransHRNet deep neural network model in this method, by introducing an STM self-attention mechanism module deep into the encoder, fundamentally compensates for the insufficient global perspective of pure CNN networks, achieving a perfect combination of fine-grained local perception and macroscopic global understanding.

[0040] Regarding the bottleneck layer: A bottleneck layer is placed between the encoder and decoder; in this TransHRNet deep neural network model, the highest-level feature map output from the fourth layer of the encoder is fed into this bottleneck layer. The bottleneck layer includes two HRM convolutional modules for final downsampling and feature extraction to capture the most abstract crack semantic information, serving as a bridge connecting the encoder and decoder.

[0041] About the decoder: In this method, the decoder comprises four decoder modules, each including an AFM attention fusion module and an HRM convolution module. This allows for the reconstruction of pixel-level segmentation maps from the abstract features extracted by the bottleneck layer through a progressive upsampling approach. This method abandons the simple feature concatenation method in traditional U-Net and introduces an AFM attention fusion module to perform a feature fusion method. This method weightedly fuses the skip connection features from the encoder with the upsampled features from the decoder. The specific steps of this feature fusion method are as follows: T1: At each layer of the decoder, a spatial attention map is generated by using high-level semantic information contained in the upsampled features from deeper layers of the encoder or bottleneck layer as a guiding signal; the generation process of this spatial attention map is as follows: First, linear projections are performed on the upsampled features and skip connection features to align the channels; then the projected results are summed and activated by the ReLU function; finally, they are passed through a convolutional layer and activated by the Sigmoid function.

[0042] T2: Please refer to Figure 6 The spatial attention map is element-wise multiplied with the skip connection features from the corresponding stage of the encoder to obtain attention-weighted skip connection features, which dynamically enhance the effective feature regions related to cracks and suppress irrelevant background noise; the calculation formula for the skip connection features is as follows:

[0043] In the formula, This represents the skip connection features that have undergone attention weighting and are ready for subsequent splicing. This represents the skip connection features from the corresponding stage of the encoder; This indicates deeper upsampling features; and They respectively represent the uses for... and Perform linear projection to align the transformation matrix or network layer with the channel dimension; This represents the transformation layer used to generate a single-channel attention map; This indicates a modified linear unit activation function; This represents the Sigmoid activation function, which will... The output is compressed to the (0, 1) interval to generate a spatial attention weight map. ; This indicates element-wise multiplication.

[0044] T3: The attention-weighted skip connection features are concatenated with the upsampled features and fed into the HRM convolution module for final feature integration.

[0045] In the decoder proposed in this method, the highest-level semantic understanding of the encoder can be perfectly combined with the spatial details of each level through the cyclic processing of upsampling and attention fusion, and finally the pixel-level segmentation of cracks is both macroscopically accurate and microscopically fine. Based on this attention fusion mechanism, the effectiveness of skip connections is significantly improved, enabling the model to recover the fine structure of cracks more accurately.

[0046] Regarding edge-aware supervision methods: To further enhance the model's sensitivity to crack edges and improve segmentation accuracy, this method innovatively introduces a lightweight Edge Detection Branch (EDB) as an auxiliary supervision task during the model training phase. Its purpose is to strengthen the shallow layer of the model encoder's representation of crack boundary details, thereby improving the edge prediction accuracy of the main segmentation task. In practical applications, this step is removed during the model inference (deployment) phase, thus incurring no additional computational overhead.

[0047] This EDB edge detection branch specifically utilizes shallow feature maps from the hybrid encoder, and the edge-aware supervision method it performs includes the following steps: S1: Construct the EDB edge detection branch and input at least two feature maps from the shallow layer of the encoder into the EDB edge detection branch.

[0048] In one specific embodiment, the output feature map of the encoder's first stage is selected. Output feature map of the second stage These two layers are used as input to the EDB edge detection branch. This choice is made because these two layers retain the richest, unsampled, high-resolution spatial detail information, which is crucial for accurate edge localization.

[0049] S2: The EDB edge detection branch processes the feature map to generate an edge prediction map. Specifically, it includes the following sub-steps: S2-1: Perform spatial size matching and channel dimension adjustment on at least two feature maps.

[0050] In the above specific embodiments, the feature map and feature map With different spatial resolutions and channel dimensions, such as feature maps The dimensions are [B, C1, H / 4, W / 4], and the feature map is... The dimensions are [B,C2,H / 8,W / 8] (C1 and C2 are the number of channels, and [H / 4,W / 4] and [H / 8,W / 8] are the spatial dimensions); therefore, processing and alignment are required first.

[0051] Specifically, for the output feature map of the second stage Please refer to: Figure 3 First, a 1x1 convolutional layer is used to transform the features and reduce the channel dimensionality, adjusting the number of channels from C2 to a preset intermediate number of channels M. Then, bilinear interpolation and other upsampling methods are used to refine the processed feature map. The spatial dimensions are enlarged from [H / 8, W / 8] to [H / 4, W / 4], so that they are aligned with the feature map. The dimensions are matched. This applies to the output feature map of the first stage. The feature map is processed through another 1x1 convolutional layer in parallel with the above process. The number of channels is adjusted from C1 to the preset number of intermediate channels M.

[0052] S2-2: At least two feature maps that have undergone spatial size matching and channel dimension adjustment are concatenated and feature extraction is performed through a convolutional layer to generate edge feature representations.

[0053] In the specific embodiments described above, the processed and size-aligned feature map and feature map The feature maps are concatenated along the channel dimension to form a fused feature map containing 2M channels; this fusion operation combines the feature maps... The finest details and feature maps The fused feature map is then fed into a 3x3 convolutional layer for deep processing. This convolutional layer learns and integrates the spatial correlation between the two scale features to generate a more expressive edge feature representation containing M channels.

[0054] S2-3: The edge feature representation is fed into a prediction head, which maps the edge feature representation into a single-channel edge prediction map.

[0055] In the specific embodiment described above, the refined edge feature representation is fed into a final prediction head consisting of 1x1 convolutions. This prediction head is responsible for mapping M feature channels into a single-channel output; this output is the edge prediction map. In the image, the value of each pixel represents the confidence level that the location is the edge of a crack.

[0056] S3: Calculate the edge-aided loss based on the difference between the edge prediction map and the true edge label.

[0057] S4: The edge-assisted loss and the loss of the main segmentation task are weighted and summed to obtain the total loss function of the model, which is used to guide the update of the weight parameters in the model network. That is, by minimizing this joint loss function, the gradient of backpropagation will simultaneously optimize the parameters of the main segmentation path and the EDB edge detection branch, thereby achieving explicit supervision of the edge representation ability of the shallow layer of the encoder.

[0058] The expression for the total loss function of the model is as follows:

[0059] In the formula, This represents the total loss function of the model that needs to be minimized; This represents the loss of the main segmentation task, used to measure the difference between the segmentation map output by the model and the ground truth. This represents the edge-aided loss, used to measure the difference between the edge prediction map output by the EDB edge detection branch and the true edge label; This represents a hyperparameter, namely the balancing weight factor of the edge-assisted loss. The weight is used to adjust the contribution of the main segmentation task and the edge-assisted task during the model training process, so as to ensure that the model does not deviate from the goal of the main segmentation task while learning edge information.

[0060] In this method, by introducing an auxiliary task of edge-aware supervision, the model is forced to learn accurate edge localization. This not only directly improves the feature quality of the first two encoder layers, but also, through the skip connections and attention fusion mechanism in the decoder, these high-quality features can be effectively transferred to the final segmentation result, thereby significantly improving the clarity, continuity and integrity of crack boundaries and solving the problem of blurred edges in the prior art.

[0061] Based on the above structure and method, in a specific embodiment, the overall working process of the TransHRNet deep neural network model can be briefly described as follows: The TransHRNet deep neural network model is trained and tested on the PyTorch platform. During the model training phase, the preprocessed image tensor (size [B,3,512,512]) is input into the hybrid encoder.

[0062] In the hybrid encoder, in the first and second downsampling layers, a pure convolutional network composed of HRM convolutional modules is used to extract local details and texture features of the image, and then downsampled step by step to generate feature maps. and feature map In the third and fourth layers, a combined HRM-STM-HRM structure is used to downsample the feature maps after they have been processed by the HRM convolutional module. These maps are then fed into the STM self-attention mechanism module for global context modeling to capture long-range dependencies in the cracks. Finally, a hierarchical feature map containing rich local details and global semantic information is output. and feature map The feature map output by the encoder The bottleneck layer performs the deepest semantic extraction to obtain the bottleneck feature b.

[0063] The upsampled features then enter the attention decoder for progressive upsampling. At each layer of decoding, the upsampled features from the bottleneck layer are fused with the skip connection features from the corresponding stage of the encoder. This fusion process is completed by the AFM attention fusion module. The upsampled features are used as a guiding signal to generate a spatial attention map, and then the skip connection features are weighted to highlight crack-related features and suppress noise. This process is repeated progressively during upsampling to gradually restore the abstract features to a high-resolution, fine-grained feature map.

[0064] The feature map output by the decoder is restored to its original size through a final upsampling layer, and then passed through a 1x1 convolutional segmentation head to generate an original prediction result (logits) of size [B, 2, 512, 512]. Further, a joint loss function is calculated, and the original prediction result obtained from forward propagation is used to... The main segmentation loss is calculated by comparing it with the true ground truth mask, specifically using a combination of cross-entropy loss and Dice loss.

[0065] Meanwhile, the feature map of the shallow layer of the encoder and feature map It will be fed into the EDB edge detection branch to generate an edge prediction map. Then, the predicted edge map is compared with the edge labels generated in real time from the ground truth mask, and the edge auxiliary loss is calculated. After calculating the total loss function based on the expression of the model's total loss function, the weight parameters of the entire network (including the backbone network and the EDB edge detection branch) are updated through the backpropagation algorithm.

[0066] Please refer to Figure 8The diagram illustrates the loss functions during training. In the figure: Total Loss represents the overall loss; Main Loss represents the main segmentation loss, composed of Focal Loss and Dice Loss; Dice Loss primarily focuses on the spatial overlap between the predicted results and the true labels; Focal Loss is an improved cross-entropy loss, specifically focusing on pixels at object edges. After Dice Loss has helped the model find the general outline, Focal Loss concentrates on optimizing blurry, difficult pixels located at the edges of cracks, thus finely optimizing the segmentation results; Edge Loss represents the edge-assisted loss; Main Loss, as the loss for the core task, shows a decreasing trend largely consistent with the total loss. Specifically, Main Loss rapidly decreases from 0.962 and eventually stabilizes at around 0.07, indicating that the main part of the model (the architecture composed of the encoder and decoder) performs well in learning the segmentation task. Furthermore, the efficient descent of Edge Loss proves that the Edge Detection Branch (EDB) is learnable and successfully captures edge information from shallow features. More importantly, its descent does not conflict with the descent of Main Loss, indicating that learning edge information not only does not interfere with the main segmentation task but also plays a positive role in promoting it. This perfectly validates the original design intent of this method: to help the main network better characterize crack boundaries by explicitly supervising edges, thereby improving the overall segmentation quality.

[0067] In practical applications, the AdamW optimizer is used to optimize the model, and a cosine annealing learning rate scheduling strategy is used. The above training steps are iterated 300 times on the training set until the model's performance metrics (mean intersection-union ratio mIoU and mean Dice) on the validation set converge.

[0068] In addition, to verify the actual performance and beneficial effects of the proposed TransHRNet deep neural network model, edge-aware supervision method, and attention fusion mechanism, this embodiment performs a comprehensive performance evaluation of the trained TransHRNet deep neural network model after training is completed, and compares it with existing traditional semantic segmentation models. The specific process and results are as follows: This validation experiment was conducted on the same hardware and software platform as the training steps, ensuring environmental consistency. Furthermore, an independent test set not used in training was employed for the final performance evaluation to guarantee the objectivity and fairness of the results. Specifically, the classic U-Net architecture, widely recognized in semantic segmentation, was selected as the baseline model. To ensure fairness in the comparison, this U-Net model was trained under the same experimental conditions (using the same dataset, data augmentation, and training epochs). Mean Intersection over Union (mIoU) and mean Dice coefficient were used as quantitative evaluation metrics. These two metrics accurately measure the degree of overlap between the predicted crack regions and the ground truth labeled regions, and are considered the gold standard for evaluating the performance of segmentation tasks.

[0069] After a complete training process, the TransHRNet deep neural network model proposed in this invention demonstrates excellent segmentation performance on the test set. In one specific embodiment, the best-performing model achieved the following on the test set: an average intersection-union ratio (mIoU) of 0.7796 and an average Dice coefficient of 0.8739; some semantic segmentation results are shown below. Figure 9 As shown in the figure (a7~a217 represent numbers, png represents image format), the results indicate that the method of the present invention can perform high-precision pixel-level segmentation of crack images and effectively distinguish cracks from the background.

[0070] The performance of the model of this invention is directly compared with that of the traditional U-Net baseline model. The results are shown in the table below: Table 1 Performance data of TransHRNet deep neural network model and traditional U-Net baseline model

[0071] The above experimental results provide strong quantitative evidence for the beneficial effects claimed in this invention. Compared to the pure CNN encoder of traditional U-Net, the model of this invention achieves significant improvements in both the average intersection-over-union ratio (IoU) and the average Dice coefficient. This demonstrates the superior performance of the hybrid encoder in this invention, which, by combining local detail awareness with global context modeling, can extract more discriminative features, thereby achieving more robust crack recognition in complex backgrounds. Furthermore, the significant performance improvement is also attributed to the intelligent filtering of skip connection features by the AFM attention fusion module and the enhancement of the network's shallow edge representation capabilities by the EDB edge detection branch during the training phase. The synergistic effect of these two innovations enables the model of this invention to more accurately recover the fine structure of cracks and ensure the clarity and continuity of boundaries in the final segmentation results. This beneficial effect is directly reflected in the higher average IoU and average Dice coefficient values.

[0072] In summary, the quantitative experimental results of this embodiment fully and powerfully verify the advanced nature and feasibility of the technical solution of the present invention, as well as its significant beneficial effects in improving the accuracy and robustness of crack detection, from both the perspectives of absolute performance and relative advantages.

[0073] It should be noted that the parts not described in detail or in elaboration in the above solutions are all prior art and do not constitute improvements made by this invention to existing technology, nor are they within the protection scope of this invention's technical solutions. Therefore, they will not be elaborated upon further in this document. Of course, the above content is merely a preferred embodiment of this invention and should not be considered as limiting the scope of the embodiments of this invention. This invention is also not limited to the above examples; equivalent changes and improvements made by those skilled in the art within the substantial scope of this invention should all fall within the patent coverage of this invention.

Claims

1. A crack semantic segmentation method based on global-local feature fusion and edge perception, characterized in that, Includes the following steps: Step 1: Obtain the crack image to be processed; Step 2: The crack image is processed using the TransHRNet deep neural network model to generate crack semantic segmentation results; The TransHRNet deep neural network model includes an encoder and a decoder, wherein: The encoder includes: an HRM convolution module for extracting local features, and an STM self-attention mechanism module for performing global context modeling. The decoder includes an AFM attention fusion module, which is used to perform weighted fusion of skip connection features from the encoder and upsampled features from the decoder. During the training phase of the TransHRNet deep neural network model, the following edge-aware supervision method is implemented: an EDB edge detection branch is introduced, which uses the features of the shallow layer of the encoder to learn and predict the boundary of the crack, thereby enhancing the ability of the shallow layer of the encoder to represent the details of the crack boundary.

2. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 1, characterized in that, The edge-aware supervision method specifically includes the following steps: S1: Construct the EDB edge detection branch and input at least two feature maps from the shallow layer of the encoder into the EDB edge detection branch; S2: The EDB edge detection branch processes the feature map to generate an edge prediction map; S3: Calculate the edge-aided loss based on the difference between the edge prediction map and the actual edge labels; S4: The edge-assisted loss and the loss of the main segmentation task are weighted and summed to obtain the total loss function of the model, which is used to guide the update of the weight parameters in the model network.

3. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 2, characterized in that, S2 includes the following sub-steps: S2-1: Perform spatial size matching and channel dimension adjustment on at least two of the feature maps; S2-2: At least two feature maps that have undergone spatial size matching and channel dimension adjustment are concatenated and feature extraction is performed through a convolutional layer to generate edge feature representations; S2-3: The edge feature representation is fed into a prediction head, which maps the edge feature representation into a single-channel edge prediction map.

4. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 2, characterized in that, The expression for the total loss function of the model is as follows: In the formula, This represents the total loss function of the model. This represents the loss of the main segmentation task. Indicates edge-assisted loss, This represents the balancing weight factor for edge-assisted loss.

5. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 1, characterized in that: The encoder performs multiple encoding stages, which are divided into shallow encoder layers and deep encoder layers; The shallow layer of the encoder uses an HRM convolution module; the deep layer of the encoder uses a combined structure including an HRM convolution module and an STM self-attention mechanism module.

6. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 5, characterized in that: In the shallow layer of the encoder, the HRM convolution module has two paths: a computation path and a residual path. The computation path includes a first 1x1 convolutional layer, a 3x3 depth convolutional layer, and a second 1x1 convolutional layer. The output of the HRM convolution module is the result of information fusion between the output of the computational path and the output of the residual path.

7. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 5, characterized in that: The combined structure includes a first HRM convolutional module, an STM self-attention mechanism module, and a second HRM convolutional module arranged in sequence. The steps for encoding the deep layers of the encoder using this combined structure are as follows: A1: The feature map is fed into the first HRM convolutional module, which preprocesses the feature map; A2: The preprocessed feature map is fed into the STM self-attention mechanism module, which performs global context modeling on the feature map and captures global features. A3: The feature map is fed into the second HRM convolutional module, which fuses the global features output by the STM self-attention mechanism module with the local features output by the shallow layer of the encoder.

8. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 1, characterized in that: A bottleneck layer is provided between the encoder and the decoder; the bottleneck layer includes two HRM convolutional modules.

9. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 1, characterized in that: The decoder includes several decoder modules, and each decoder module includes an AFM attention fusion module and an HRM convolution module; The AFM attention fusion module performs a feature fusion method, including the following steps: T1: At each layer of the decoder, the upsampled features from deeper layers are used as guiding signals to generate a spatial attention map; T2: The spatial attention map is then attention-weighted with the skip connection features from the corresponding stage of the encoder; T3: The attention-weighted skip connection features are concatenated with the upsampled features and fed into the HRM convolution module for feature integration.

10. The crack semantic segmentation method based on global-local feature fusion and edge awareness according to claim 9, characterized in that: In T1, the spatial attention map is generated as follows: first, the upsampled features and skip connection features are linearly projected to align the channels; then the projected results are summed and activated by the ReLU function; then, they are passed through a convolutional layer and the Sigmoid activation function.

Images