Crack detection image segmentation method based on improved CNN-Transform hybrid architecture

Abstract

The invention discloses a crack detection image segmentation method based on an improved CNN-Transform hybrid architecture. The method comprises the following steps: step 1, inputting and preprocessing an image; 2, coding the mixed features; 3, semantic modeling is carried out; step 4, performing cascade decoding; step 5, outputting a crack mask; and step 6, loss function design and training optimization. The invention provides an improved CNN-Transform hybrid architecture, a CNN module with depth separable convolution and transfer learning optimization is organically combined with a lightweight Transform, the precision and robustness of crack identification are improved, the model volume and the operation load are effectively reduced, and the method has wide engineering application prospects and technical popularization values and is suitable for popularization and application. The method is particularly suitable for concrete structure crack detection scenes and can be widely applied to intelligent crack inspection of civil engineering such as dam bodies, bridges, tunnels and subway linings.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and structural health monitoring technology, and relates to an image segmentation method, specifically an image segmentation method based on a lightweight CNN-Transformer hybrid attention network. Background Technology

[0002] In recent years, image semantic segmentation technology has developed rapidly, driven by deep learning, providing strong technical support for the automatic detection of concrete cracks. Image segmentation models based on convolutional neural networks (CNNs), such as U-Net, DeepLabV3+, and FCN, have been widely used in building defect detection scenarios. These models extract local texture and edge features through stacked convolution operations and achieve pixel-level target recognition and segmentation through multi-level feature fusion. However, due to the limited receptive field of the convolution kernel itself, these models often lack global perception capabilities for targets such as cracks, which have slender shapes, irregular distributions, and large scale variations, and are prone to problems such as incomplete recognition and missegmentation.

[0003] To address the limitations of CNN models in global modeling, the Transformer architecture has been introduced into computer vision tasks in recent years. Vision Transformer (ViT) and its variants can model long-range dependencies across the entire image using a self-attention mechanism, thus compensating for the shortcomings of CNNs in capturing semantic information. However, Transformer models also have inherent disadvantages: firstly, their computational complexity increases quadratically with input resolution, making them unsuitable for deployment on mobile devices; secondly, their ability to extract low-level visual features is inferior to CNNs, especially in recognizing fine-grained textures such as cracks. Therefore, using either CNN or Transformer alone makes it difficult to maintain detection accuracy while simultaneously achieving efficiency and lightweight design.

[0004] To address the aforementioned issues, the academic community has gradually proposed a hybrid structure of "CNN + Transformer," combining the advantages of both to achieve stronger expressive power and wider application adaptability. This type of hybrid structure typically uses CNNs for local detail extraction of images, while Transformers are used to model semantic information across regions, thus achieving breakthroughs in multi-scale information fusion and structural recognition. However, existing methods still suffer from problems such as complex network structures, large number of parameters, and high computational resource consumption, making it difficult to meet the engineering requirements of on-site deployment and real-time detection on mobile terminals. Especially in concrete crack detection scenarios, the model needs to simultaneously possess high accuracy, fast response, and low power consumption to support long-term, low-energy, and unmanned intelligent inspection.

[0005] Therefore, there is an urgent need for a hybrid network structure that is highly accurate, has strong generalization ability, and is lightweight, capable of balancing global modeling and local detail representation, and can be deployed on mobile devices. Summary of the Invention

[0006] To address the problems of "the incompatibility between local details and global semantics," "high computational resource consumption," and "difficulty in mobile deployment" in existing crack detection methods, this invention provides a crack detection image segmentation method based on an improved CNN-Transformer hybrid architecture. This method proposes an improved CNN-Transformer hybrid architecture that organically combines depthwise separable convolution, transfer learning-optimized CNN modules, and a lightweight Transformer. This not only improves the accuracy and robustness of crack recognition but also effectively reduces model size and computational load, possessing broad engineering application prospects and technological promotion value. It is particularly suitable for crack detection in concrete structures and can be widely applied to intelligent crack inspection in civil engineering projects such as dams, bridges, tunnels, and subway linings.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A crack detection image segmentation method based on an improved CNN-Transformer hybrid architecture includes the following steps:

[0009] Step 1: Image Input and Preprocessing:

[0010] Step 1.1: Image Acquisition: Acquire color images of the concrete structure surface using a mobile terminal device;

[0011] Step 1.2: Image scaling: Scale the acquired images to a uniform size using bilinear interpolation.

[0012] Step 1.3: Color and Grayscale Normalization: Normalize the image pixels;

[0013] Step 1.4: Tensor Construction: Convert the image into a 3D tensor;

[0014] Step 2: Hybrid Feature Encoding:

[0015] Step 2.1: Backbone Network Selection and Structure Optimization: An improved VGG19 was selected as the backbone feature extraction network. Specifically, the parameters of the first three convolutional layers of VGG19 were frozen to maintain the low-level feature extraction capability. Only the subsequent layers were fine-tuned. The fine-tuning layers were located in the higher layers of VGG19, namely the Conv4_x and Conv5_x layers. A small learning rate (1 / 10 of the learning rate of the new layer) was set for these layers to avoid weight oscillations and ensure stable convergence. After the output of the Conv5_x layer, a 1×1 convolutional compression layer was added to compress the channels of the high-level feature maps and unify the output dimension, thereby reducing computational complexity and providing concise and efficient input features for the subsequent Transformer module. Following the fine-tuning layer and the 1×1 convolutional compression layer, a further 1×1 convolutional layer is introduced in the decoder to further optimize the feature fusion process. To prevent overfitting and improve generalization ability, BatchNorm and Dropout (with a ratio of 0.3 to 0.5) are introduced in the fine-tuning layer. During the improvement process, the original fully connected layers (fc6, fc7, fc8) of VGG19 are removed and replaced with lightweight convolutional layer structures to reduce computational cost and enhance the model's transferability and training stability. Specifically, after the high-level feature maps, feature dimensionality reduction and semantic mapping are achieved through 1×1 convolutions and global average pooling layers, thus avoiding the large number of parameters and computational burden brought by traditional fully connected layers. This not only effectively reduces computational complexity and memory overhead but also improves the network's transfer performance on small-sample crack datasets, making training more stable and convergence faster.

[0016] Step 2.2: Multi-scale feature extraction: The input image is processed through layer-by-layer convolution and pooling operations to extract feature maps at different scales;

[0017] Step 2.3: After the mid-to-high level output, introduce a 1×1 convolution to perform channel compression and dimension unification processing;

[0018] Step 2.4: Finally, output three sets of feature maps at different scales: low-level texture map, mid-level structure map, and high-level semantic map;

[0019] Step 3: Semantic Modeling

[0020] Step 3.1: Patch partitioning and flattening: Divide the output feature map of the encoder intermediate layer into image patches and flatten them into a 1D vector sequence in spatial order; each patch is a feature sequence.

[0021] Step 3.2: Embedding layer mapping: Input each patch into a linear transformation layer to convert it into a feature embedding vector of uniform dimension;

[0022] Step 3.3: Position Encoding: To preserve spatial position information, absolute position encoding is added to incorporate spatial order into the feature sequence;

[0023] Step 3.4 Multi-head self-attention calculation: Input the sequence into the Transformer backbone;

[0024] Step 3.5: Output Reconstruction: The output sequence is rearranged into a 2D feature map, which is then concatenated or weighted and fused with the original CNN feature map to construct an representation tensor that combines global semantics and local details.

[0025] Step 4: Cascaded Decoding:

[0026] Step 4.1: Decoding: The decoder consists of 4 upsampling units, each responsible for doubling the size of the feature map and finally restoring it to the same size as the original image;

[0027] Step 4.2: Upsampling: Spatial resolution is increased by using bilinear interpolation upsampling or deconvolution;

[0028] Step 4.3: Depthwise Separable Convolution: After each level of scaling, Depthwise Separable Convolution is introduced: first, intra-channel convolution is performed, and then 1×1 channel fusion is performed;

[0029] Step 4.4: Skip Connection Fusion: Introduce encoder feature maps of the same scale as the decoding layer, and combine them with the decoding feature maps through channel splicing or weighted fusion to achieve cross-layer information fusion and edge detail restoration;

[0030] Step 4.5: Regularization: Apply Dropout or regularization restrictions after decoding each layer;

[0031] Step 5: Crack Mask Output:

[0032] Step 5.1: Channel Mapping and Dimension Compression: Compress the channels of the final feature map using 1×1 convolution to output a single-channel or dual-channel prediction map;

[0033] Step 5.2: Activation function normalization: Output the crack probability map in the 0~1 interval through the Sigmoid function, or use Softmax to perform pixel-level multi-class distribution fitting;

[0034] Step 5.3: Binarization threshold setting:

[0035] Step 5.3.1: In mobile or embedded device deployments, when the Sigmoid output value is greater than the threshold, it is determined to be a crack pixel;

[0036] Step 5.3.2: In the evaluation or fine annotation stage, determine the optimal segmentation threshold to adapt to different image qualities and crack morphologies. The final mask image is a 0 / 1 binary image.

[0037] Step 5.4: Visualization and Overlay: Overlay the masking results onto the original image, highlighting the crack areas with red, green, or transparent masks to achieve a comparative display;

[0038] Step 5.5: Output Management:

[0039] Step 5.5.1: Upload the results to the edge gateway or monitoring terminal via Socket or HTTP protocol;

[0040] Step 5.5.2: Upload the crack detection results to the cloud platform by calling the RESTful API or MQTT protocol;

[0041] Step 5.5.3: If the model is deployed on an Android APP, the recognition results are sent back to the APP interface for image preview, result saving, and user tagging feedback to achieve a closed loop;

[0042] Step 6: Loss Function Design and Training Optimization:

[0043] Step 6.1: Joint Loss Function Design: A joint loss function mechanism combining Dice Loss and Focal Loss is adopted;

[0044] Step 6.2: Optimizer Setup and Learning Rate Scheduling:

[0045] Step 6.2.1: Use the AdamW optimizer;

[0046] Step 6.2.2: Set the base learning rate to 10. -4 ;

[0047] Step 6.2.3: Use the cosine annealing learning rate decay mechanism to make the learning rate gradually and smoothly decay during the training process;

[0048] Step 6.2.4: In the first 5 to 10 epochs, use a linear warm-up approach to slowly increase the learning rate;

[0049] Step 6.3: Regularization Strategies and Data Augmentation:

[0050] Step 6.3.1: Introduce Dropout in the intermediate feature layer;

[0051] Step 6.3.2: Introduce L2 penalty to the model weights;

[0052] Step 6.3.3: Enhance the diversity of training data through online or offline data augmentation operations;

[0053] Step 6.3.4: Oversample the cracked samples in the training dataset, or undersample the samples without cracks;

[0054] Step 6.4: Model Compression and Multi-Terminal Deployment:

[0055] Step 6.4.1: After training, perform structural pruning on some redundant channels or layers in the model, and optimize the lightweight student model using knowledge distillation techniques;

[0056] Step 6.4.2: Use the Post-training Quantization strategy to convert the model from a 32-bit floating-point representation to an 8-bit fixed-point format;

[0057] Step 6.4.3: Export the model as ONNX format for use in the PyTorch / ONNXRuntime environment.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] 1. Innovative Hybrid Attention Structure: Integrating Local Convolutional Features and Global Attention Information. Traditional image segmentation methods based on Convolutional Neural Networks (CNNs) perform well in local feature extraction, but their limited receptive field makes it difficult to effectively model long-distance dependencies in crack images. While the Transformer structure has the ability to model global semantics, it is weak in representing low-level details and has high resource consumption. This invention innovatively decouples and integrates a lightweight CNN encoder with a Transformer attention module, designing a "local-global hybrid modeling" strategy for crack images. By introducing a multi-head self-attention mechanism into the mid-level feature map, it not only retains the ability of convolution to capture fine-grained features such as edge textures, but also significantly enhances the coherence and consistency of the model in modeling long-path crack structures.

[0060] 2. Deployability-Oriented Lightweight Structure Design: Addressing the issues of redundant parameters and difficult deployment in traditional semantic segmentation models, this invention introduces several lightweight techniques, including but not limited to: 1) Using a simplified version of VGG19 for the backbone network, freezing some convolutional groups to train only high-level semantic layers, reducing training costs; 2) Replacing standard convolutions with four-tiered depthwise separable convolutions in the decoding structure, significantly reducing computational complexity and model size; 3) Limiting the Transformer part to four EncoderBlocks, maintaining global modeling effectiveness while suppressing network complexity. In actual testing, the model parameters of this invention are controlled within 9M, the compressed model file is less than 8MB, and the average inference time is no more than 80ms, fully meeting the online deployment requirements of platforms with limited computing power, such as Jetson Nano and Android mobile devices.

[0061] 3. Robust Optimization Mechanism for Small Samples Applicable to Crack Target Characteristics: Concrete cracks in images are often characterized by being small, discontinuous, and having low contrast, making them typical small and weakly textured targets. To improve the detection accuracy for this type of target, this invention innovates in the design of the loss function, proposing to combine Dice Loss (increasing the overlap between the predicted and ground truth regions) and Focal Loss (suppressing easily classified samples and enhancing attention to difficult-to-classify samples) into a weighted joint loss function. This effectively alleviates the problem of class imbalance between foreground and background, and strengthens the model's response capability to the edge regions of small cracks. Simultaneously, in terms of data augmentation, strategies such as affine transformation, brightness perturbation, random occlusion, and crack synthesis interpolation are introduced, significantly improving the robustness of the model under different shooting conditions and background environments.

[0062] 4. A complete closed-loop integrated detection-display-output process design: Unlike traditional detection models that only generate prediction maps, this invention's system constructs a structured mask output and visualization integration mechanism at the model output end. Specifically, this includes: generating a crack probability map based on 1×1 convolution, with Sigmoid activation for output; supporting fixed and adaptive thresholds to generate binary mask maps; allowing crack masks to be superimposed on the original image to generate a highlighted display image for easy human-computer interaction; automatically outputting structured crack information (coordinates, length, direction, area), and supporting the export of detection reports in JSON or PDF format for convenient backend platform integration. This closed-loop solution not only improves detection accuracy but also enhances the system's engineering practicality and integration adaptability, making it suitable for scenarios such as smart tunnels, bridge BIM platforms, and hydraulic structure safety early warning systems. Attached Figure Description

[0063] Figure 1 This is a diagram of the CNN-Transformer hybrid architecture.

[0064] Figure 2 This is a diagram of the transformer network structure. Detailed Implementation

[0065] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0066] This invention provides a crack detection image segmentation method based on an improved CNN-Transformer hybrid architecture, the method comprising the following steps:

[0067] Step 1: Image Input and Preprocessing:

[0068] Step 1.1: Image Acquisition: Obtain color images of the concrete structure surface through mobile terminal devices (such as mobile phones, industrial cameras, or drone-mounted modules). The images may have issues such as changes in lighting, tilted shooting, or inconsistent resolution.

[0069] Step 1.2: Image scaling: Scaling the acquired images to a uniform size (512×512 pixels) using bilinear interpolation to reduce geometric distortion and ensure consistency in subsequent feature extraction.

[0070] Step 1.3: Color and grayscale normalization: Normalize the image pixels (divide by 255 to normalize to [0,1]), and optionally use enhancement methods such as RGB-to-Gray transformation and histogram equalization to improve the model's ability to identify dark cracks.

[0071] Step 1.4: Tensor Construction and Batch Loading: Convert the image into a 3D tensor (C×H×W). If it is a training process, pack it into the data loader in batches to provide a standardized data format for subsequent network input.

[0072] Step 2: Hybrid Feature Encoding:

[0073] Step 2.1: Backbone Network Selection and Structure Optimization: An improved VGG19 network was selected as the backbone feature extraction network, balancing model complexity and expressive power. This network has a deep hierarchical structure and a uniform receptive field expansion strategy, making it suitable for multi-scale representation of slender targets such as cracks. To adapt to the crack detection task, the parameters of the first three convolutional layers of the network were frozen, and only the higher layers were fine-tuned. A small learning rate (e.g., 1 / 10 of the original learning rate) was set for Conv4_x and Conv5_x to avoid weight oscillations; while the newly added 1×1 convolutional compression layer and the decoder part used the standard learning rate to ensure fast convergence. BatchNorm and Dropout (probabilities 0.3~0.5) were introduced in the fine-tuning layers to prevent overfitting and improve generalization performance. The fully connected layer structure in the original VGG was removed because this structure is mainly used for classification tasks, while in image segmentation tasks, fully connected layers are not suitable for processing spatial information. Removing the fully connected layer can effectively reduce the computational cost and improve the model's transferability and training stability.

[0074] Step 2.2: Multi-scale Feature Extraction Mechanism: The input image is processed through layer-by-layer convolution and pooling operations to extract feature maps at different scales. Low-level features focus on low-order visual information such as crack edges and textures; mid-level features focus on region structure and crack direction; high-level features focus on semantic fusion expression, suitable for information interaction with the Transformer. Each convolutional layer is configured with BatchNorm and ReLU activation functions to improve non-linear expression and training convergence.

[0075] Step 2.3: To reduce the input complexity of subsequent modules, a 1×1 convolution is introduced after the mid-to-high-level outputs for channel compression and dimension unification, ensuring that the feature maps retain sufficient discriminative power without incurring redundant computational burden. This also facilitates the serialization input of the Transformer module.

[0076] Step 2.4: The final output consists of three sets of feature maps at different scales: a low-level texture map, a mid-level structure map, and a high-level semantic map, which are used for subsequent skip connections, global semantic modeling, and upsampling fusion, respectively. This hierarchical design provides the model with stronger spatial and semantic expressive capabilities, helping to improve the edge fidelity and structural integrity of crack segmentation.

[0077] Step 3: Semantic Modeling

[0078] Step 3.1: Patch partitioning and flattening: Divide the output feature map of the encoder intermediate layer into 16×16 or 32×32 image blocks (Patches) and flatten them into a 1-dimensional vector sequence in spatial order; each Patch is a feature sequence.

[0079] Step 3.2: Embedding layer mapping: Input each patch into a linear transformation layer to convert it into a feature embedding vector of uniform dimension (256 dimensions).

[0080] Step 3.3: Location Encoding: To preserve spatial location information, learnable or fixed absolute location encoding is added to integrate spatial order into the feature sequence.

[0081] Step 3.4 Multi-head Self-Attention Calculation: Input the sequence into the Transformer backbone, which includes several EncoderBlocks. Each Block contains: a Multi-head Self-Attention module to calculate the weighted dependency between any two positions; a FeedForward network module; layer normalization and residual connections.

[0082] Step 3.5: Output Reconstruction: The output sequence is rearranged into a 2D feature map, which is then concatenated or weighted and fused with the original CNN feature map to construct an representation tensor that combines global semantics and local details.

[0083] Step 4: Cascaded Decoding:

[0084] Step 4.1: Decoding Structure: The decoder consists of 4 upsampling units, each responsible for doubling the size of the feature map and finally restoring it to the same size as the original image.

[0085] Step 4.2: Upsampling method: Use bilinear interpolation upsampling or deconvolution (Transposed Conv) to increase spatial resolution, balancing smoothness and reconstruction accuracy.

[0086] Step 4.3: Depthwise Separable Convolution: After each level of amplification, Depthwise Separable Convolution is introduced: First, intra-channel convolution is performed (to extract spatial structure), and then 1×1 channel fusion is performed, which reduces the number of parameters while retaining the ability to perceive structure.

[0087] Step 4.4: Skip Connection Fusion: Introduce encoder feature maps of the same scale as the decoding layer, and combine them with the decoding feature maps through channel splicing or weighted fusion to achieve cross-layer information fusion and edge detail restoration.

[0088] Step 4.5: Regularization mechanism: Apply Dropout or regularization constraints after decoding each layer to avoid overfitting and improve the generalization ability of the decoder.

[0089] Step 5: Crack Mask Output:

[0090] Step 5.1: Channel Mapping and Dimension Compression: Compress the channels of the final feature map using 1×1 convolution to output a single-channel (grayscale) or dual-channel (crack / non-crack) prediction map.

[0091] Step 5.2: Activation function normalization: Output the crack probability map in the 0~1 interval through the Sigmoid function, or use Softmax to perform pixel-level multi-class distribution fitting.

[0092] Step 5.3: Binarization threshold setting:

[0093] Step 5.3.1: In mobile or embedded device deployments, to pursue inference efficiency, p=0.5 is used as the threshold by default, that is, when the Sigmoid output value is greater than 0.5, it is determined to be a crack pixel.

[0094] Step 5.3.2: During the evaluation or fine-grained annotation stage, the optimal segmentation threshold can be automatically determined using the Otsu algorithm, the maximum inter-class variance method, or a custom heuristic strategy, adapting to different image qualities and crack morphologies. The final mask image is a 0 / 1 binary image, serving as the basis for subsequent overlay, output, and evaluation.

[0095] Step 5.4: Visualization and Overlay: Overlay the masking results onto the original image, highlighting the crack areas with red, green, or transparent masks to achieve a comparative display.

[0096] Step 5.5: Output Management:

[0097] Step 5.5.1: Supports saving multiple formats such as masked images, overlay images, and original images with annotations (e.g., PNG, TIFF, JSON, etc.) for easy review later;

[0098] Step 5.5.2: Upload the results to the edge gateway or monitoring terminal via Socket or HTTP protocol;

[0099] Step 5.5.3: Upload the crack detection results to the cloud platform by calling the RESTful API or MQTT protocol, supporting access to smart water conservancy, digital bridge, and smart tunnel systems;

[0100] Step 5.5.4: If the model is deployed on an Android APP, the recognition results can be sent back to the APP interface for image preview, result saving, user tagging feedback, and other functions to achieve a closed loop.

[0101] Step 6: Loss Function Design and Training Optimization:

[0102] Step 6.1: Joint Loss Function Design: To effectively improve the robustness and accuracy of the network in semantic segmentation tasks, this invention adopts a joint loss function mechanism of Dice Loss and Focal Loss:

[0103] The Dice Loss function is primarily used to optimize the overlap between the foreground (crack) region and the prediction region, and is particularly sensitive to small object segmentation tasks. Its calculation formula is:

[0104]

[0105] Where TP represents the number of pixels correctly predicted as cracks, FP represents the number of pixels that were misclassified as cracks from the background, and FN represents the number of pixels that were misclassified as background from the cracks. Dice Loss can enhance attention to small crack areas, prevent crack areas from being overly ignored, and effectively improve IoU (Intersection over Union) and F1-score.

[0106] Focal Loss Function: This loss function is used to address the problem of crack categories having an extremely low proportion in the samples. Its formula is as follows:

[0107]

[0108] Where p is the predicted probability, γ is the focusing factor (usually taken as 2), and α is the inter-class balance factor. Focal Loss can adaptively reduce the focus on easily classified samples and enhance the ability to distinguish difficult-to-classify samples (blurred boundaries, local reflections, false cracks).

[0109] The two are weighted and combined into a total loss function:

[0110]

[0111] In this invention, λ1 = 0.5 and λ2 = 1.0. This combination method exhibits good convergence and accuracy in model training and can effectively handle the task of detecting small target cracks in complex backgrounds.

[0112] Step 6.2: Optimizer Setup and Learning Rate Scheduling:

[0113] Step 6.2.1: The AdamW optimizer (with weight decay) is adopted, which has advantages over the traditional SGD in terms of convergence speed and generalization performance, and is especially suitable for deep neural networks.

[0114] Step 6.2.2: Set the base learning rate to 10. -4 Layered learning rates can be set for different parameter sets; for example, the Transformer module uses smaller initial values ​​to avoid gradient oscillations.

[0115] Step 6.2.3: Use the cosine annealing learning rate decay mechanism to gradually and smoothly decay the learning rate during training, preventing fluctuations in the later stages of training.

[0116] Step 6.2.4: In the first 5 to 10 epochs, the learning rate is slowly increased using a linear warm-up approach, which helps the model to be stably initialized in the early stages.

[0117] Step 6.3: Regularization Strategies and Data Augmentation:

[0118] Step 6.3.1: Introduce Dropout (proposal rate 0.3~0.5) into the intermediate feature layer to prevent overfitting of feature representation and improve the model's generalization ability.

[0119] Step 6.3.2: Introduce L2 penalty to the model weights to prevent the parameters from growing indefinitely.

[0120] Step 6.3.3: Enhance the diversity of training data through online or offline data augmentation operations, including but not limited to: random horizontal / vertical flipping; random cropping and scaling; color perturbation (such as brightness / contrast changes); Gaussian blur and salt-and-pepper noise to simulate real scenes; affine transformation and rotation to enhance crack orientation adaptability.

[0121] Step 6.3.4: Oversample the cracked samples in the training dataset or undersample the samples without cracks to alleviate the class imbalance problem.

[0122] Step 6.4: Model Compression and Multi-Terminal Deployment:

[0123] Step 6.4.1: After training, perform structural pruning on some redundant channels or layers in the model, and optimize the lightweight student model using knowledge distillation techniques to reduce the number of parameters and inference latency.

[0124] Step 6.4.2: The Post-training Quantization strategy is used to convert the model from a 32-bit floating-point representation to an 8-bit fixed-point format, which greatly reduces memory usage and improves inference speed.

[0125] Step 6.4.3: Export the model as ONNX format for use in the PyTorch / ONNXRuntime environment.

[0126] Step 6.4.4: Tested on Snapdragon 865 devices, Jetson Nano and Raspberry Pi 4B platforms, the average inference time for crack image segmentation is <100ms and the model loading time is <1s, which is significantly better than traditional architectures such as U-Net or DeepLabV3+.

[0127] Example:

[0128] This embodiment uses a mobile terminal device to acquire images of the concrete structure surface. Combining a convolutional neural network (CNN) with a Transformer network based on self-attention mechanism, and through a specific preprocessing process, feature extraction structure, decoding and reconstruction mechanism, and mask output strategy, it achieves accurate segmentation and identification of concrete crack areas. It has high accuracy, high recall, and lightweight deployment capabilities, and is suitable for surface inspection tasks of various concrete structures such as bridges, tunnels, culverts, and retaining walls.

[0129] Step 1: Image Acquisition and Preprocessing: In implementing this invention, a high-definition CMOS industrial camera (1920×1080 resolution) is used in conjunction with supplementary lighting equipment to acquire real-world images of the concrete structure surface. The images may contain issues such as resolution inconsistencies, uneven lighting, and angular shifts due to environmental factors. Therefore, the system needs to preprocess the input images, including:

[0130] Step 1.1: Image Normalization. The input image is uniformly cropped and scaled to 512×512 pixels, and bilinear interpolation is used to handle size variations to ensure geometric consistency;

[0131] Step 1.2: Pixel normalization: Scale the pixel values ​​of the RGB image to the [0,1] range to facilitate numerical stability in subsequent convolution operations;

[0132] Step 1.3: Contrast Enhancement: Gamma correction and histogram equalization can be used to enhance the brightness of dark areas in the image and improve the visibility of fine cracks.

[0133] Step 1.4: Grayscale fusion strategy: Construct multi-channel input, including the original color image and its grayscale image, to enhance the model's channel expressiveness;

[0134] Step 1.5: Tensor Construction: Convert the processed image into a tensor of size C×H×W to facilitate input into the deep learning model for training and inference.

[0135] Step 2: Feature Extraction and Hybrid Encoding Structure. In the feature extraction stage, this invention employs an improved VGG19 network as the backbone encoder structure. This structure includes multiple convolutional and pooling layers to progressively extract texture, edge, and structural information from the image, specifically including:

[0136] Step 2.1: Freeze the first 3 groups of convolutional modules and train only the high-level convolutional weights;

[0137] Step 2.2: After each convolutional stage, batch normalization (BatchNorm) and the activation function (ReLU) are applied to ensure the non-linear expressive power of the features;

[0138] Step 2.3: Feature maps of different scales are compressed through 1×1 convolution to output a feature tensor with uniform dimension.

[0139] Simultaneously, a Transformer encoding module is connected after the mid-layer feature output, specifically including:

[0140] Step 2.4: Patch partitioning: Divide the mid-layer 2D feature map into several patch regions, and flatten each patch into a vector sequence;

[0141] Step 2.5: Positional Encoding: Add learnable positional codes to each patch to preserve spatial relationships;

[0142] Step 2.6: Multi-head self-attention mechanism: Construct semantic dependencies between any two positions using a multi-head attention mechanism;

[0143] Step 2.7: Feedforward network and normalization layer: Perform feature nonlinear mapping and residual connection to improve the stability and robustness of the representation.

[0144] Step 3: Decoding, Reconstruction, and Feature Fusion: The decoding module employs a four-level upsampling structure to gradually restore deep semantic features to the original image resolution. Specifically, this includes:

[0145] Step 3.1: Perform bilinear interpolation upsampling and deconvolution operations to expand the spatial size;

[0146] Step 3.2: Depthwise separable convolution extracts spatial features, reducing computational cost;

[0147] Step 3.3: Skip connection mechanism: In each decoding process, feature maps of the corresponding scale from the encoder are introduced, and multi-scale fusion is achieved through channel splicing;

[0148] Step 3.4: Apply Dropout and L2 regularization strategies after decoding each layer to prevent overfitting.

[0149] Step 4: Crack Mask Generation and Output Display:

[0150] Step 4.1: Channel Compression: After decoding, the model generates the final crack mask image through the following steps:

[0151] Step 4.2: Sigmoid activation: Map the output to a range of 0 to 1, representing the probability value of each pixel being a crack;

[0152] Step 4.3: Threshold binarization: Generate the mask image using a fixed threshold of 0.5 or the Otsu method;

[0153] Step 4.4: Result overlay: The mask image is merged with the original image to generate a highlighted crack overlay image for human-computer interaction and visualization.

[0154] Step 4.5: Output Management: Supports saving the resulting image and structured crack information (length, width, direction, coordinates, etc.) locally or uploading it to the cloud platform and edge gateway.

[0155] Step 5: Training Process and Optimization Strategy. The model training phase employs a joint loss function of Dice Loss and Focal Loss, with the specific design as follows:

[0156] Step 5.1: Dice Loss is used to optimize the overlap between the predicted region and the true mask, improving the IoU metric;

[0157] Step 5.2: Focal Loss is used to address the sparsity of crack category samples, reduce the influence of easily classified samples, and improve the model's ability to identify cracks with blurred edges.

[0158] Step 5.3: The loss combination is: Loss_total = λ1·Dice + λ2·Focal, λ1=0.5, λ2=1.0;

[0159] Step 5.4: The optimizer is AdamW, and the initial learning rate is 10. -4 The cosine annealing strategy is used for dynamic adjustment.

[0160] Step 5.5: Set up a warm-up mechanism during the training process, gradually increasing the learning rate in the first 5 rounds to avoid unstable convergence in the early stages;

[0161] Step 5.6: Use BatchSize=8 for training, with 120 training rounds. The validation set evaluation metric reaches its optimum in the 85th round.

[0162] Step 6: Model Deployment and Practical Application: After training, the model is exported in ONNX format and deployed as follows:

[0163] Step 6.1: Edge Deployment: Convert the model to TensorRT format and deploy it on a Jetson NX device. The average inference time is 78ms.

[0164] Step 6.2: Mobile Deployment: Convert to TensorFlow Lite format and deploy on the Android platform. The actual inference time is approximately 82ms.

[0165] Step 6.3: System Integration: Complete functions such as image acquisition, model inference, mask display, result uploading, and report generation in the mobile APP;

[0166] Step 6.4: Crack Report Output: Generates structured data including crack area, length, orientation angle, and location coordinates, which can be exported as PDF or JSON files.

[0167] Step 7: Testing Results and Performance Evaluation: The application test results in a real engineering project are as follows:

[0168] Step 7.1: Test dataset precision: IoU=89.3%, F1-score=91.2%, recall=92.6%;

[0169] Step 7.2: It performs exceptionally well compared to mainstream models such as U-Net and DeepLabV3+;

[0170] Step 7.3: Maintain high detection robustness even in complex environments such as low light, stains, and structural shadows;

[0171] Step 7.4: Engineering application feedback shows that the method provided by this invention has advantages such as rapid response, visible results, easy deployment, and scalable structure.

[0172] In summary, this embodiment clearly illustrates the specific implementation and practical application effect of the method of the present invention in concrete crack detection. Through image standardization preprocessing, lightweight CNN-Transformer hybrid feature extraction, fusion-based decoding reconstruction, and optimized mask output strategy, a high-performance, easily deployable, and scalable intelligent crack detection system is constructed, possessing significant innovation, practicality, and promotional value.

Claims

1. A crack detection image segmentation method based on an improved CNN-Transformer hybrid architecture, characterized in that... The method includes the following steps: Step 1: Image Input and Preprocessing: Step 1.1: Image Acquisition: Acquire color images of the concrete structure surface using a mobile terminal device; Step 1.2: Image scaling: Scale the acquired images to a uniform size using bilinear interpolation. Step 1.3: Color and Grayscale Normalization: Normalize the image pixels; Step 1.4: Tensor Construction: Convert the image into a 3D tensor; Step 2: Hybrid Feature Encoding: Step 2.1: Backbone Network Selection and Structure Optimization: An improved VGG19 was selected as the backbone feature extraction network. Specifically, the parameters of the first three convolutional layers of VGG19 were frozen to keep the low-level feature extraction capability unchanged, and only the high-level layers were fine-tuned. The fine-tuning layers mainly included Conv4_x and Conv5_x convolutional groups. A 1×1 convolutional compression layer was added after the output of the Conv5_x layer. A 1×1 convolutional layer was further introduced in the decoder part. BatchNorm and Dropout were introduced in the fine-tuning layers and the newly added layers. At the same time, the fully connected layers in the original VGG19 were removed. Step 2.2: Multi-scale feature extraction: The input image is processed through layer-by-layer convolution and pooling operations to extract feature maps at different scales; Step 2.3: After the mid-to-high level output, introduce a 1×1 convolution to perform channel compression and dimension unification processing; Step 2.4: Finally, output three sets of feature maps at different scales: low-level texture map, mid-level structure map, and high-level semantic map; Step 3: Semantic Modeling Step 3.1: Patch partitioning and flattening: Divide the output feature map of the encoder intermediate layer into image patches and flatten them into a 1D vector sequence in spatial order; each patch is a feature sequence. Step 3.2: Embedding layer mapping: Input each patch into a linear transformation layer to convert it into a feature embedding vector of uniform dimension; Step 3.3: Position Encoding: To preserve spatial position information, absolute position encoding is added to incorporate spatial order into the feature sequence; Step 3.4 Multi-head self-attention calculation: Input the sequence into the Transformer backbone; Step 3.5: Output Reconstruction: The output sequence is rearranged into a 2D feature map, which is then concatenated or weighted and fused with the original CNN feature map to construct an representation tensor that combines global semantics and local details. Step 4: Cascaded Decoding: Step 4.1: Decoding: The decoder consists of 4 upsampling units, each responsible for doubling the size of the feature map and finally restoring it to the same size as the original image; Step 4.2: Upsampling: Spatial resolution is increased by using bilinear interpolation upsampling or deconvolution; Step 4.3: Depthwise Separable Convolution: After each level of scaling, Depthwise Separable Convolution is introduced: first, intra-channel convolution is performed, and then 1×1 channel fusion is performed; Step 4.4: Skip Connection Fusion: Introduce encoder feature maps of the same scale as the decoding layer, and combine them with the decoding feature maps through channel splicing or weighted fusion to achieve cross-layer information fusion and edge detail restoration; Step 4.5: Regularization: Apply Dropout or regularization restrictions after decoding each layer; Step 5: Crack Mask Output: Step 5.1: Channel Mapping and Dimension Compression: Compress the channels of the final feature map using 1×1 convolution to output a single-channel or dual-channel prediction map; Step 5.2: Activation function normalization: Output the crack probability map in the 0~1 interval through the Sigmoid function, or use Softmax to perform pixel-level multi-class distribution fitting; Step 5.3: Binarization threshold setting: Step 5.3.1: In mobile or embedded device deployments, when the Sigmoid output value is greater than the threshold, it is determined to be a crack pixel; Step 5.3.2: In the evaluation or fine annotation stage, determine the optimal segmentation threshold to adapt to different image qualities and crack morphologies. The final mask image is a 0 / 1 binary image. Step 5.4: Visualization and Overlay: Overlay the masking results onto the original image, highlighting the crack areas with red, green, or transparent masks to achieve a comparative display; Step 5.5: Output Management: Step 5.5.1: Upload the results to the edge gateway or monitoring terminal via Socket or HTTP protocol; Step 5.5.2: Upload the crack detection results to the cloud platform by calling the RESTful API or MQTT protocol; Step 5.5.3: If the model is deployed on an Android APP, the recognition results are sent back to the APP interface for image preview, result saving, and user tagging feedback to achieve a closed loop; Step 6: Loss Function Design and Training Optimization: Step 6.1: Joint Loss Function Design: A joint loss function mechanism combining Dice Loss and Focal Loss is adopted; Step 6.2: Optimizer Setup and Learning Rate Scheduling: Step 6.2.1: Use the AdamW optimizer; Step 6.2.2: Set the base learning rate to 10. -4 ; Step 6.2.3: Use the cosine annealing learning rate decay mechanism to make the learning rate gradually and smoothly decay during the training process; Step 6.2.4: In the first 5 to 10 epochs, use a linear warm-up approach to slowly increase the learning rate; Step 6.3: Regularization Strategies and Data Augmentation: Step 6.3.1: Introduce Dropout in the intermediate feature layer; Step 6.3.2: Introduce L2 penalty to the model weights; Step 6.3.3: Enhance the diversity of training data through online or offline data augmentation operations; Step 6.3.4: Oversample the cracked samples in the training dataset, or undersample the samples without cracks; Step 6.4: Model Compression and Multi-Terminal Deployment: Step 6.4.1: After training, perform structural pruning on some redundant channels or layers in the model, and optimize the lightweight student model using knowledge distillation techniques; Step 6.4.2: Use the Post-training Quantization strategy to convert the model from a 32-bit floating-point representation to an 8-bit fixed-point format; Step 6.4.3: Export the model as ONNX format for use in the PyTorch / ONNXRuntime environment.

2. The crack detection image segmentation method based on the improved CNN-Transformer hybrid architecture according to claim 1, characterized in that... In step 21, the learning rate set for Conv4_x and Conv5_x is 1 / 10 of the original learning rate.

3. The crack detection image segmentation method based on the improved CNN-Transformer hybrid architecture according to claim 1, characterized in that... In step 21, the probability of Dropout is 0.3 to 0.

5.

4. The crack detection image segmentation method based on the improved CNN-Transformer hybrid architecture according to claim 1, characterized in that... In step 5.3.1, the threshold is 0.

5.

5. The crack detection image segmentation method based on the improved CNN-Transformer hybrid architecture according to claim 1, characterized in that... In step 5.3.2, the optimal segmentation threshold is automatically determined using the Otsu algorithm, the maximum inter-class variance method, and a custom heuristic strategy.

6. The crack detection image segmentation method based on the improved CNN-Transformer hybrid architecture according to claim 1, characterized in that... In step 6.1, the formula for calculating Dice Loss is: Where TP is the number of pixels correctly predicted as cracks, FP is the number of pixels that are misclassified as cracks and background, and FN is the number of pixels that are misclassified as background. The formula for calculating Focal Loss is as follows: Where p is the prediction probability, γ is the focusing factor, and α is the inter-class balance factor; The two are weighted and combined into a total loss function: Where λ1 = 0.5, λ2 = 1.

0.

7. The crack detection image segmentation method based on the improved CNN-Transformer hybrid architecture according to claim 1, characterized in that... In step 6.3.3, the enhancement operations include: random horizontal / vertical flipping; random cropping and scaling; color perturbation; Gaussian blur and salt-and-pepper noise to simulate a realistic scene; and affine transformation and rotation to enhance crack orientation adaptability.

Images