Crack segmentation model efficient learning method based on adaptive knowledge integration

Abstract

According to the crack segmentation model efficient learning method based on adaptive knowledge integration provided by the invention, a small number of parameters are embedded into a Transform model, and only parameters of non-pre-training weights are finely adjusted in a new scene, so that two tasks of continuous learning and cross-domain transfer learning can be realized. According to the method, new knowledge is introduced through Adapter, and crack detection in a subsequent scene can be enhanced under the condition that excellent performance of an initial scene is reserved. According to the method provided by the invention, only a small number of training parameters are used, and the performance of a traditional full-amount fine tuning method can be achieved and even surpassed in a subsequent scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of crack detection, and in particular is an efficient learning method for a crack segmentation model based on adaptive knowledge integration. Background Art

[0002] Crack detection is a crucial component of structural health monitoring. Image-based crack detection methods offer a non-destructive, cost-effective, and efficient alternative. Pixel-level identification of cracks in an image allows for the measurement of crack parameters. Deep learning-based crack segmentation methods, among others, do not require pre-definition of crack features or image preprocessing. They automatically learn features from a large number of samples, demonstrating superior accuracy and generalization. Crack detection involves pixel-level identification—determining whether each pixel in an image is a crack. Pixel-level crack identification involves capturing an image containing a crack and, after passing it through a deep learning network, outputting a binary image. This binary image is then used to measure key parameters such as the crack's length, width, and area.

[0003] Despite the numerous advantages of deep learning-based crack detection methods, some challenges remain. Their generalization capabilities remain limited. When encountering new, unseen crack features, the model may misdetect or miss detections. Therefore, learning on new data is necessary to enhance the model's effectiveness in subsequent scenarios. However, during continuous learning, when the model is retrained on new data, it is prone to losing or forgetting previously acquired features, patterns, or knowledge, thereby disrupting effective knowledge retention and leading to performance degradation in the initial scenario and catastrophic forgetting. Furthermore, as the model size and training data increase, retraining the model consumes significant computing resources and time, especially on devices with limited capacity, such as edge devices. Specifically, for edge devices, storing multiple models for different scenarios is impractical. Furthermore, acquiring large amounts of high-quality crack image data requires significant human and material resources, and data distribution may vary across different domains. Existing traditional transfer learning methods to address this issue require fine-tuning a large number of parameters across the board and merging datasets from different scenarios for training, which can degrade the model's performance in the original scenario. Summary of the Invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention provides an efficient learning method for a crack segmentation model based on adaptive knowledge integration, which can achieve continuous learning, including the following steps: Step 1: Load the pre-trained weights of the Transformer model, train it in the initial scenario, and fine-tune all the parameters in the Transformer model; Step 2: Embed the Adapters module in the Transformer model; Step 3: Load the training weights obtained in the initial scenario and freeze them. When training in subsequent scenarios, fine-tune the parameters of the Adapters module; Step 4: Use the trained model to detect cracks in subsequent scenarios, and disable the parameters of the embedded Adapters module to detect cracks in the initial scenario.

[0005] Furthermore, step 3 also includes replacing the training weights obtained in the initial scenario with the Transformer weights obtained in step 3, which can achieve continuous learning while fully retaining prior knowledge.

[0006] The present invention also includes an efficient learning method for a crack segmentation model based on adaptive knowledge integration, which can achieve cross-domain learning and includes the following steps: Step 1: Embed the Adapters module into the Transformer model.

[0007] Step 2: Load the weights trained on a large public dataset and fine-tune only the parameters of the non-pretrained weights, including the Transformer module parameters and the embedded Adapters module parameters that do not match the pretrained weights.

[0008] Furthermore, the embedded Adapters module includes a fully connected layer projected to 1 / 4 dimension, a GELU activation function, and a fully connected layer restored to the original dimension.

[0009] Furthermore, the Adapters module is embedded in the Transformer using a residual connection, with the same position as the original residual connection.

[0010] Furthermore, during training, the image data is augmented by randomly scaling it to 0.8-1.2 times of 512, and randomly flipping and cropping it horizontally, so that the image is input into the network for training with a size of 512×512.

[0011] Furthermore, the model was trained for 200 iterations using the combined cross entropy and Dice loss function and the AdamW optimizer, with momentum and weight decay set to 0.9 and 0.01, respectively.

[0012] Furthermore, during training, the learning rate update strategies of warmup and poly are adopted.

[0013] Furthermore, we fine-tuned the Transformer model parameters using an initial learning rate of 0.00006, which achieved the best performance on the MasonryCrack test set. We also fine-tuned the Adapters module parameters using a learning rate of 0.003, which is sufficient to retain most prior knowledge.

[0014] Furthermore, when the training loss shows NaN (Not a number) and training fails, reduce the learning rate. When the training device memory is insufficient, reduce the number of adapters embedded in each Transformer, or only embed the top adapter for fine-tuning.

[0015] The present invention proposes an adaptive knowledge integration learning method for the crack segmentation Transformer model. The adapter is embedded in the Transformer model, and only the parameters that lack pre-trained weights are fine-tuned in subsequent scenarios. This method can achieve two tasks: continuous learning and cross-domain transfer learning. Cross-domain transfer learning refers to the transfer of knowledge learned in the source domain to the target domain to improve the learning efficiency or effect in the target domain. That is, by introducing new knowledge through the adapter, the crack detection effect in subsequent scenarios can be enhanced while retaining the excellent performance of the initial scenario. The proposed method can achieve or even exceed the performance of traditional full-scale fine-tuning methods in subsequent scenarios with only a small number of training parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a framework diagram of the invention of adaptive knowledge integration learning method; Figure 2 It is the embedded diagram of the Adapter in the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] The efficient learning method of the crack segmentation model based on adaptive knowledge integration in this embodiment, wherein the continuous learning method includes the following steps: Step 1: Load the pre-trained weights of the Transformer model, train it in the initial scenario, and fine-tune all the parameters in the Transformer model; Step 2: Embed the Adapters module in the Transformer model; Step 3: Load the training weights obtained in the initial scenario and freeze them. When training in subsequent scenarios, fine-tune the Adapters module; Step 4: Use the trained model to detect cracks in subsequent scenarios, and disable the parameters of the embedded Adapters module to detect cracks in the initial scenario.

[0020] Select a high-performance crack semantic segmentation model, such as SegFormer-B0, which has pre-trained weights on large-scale datasets such as ImageNet-K and ADE20K. Randomly sample the crack dataset corresponding to the initial scene and split it into training, validation, and test sets in an 8:1:1 ratio.

[0021] During model training, online data augmentation was used to enhance the robustness and generalization performance of the trained model. Image data augmentation consisted of random scaling to 0.8–1.2 times 512, followed by random horizontal flipping and cropping, resulting in images fed into the network at a size of 512×512. The model was trained for 200 iterations using a combined cross-entropy and Dice loss function and the AdamW optimizer, with momentum and weight decay set to 0.9 and 0.01, respectively. An initial learning rate of 0.00006 was used, and to ensure numerical stability in the early stages of training and convergence later in the training, warmup and poly learning rate update strategies were employed, respectively. The model training weights were obtained by selecting the model weight with the highest Dice coefficient on the validation set. The generalization performance of the model was verified on the test set.

[0022] Among them, Figure 1 As shown in (a), the continuous learning method that fully preserves prior knowledge includes the following steps: Step 1. Load the Transformer model (with parameters w Function: , the traditional fine-tuning method adjusts the parameters w Pre-trained weights, fully fine-tuned in the initial scenario (such as CrackFusion), to obtain Transformer weight version 1 (i.e. Transformer weight v1), and get .

[0023] Full fine-tuning involves training and adjusting the parameters of all layers in the Transformer model, including both pre-trained and non-pre-trained weights. Specifically, full fine-tuning adjusts the following: the weights of all encoder layers, including convolutional layers, attention mechanisms, and feedforward networks. This may also affect decoder layers and the model's output layer. It also adjusts any newly added layers or parameters, such as the weights of adapters introduced into the model.

[0024] Since the task of continuous learning is the same, the pre-trained Transformer weights remain consistent in size and can therefore be loaded entirely during training.

[0025] Step 2: Embed a small number of parameters in the Transformer model, that is, embed the Adapter module.

[0026] like Figure 2 As shown in (a), the Adapter is designed as a bottleneck structure. The Adapter is embedded in the Transformer model, including two fully connected layers and an activation layer using the GELU activation function. The initial fully connected layer projects the input to a lower dimension (1 / 4 in this paper), and the subsequent layers restore it to the original dimension. The adaptive Transformer module obtained by embedding the Adapter module is shown in Figure 2 (c) shows the results for different Transformer models.

[0027] The embedded adapter is embedded into the Transformer block using a residual connection, in the same position as the original residual connection. Testing has shown that this embedding method achieves better learning results than concatenation, hybrid connections, or other placements, and better preserves prior knowledge. Introducing two adapters into each Transformer module increases the number of parameters by approximately 0.2M. In contrast, introducing one adapter into each Transformer module reduces the number of trainable parameters to approximately 0.1M.

[0028] The basic structure of the Transformer module consists of two parts: multi-head self-attention and multi-layer perceptron. Figure 2 (b) Although the multi-head self-attention and multi-layer perceptron modules may be changed, the typical structure of the Transformer module is as follows Figure 2 (b) Remain consistent.

[0029] Step 3: Load the training weights obtained in the initial scene and freeze them, keeping the Transformer weights from the initial scene In the subsequent scenarios, only the embedded parameters Adapter are fine-tuned, and the learning rate used is 0.003. Other settings are the same, and the fine-tuning parameters account for only a small part, and the obtained .

[0030] Although the Transformer parameters remain fixed during adapter weight fine-tuning, statistical issues in the BatchNormalization layer may cause minor weight adjustments. This can cause the Transformer weights to shift from version 1 (v1) to version 2 (v2) after adapter fine-tuning. Version 1 is obtained by fully fine-tuning the pre-trained weights in the initial scenario, i.e., the Transformer weights v1 obtained in step 1.

[0031] Step 4: Two different methods are proposed based on the results obtained. Method 1 converts the model obtained in step 3 into Detect cracks in subsequent scenes, while using Method 2 extracts the adapter weights and merges them with the Transformer weights version 1 to obtain , used when applying Identify cracks in subsequent scenes, while using .

[0032] Subsequent scenarios are relative to the initial scenario and refer to scenarios where the model is applied in an environment different from the initial training data or task. For example, if the initial scenario is trained on a dataset of a specific type of crack (such as CrackFusion: a road crack dataset), the subsequent scenario might be applied to a different type of crack detection task (such as MasonryCrack: a crack dataset on masonry structures).

[0033] Both methods use a single weight during application and deactivate the Adapter when segmenting the cracks in the initial scene. The function that works during the deactivation process uses the method of Algorithm 2 to Transformed into the original .

[0034] Algorithm 1 shows the pseudo code for the PyTorch version of building the Adapter, while Algorithm 2 shows how to build an adaptive Transformer block with the first Adapter connected serially and the second connected residually.

[0035] The pseudo code of Algorithm 1 Adapter is as follows: class AdaptiveTransformerBlock(): def __init__(self, dim, num_heads, mlp_ratio, use_adapter=[True,True]): self.norm1 = LayerNorm (dim) self.norm2 = LayerNorm (dim) self.attn = MultiheadAttention(dim, num_heads) self.mlp = MLP(dim, mlp_ratio) self.adapter1 = Adapter(dim) if use_adapter[0] else nn.Identity() self.adapter2 = Adapter(dim) if use_adapter[1] else None def forward(self, x): x = x + self.adapter1(self.attn(self.norm1(x))) if self.adapter2 is not None: x = x + self.mlp(self.norm2(x)) + self.adapter2(x) else: x = x + self.mlp(self.norm2(x)) return x The pseudo - code of Algorithm 2, the Adaptive Transformer block, is as follows: class AdaptiveTransformerBlock(): def __init__(self, dim, num_heads, mlp_ratio, use_adapter=[True,True]): self.norm1 = LayerNorm (dim) self.norm2 = LayerNorm (dim) self.attn = MultiheadAttention(dim, num_heads) self.mlp = MLP(dim, mlp_ratio) self.adapter1 = Adapter(dim) if use_adapter[0] elsenn.Identity() self.adapter2 = Adapter(dim) if use_adapter[1] else None def forward(self, x): x = x + self.adapter1(self.attn(self.norm1(x))) if self.adapter2 is not None: x = x + self.mlp(self.norm2(x)) + self.adapter2(x) else: x = x + self.mlp(self.norm2(x)) return x In addition, if Figure 1 (b) The efficient cross-domain transfer learning method based on Adapter includes the following steps: Step 1: Embed the Adapters module into the Transformer model.

[0036] Step 2: Load the weights trained on a large public dataset. Follow the previous method to fine-tune only the parameters without pre-trained weights, including the original module and embedded parameters that do not match the pre-trained weights. The initial learning rate is 0.003. If the training loss has the problem of NaN (Not a number) and cannot be trained normally, the learning rate can be reduced.

[0037] When the training device has limited memory, it is recommended to prioritize reducing the number of adapters embedded in each Transformer block to further reduce the number of trainable parameters. Alternatively, you can embed an adapter only at the top for fine-tuning.

[0038] Since the number of adapter parameters is only related to the backbone network, this method is more advantageous when the number of decoder parameters is large.

[0039] To verify the applicability of different models and analyze their performance differences with varying model parameter counts, we tested multiple Transformer models. Given the Swin Transformer's excellent feature extraction capabilities for tasks like semantic segmentation and the U-Net's symmetrical structure for detecting small cracks, we used a minimal version of the Swin Transformer, Swin-T, as the backbone network. We designed a ConvHead decoder based on the U-Net. Starting with the SegFormer N1 model, we sequentially replaced the decoder and encoder to create models N2-N4. The specific settings are shown in Table 1: Model encoder Decoder Model parameter quantity / M Adapter parameter quantity / M Proportion / % N1 MiT All-MLP 3.7 0.2 5.4 N2 MiT ConvHead 4.6 0.2 4.3 N3 Swin-T All-MLP 28.2 2.2 7.8 N4 Swin-T ConvHead 35.9 2.2 6.1 Two Adapters are introduced into each Transformer block.

[0040] To comprehensively evaluate the effectiveness of the adaptive knowledge integration method in various scenarios and enhance its applicability, training was conducted on four different datasets. Models N1-N4, which only fine-tune the adapter, achieved comparable or even superior performance to the fully fine-tuned method in subsequent scenarios, while better preserving prior knowledge. On the MasonryCrack test set, they demonstrated performance comparable to the fully fine-tuned method, ranging from 98.9% achieved by N2 to a peak of 99.9% achieved by N4. The proposed method significantly maintained performance on the CrackFusion test set, achieving at least 99.3% on N3. On the DeepCrack test set, the traditional fully fine-tuned method achieved a performance of 94.6%, while N1-N4 achieved performances of 94.6%, 94.0%, 92.9%, and 92.0%, respectively. On the NoiseCrack test set, the traditional fully fine-tuned method achieved a performance of 94.1%, while N1-N4 achieved performances of 94.1%, 93.5%, 92.2%, and 91.4%, respectively. The results show that N1-N4 can achieve more than 90% of the performance of the traditional full-scale fine-tuning method on the test set. Among them, DeepCrack, NoiseCrack, CrackFusion, and MasonryCrack represent the following datasets respectively: DeepCrack: This dataset includes images capturing cracks in buildings and roads, with different texture sequences.

[0041] NoiseCrack: This dataset contains images of road cracks showing various noise elements such as stains, road markings, shadows, manhole covers, and cars.

[0042] CrackFusion: A unified dataset formed by merging the DeepCrack and NoiseCrack datasets in a certain ratio. This fusion provides a more diverse set of crack images, simulating real-world crack conditions and can be used to test the continuous learning capabilities of the method.

[0043] MasonryCrack: This dataset includes images of masonry cracks, showing different backgrounds and different field of view sizes.

[0044] The feature extraction mechanism in continuous learning is similar, but the direct application of cross-domain pre-trained weights still requires verification. The model was fine-tuned on four datasets using the pre-trained weights from ADE20K. The results on the respective test sets are shown in Table 2. Compared to FFT-A, M1A23 achieved performance exceeding 98.1% in all scenarios, even surpassing the traditional method by 0.2% on the CFD test set, using only 5.4% of the training parameters from the full fine-tuning. This is shown in the table below: method T0 T3 T4 T5 The number of parameters trained / M FFT-A 58.8 77.7 52.7 79.2 3.7 M1A23 57.7 76.3 52.9 79.1 0.2 When using the proposed method, additional fine-tuning is required due to the mismatch of the final prediction layer weights when loading the pre-trained weights. T0 represents the test set of CrackFusion, while T1-5 correspond to the test sets of DeepCrack, NoiseCrack, MasonryCrack, CFD, and ConcreteCrack, respectively. FFT-A represents full fine-tuning using the ADE20K pre-trained weights. M1A23 represents the method using Figure 2 (c) Configuration of the adaptive transformer block.

[0045] Compared with the prior art, the present invention has the following characteristics and advantages: An adaptive knowledge integration method is proposed to achieve efficient learning for pixel-level crack recognition. Traditional full fine-tuning methods require tuning all parameters of the Transformer model, including the MSA and MLP layers (which primarily extract various crack morphological features, such as the overall crack shape, crack edge information, width variation, and contrast with the background). This results in high computational complexity and a large number of parameters. By introducing adapters (which primarily extract feature differences relevant to the crack segmentation task, such as crack morphology and surface texture variations in new scenarios, allowing the model to quickly adapt to new tasks without compromising the original capabilities of the pre-trained model), this paper fine-tunes only the parameters in the adapter modules and the parameters of non-pretrained layers for crack segmentation, reducing the number of trainable parameters by over 90%. Experiments show that this approach significantly reduces computational complexity while achieving crack segmentation results comparable to or even better than full fine-tuning, enabling the model to run efficiently on resource-constrained edge devices.

[0046] Two simple yet effective methods are proposed to leverage adapters to acquire new knowledge during continuous learning. One method fully preserves the knowledge from the initial crack segmentation scenario by freezing the pre-trained Transformer module parameters, while only fine-tuning the adapter parameters. This avoids catastrophic forgetting and maintains efficient crack identification in subsequent scenarios. The other method allows the adapter to adapt to subsequent scenarios (new crack morphology and distribution characteristics), acquiring new knowledge from subsequent scenarios and achieving better segmentation results in subsequent crack segmentation tasks. This method effectively addresses the problem of catastrophic forgetting, enables efficient continuous learning, retains approximately 98% of the initial scene knowledge, and continues to improve performance in subsequent scenarios.

[0047] This paper introduces an efficient cross-domain transfer learning method based on adapters. This method freezes pre-trained weights (e.g., encoder and decoder weights, parameters in the attention mechanism) from large public datasets across different domains (such as ImageNet-1K and ADE20K, which include images of animals, everyday objects, scene elements, and food). The adapters are then embedded as an adaptive conversion method for the feature extraction mechanism. Models using these pre-trained weights are fine-tuned on multiple test datasets and compared with traditional full-scale fine-tuning methods. In the field of crack detection, adapters effectively capture details related to crack features, such as crack morphology, texture, and boundaries, significantly improving crack segmentation performance without increasing model complexity and reducing the computational resources and time required for training.

[0048] The method proposed in this invention is applicable to various Transformer-based semantic segmentation models.

[0049] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An efficient learning method for crack segmentation model based on adaptive knowledge integration, characterized by: The following steps are involved: Step 1: Load the pre-trained weights of the Transformer model, train it in the initial scenario, and fine-tune all the parameters in the Transformer model; Step 2: Embed the Adapters module in the Transformer model; Step 3: Load the training weights obtained in the initial scenario and freeze them. When training in subsequent scenarios, fine-tune the parameters of the Adapters module. Step 4: Use the trained model to detect cracks in subsequent scenarios, and disable the parameters of the embedded Adapters module to detect cracks in the initial scenario.

2. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to claim 1 is characterized by: Step 3 also includes replacing the training weights obtained in the initial scenario with the Transformer weights obtained in step 3.

3. An efficient learning method for crack segmentation model based on adaptive knowledge integration, characterized by: The following steps are involved: Step 1: Embed the Adapters module into the Transformer model; Step 2: Load the weights trained on a large public dataset and fine-tune only the parameters of the non-pre-trained weights, including the Transformer module parameters and the embedded Adapters module parameters that do not match the pre-trained weights.

4. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to any one of claims 1 to 3, characterized in that: The embedded Adapters module includes a fully connected layer projected to 1 / 4 dimension, a GELU activation function, and a fully connected layer restored to the original dimension.

5. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to any one of claims 1 to 3, characterized in that: The Adapters module is embedded in the Transformer using a residual connection, which is the same as the original residual connection position.

6. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to any one of claims 1 to 3, characterized in that: During training, the image data is augmented by randomly scaling it to 0.8-1.2 times of 512, and randomly flipping and cropping it horizontally, so that the image is input into the network for training with a size of 512×512.

7. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to any one of claims 1 to 3, characterized in that: The model was trained for 200 iterations using the cross entropy and Dice joint loss function and the AdamW optimizer, with momentum and weight decay set to 0.9 and 0.01, respectively.

8. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to any one of claims 1 to 3, characterized in that: During training, the learning rate update strategies of warmup and poly are adopted.

9. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to claim 8, characterized in that: An initial learning rate of 0.00006 was used for fine-tuning the parameters in the Transformer model, and a learning rate of 0.003 was used for fine-tuning the parameters in the Adapters module.

10. The efficient learning method for crack segmentation model based on adaptive knowledge integration according to claim 8, characterized in that: When the training loss has the NaN problem and cannot be trained normally, reduce the learning rate; when the training device has insufficient memory, reduce the number of Adapters embedded in each Transformer, or only embed the Adapter at the top for fine-tuning.

Images