Pixel-level identification method of bridge surface cracks based on visual large model SAM

By modifying the Segment Anything Model, adding a crack feature low-rank adapter and pixel-level decoding module, the problem of the bridge's apparent crack pixel-level identification model has been solved, and high accuracy and low cost pixel-level identification effect is achieved. .

CN118379237BActive Publication Date: 2024-11-15HARBIN INST OF TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410289927.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-14
Publication Date
2024-11-15
Estimated Expiration
2044-03-14

AI Technical Summary

Technical Problem

Existing bridge apparent crack pixel-level identification models require a large amount of labeling data, and the generalization ability is poor, resulting in high labeling costs and low identification efficiency.

Method used

By modifying the Segment Anything Model (SAM), adding a crack feature low-rank adapter and a pixel-level identification and decoding module, and combining a combined loss function of DICE loss and cross-entropy loss for training, an end-to-end bridge apparent crack pixel-level identification model SAM -Crack.

Benefits of technology

It significantly reduces the need for labeling data, improves the accuracy of pixel-level identification of apparent cracks on the bridge, reduces data labeling and training costs, and improves identification efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118379237B_ABST
    Figure CN118379237B_ABST
Patent Text Reader

Abstract

The present invention proposes a method for pixel-level identification of apparent cracks in bridges based on a visual large model SAM, including: collecting images of apparent cracks in bridges and annotating them at pixel level, using random sampling to divide training, verification and test sets; modifying the visual large model SAM, deleting the part after image embedding, adding a crack feature low-rank adapter to the remaining part, forming a SAM-Crack encoder; adding a crack pixel-level identification decoding module after the SAM-Crack encoder to form a complete SAM-Crack model; using a training set to train the SAM-Crack model, the training loss function is a combination of DICE loss and cross entropy loss; using the trained SAM-Crack model to process the apparent crack image of the bridge to be identified, and generating a crack pixel-level identification result. The present invention greatly improves the crack identification accuracy under limited annotated data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent identification of apparent bridge diseases, and in particular to a method for pixel-level identification of apparent cracks in bridges based on a visual large model (SAM). Background Art

[0002] The service performance of bridges will inevitably degrade with the increase of service time and the coupling of complex loads, which poses a great threat to the safe operation of bridges and the smooth operation of transportation networks. The most direct reflection of the degradation of bridge service performance is the initiation and development of apparent diseases, and bridge apparent cracks are one of the most important types of apparent diseases. It is of great significance to identify bridge apparent cracks. Apparent crack identification based on maintenance specifications heavily relies on the practical experience of maintenance engineers, and the identification results are susceptible to subjectivity and low efficiency. To solve this problem, researchers have developed a bridge apparent disease identification method based on computer vision and deep learning, and completed the leap from "image-level accuracy" to "local image-level accuracy" to "pixel-level accuracy".

[0003] The technical core and essence of the current deep learning-based pixel-level identification method for bridge surface defects are still the target segmentation algorithm, mainly including those based on the convolutional neural network architecture and the Transformer architecture. These methods usually require a large amount of labeled training data, resulting in high labeling costs and poor model generalization. The emergence of Segment Anything Model (SAM) is a milestone in the field of target segmentation. Its powerful generalization ability greatly reduces the amount of labeled data required. However, the use of SAM requires the provision of corresponding prompts. Because it is trained on general target segmentation data, the performance of its global segmentation mode that does not require prompts on bridge surface crack images is unsatisfactory. Therefore, how to achieve end-to-end pixel-level accurate identification of bridge surface cracks while retaining SAM's powerful generalization ability, thereby greatly reducing the huge demand for labeled data in the current surface crack pixel-level identification model, is a key technical problem that needs to be solved urgently. Summary of the invention

[0004] Based on the above shortcomings, the present invention provides a pixel-level identification method for bridge apparent cracks based on a visual large model SAM, in order to solve the problems of high annotation cost and poor generalization ability required for the training of the existing bridge apparent crack pixel-level identification model.

[0005] The technical solution adopted by the present invention is as follows: a method for pixel-level identification of apparent cracks in bridges based on a visual large model SAM, the steps are as follows:

[0006] Step S1: take photos of the apparent cracks of the bridge to obtain an image set containing different lighting conditions, different shooting angles, different shooting distances, and different crack shapes, and use the image segmentation task annotation tool to manually or semi-automatically annotate the cracks in the image at the pixel level. The annotation is saved in a single-channel 8-bit binary image format, with the pixel where the crack is located being 1 and the pixel where the background is located being 0, to form an annotation set, and use a random sampling method to divide the image set and the corresponding annotation set into a training set, a validation set, and a test set;

[0007] Step S2: The image embedding in the visual large model SAM is used as the boundary to divide it into the front and back parts, and the part after the image embedding is deleted, including the decoder, prompt encoder and mask encoder of SAM;

[0008] Step S3: Modify the image embedding front and image embedding part in the visual large model SAM, add crack feature low-rank adapters before and after the Q mapping layer and before and after the V mapping layer in each Transformer block, and initialize the hyperparameter A in the crack feature low-rank adapter to a Gaussian normal distribution with a mean of 0, the hyperparameter B is initialized to 0, and the hyperparameter r is set to 4, forming a crack identification large model SAM-Crack encoder;

[0009] Step S4: adding a crack pixel-level identification decoding module after the crack identification large model SAM-Crack encoder as a crack identification large model SAM-Crack decoder, the decoder is a serial structure, from front to back is a deconvolution block - multi-scale convolution block - convolution block - interpolation block, the decoder and the encoder are connected in series to form a complete crack identification large model SAM-Crack;

[0010] Step S5: Use the training set obtained in step S1 to train the large crack identification model SAM-Crack. The input of the large crack identification model SAM-Crack is a three-channel RGB image or a single-channel grayscale image, and the output is a single-channel binary image. The large crack identification model SAM-Crack predicts that the pixels at the crack are 1, and the pixels at other positions are 0. The training loss function is a combination of DICE loss + cross entropy loss, and the weight of the two losses is 1:1. For the large crack identification model SAM-crack encoder, only the weight of the crack feature low-rank adapter is updated during training, and the weights of other network layer parameters are retained as the original SAM and are not updated. For the large crack identification model SAM-Crack decoder, the weights are initialized first and then updated.

[0011] Step S6: Input the bridge apparent crack image to be identified or the test set obtained in step S1 into the trained crack identification large model SAM-Crack. The output is the pixel-level identification result of the bridge apparent crack. The identification result is saved in a single-channel 8-bit binary image, where the pixel where the crack is located is 1 and the pixel where the background is located is 0.

[0012] Furthermore, the specific structure of the crack pixel-level identification and decoding module described in step S4 is as follows: the module first comprises two deconvolution layers connected in series, and the output size of the deconvolution layer is twice the input size; the deconvolution layer is followed by a multi-scale convolution layer, and the hyperparameter required to be set for this layer is the number of multi-scales, which is set to an integer of 4-8 according to the complexity of the image; the multi-scale convolution layer is followed by a 1×1 convolution layer, and the output channel is 1; and finally, there are two interpolation layers, and the output size of each interpolation layer is twice the input size, and the interpolation method uses the nearest neighbor or polynomial.

[0013] Furthermore, the training hyperparameters of the crack identification large model SAM-Crack described in step S5 are set as follows: during training, the initial learning rate is set to 0.000005-0.00002, and the learning rate variation strategy should adopt the cosine annealing method or the multi-step descent method.

[0014] Furthermore, the training process of the crack identification large model SAM-Crack described in step S5 is stopped when the training loss curve of SAM-Crack gradually stabilizes, that is, the change in the loss value between each two Epochs in 15 consecutive Epochs is no more than 10%, and the Dice scores on the validation set no longer increase for 15 consecutive Epochs.

[0015] On the other hand, the object of the present invention is implemented by a computer system, which includes a processor and a computer-readable storage medium communicatively connected to the processor, wherein computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed by the processor, the steps of the method described above are implemented.

[0016] The beneficial effects and advantages of the present invention are as follows: in view of the problem that the current bridge apparent crack pixel-level identification model requires a lot of labeled data and has poor generalization ability, the present invention realizes end-to-end bridge apparent crack pixel-level identification by modifying the SAM model, greatly improving the crack identification accuracy under limited labeled data. The present invention conveniently and accurately improves the accuracy of bridge apparent crack identification and significantly reduces data annotation costs and training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The framework diagram of the pixel-level identification model for bridge surface cracks proposed by the present invention;

[0018] Figure 2The test result diagram of the large-scale model SAM-Crack for pixel-level identification of bridge surface cracks proposed by the present invention; DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the embodiments and drawings, but the present invention is not limited thereto. Those skilled in the art may modify the present invention in form and detail, and these modifications are within the scope of protection of the present invention.

[0020] Example 1

[0021] like Figure 1 As shown in FIG. 1 , a pixel-level identification method for bridge surface cracks based on a visual large model SAM is shown in FIG. 1 . The steps are as follows:

[0022] Step S1: Use a mobile phone and a drone with a camera to take pictures of the apparent cracks of the bridge to obtain an image set containing different lighting conditions, different shooting angles, different shooting distances, and different crack shapes, and use the image segmentation task annotation tool labelme to manually annotate the cracks in the image at the pixel level. The annotation is saved in a single-channel 8-bit binary image format, with the pixel where the crack is located as 1 and the pixel where the background is located as 0 to form an annotation set, and use a random sampling method to divide the image set and the corresponding annotation set into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0023] Step S2: Taking the image embedding in the visual large model SAM as the boundary, it can be divided into two parts, and the part after the image embedding is deleted (excluding the image embedding), including the SAM decoder, prompt encoder and mask encoder;

[0024] Step S3: Modify the part before the image embedding in the visual large model SAM (including image embedding), add crack feature low-rank adapters before and after the Q mapping layer and before and after the V mapping layer in each Transformer block, and initialize the hyperparameter A in the crack feature low-rank adapter to a Gaussian normal distribution with a mean of 0, the hyperparameter B is initialized to 0, and the hyperparameter r is set to 4, forming a crack identification large model SAM-Crack encoder;

[0025] Step S4: Add a crack pixel-level identification decoding module after the crack identification large model SAM-Crack encoder. The module is a serial structure, which is a deconvolution block-multi-scale convolution block-convolution block-interpolation block from front to back. The module is used as a crack identification large model SAM-Crack decoder. The encoder and the decoder formed in step S3 are connected in series to form a complete crack identification large model SAM-Crack; wherein the specific structure of the crack pixel-level identification decoding module is as follows: the module first has two serially connected deconvolution layers, and the output size of the deconvolution layer is twice the input size; the deconvolution layer is followed by a multi-scale convolution layer, and the hyperparameter of the layer needs to be set is the number of multi-scales, which is set to an integer between 4 and 8 (including 4 and 8) according to the complexity of the image; the multi-scale convolution layer is followed by a 1×1 convolution layer, and the output channel is 1; finally, there are two interpolation layers, and the output size of each interpolation layer is twice the input size, and the interpolation method uses the nearest neighbor.

[0026] Step S5: Use the training set obtained in step S1 to train the crack identification model SAM-Crack. The training hardware uses Nvidia GTX 3090Ti. The input of the crack identification model SAM-Crack is a three-channel RGB image, and the output is a single-channel binary image (the model predicts that the pixels at the crack are 1, and the pixels at other positions are 0). The training loss function is a combination of DICE loss + cross entropy loss. The weight of the two losses is 1:1. The loss function is specifically: For the encoder of SAM-crack, only the weight of the crack feature low-rank adapter is updated during training, and the weights of other network layer parameters are retained as the original SAM and are not updated. For the crack identification large model SAM-Crack decoder, the weights are initialized first and then updated. Among them, the training hyperparameters of the crack identification large model SAM-Crack are set as follows: the initial learning rate should be set to 0.00001 during training, and the learning rate change strategy should adopt the cosine annealing method; the training process of the crack identification large model SAM-Crack stops when the training loss curve of SAM-Crack gradually stabilizes (the change in the loss value between each two Epochs in 15 consecutive Epochs is no more than 10%), and the Dice scores on the validation set for 15 consecutive Epochs no longer increase

[0027] Step S6: Input the test set into the trained crack identification model SAM-Crack, and the output is the pixel-level identification result of the bridge surface cracks. The identification result is saved in a single-channel 8-bit binary image, where the pixel where the crack is located is 1 and the pixel where the background is located is 0.

[0028] Example 2

[0029] In order to verify the effectiveness and accuracy of the crack identification large model SAM-Crack, this embodiment uses Deeplab v3p, one of the most advanced methods of convolutional neural network architecture in the bridge apparent crack pixel-level identification method, and Segformer, one of the most advanced methods of Transformer architecture, as comparison methods. Eight commonly used crack pixel-level identification datasets are used, namely CFD, Crack500, Cracktree200, DeepCrack, EugenMiller, GAPs, Rissbilder and Volker datasets (respectively recorded as datasets 1-8), which are divided into two working conditions. Working condition 1 is training with only 1% of the data in the training set, and working condition 2 is training with 100% of the data in the training set. Table 1 shows the Dice scores of crack identification of the three methods on the eight datasets under the two working conditions. It can be seen that no matter when the scale of labeled data used for training is very limited (condition 1) or not (condition 2), the present invention has achieved a significant improvement in crack identification accuracy compared with the other two methods. It is particularly noteworthy that when the present invention uses only 1% of the training data, the crack identification accuracy has greatly exceeded that of Deeplab v3+ using 100% of the training data, and is on par with the very powerful Segformer using 100% of the training data.

[0030] Table 1 Dice scores of crack identification of three methods on eight data sets under two working conditions (×100%)

[0031]

[0032] like Figure 2 The test result comparison diagram shown reflects the accuracy of the method of the present invention in identifying cracks. The first and third rows are original crack images, and the second and fourth rows are extracted crack areas. Therefore, the accuracy and effectiveness of the method of the present invention are verified.

Claims

1. A pixel-level identification method for bridge apparent cracks based on visual large model SAM, characterized in that: Here are the steps: Step S1: take photos of the apparent cracks of the bridge to obtain an image set containing different lighting conditions, different shooting angles, different shooting distances, and different crack shapes, and use the image segmentation task annotation tool to manually or semi-automatically annotate the cracks in the image at the pixel level. The annotation is saved in a single-channel 8-bit binary image format, with the pixel where the crack is located being 1 and the pixel where the background is located being 0, to form an annotation set, and use a random sampling method to divide the image set and the corresponding annotation set into a training set, a validation set, and a test set; Step S2: The image embedding in the visual large model SAM is used as the boundary to divide it into the front and back parts, and the part after the image embedding is deleted, including the decoder, prompt encoder and mask encoder of SAM; Step S3: Modify the image embedding front and image embedding part in the visual large model SAM, add crack feature low-rank adapters before and after the Q mapping layer and before and after the V mapping layer in each Transformer block, and initialize the hyperparameter A in the crack feature low-rank adapter to a Gaussian normal distribution with a mean of 0, the hyperparameter B to 0, and the hyperparameter r Set it to 4 to form the crack identification large model SAM-Crack encoder; Step S4: adding a crack pixel-level identification decoding module after the crack identification large model SAM-Crack encoder as a crack identification large model SAM-Crack decoder, the decoder is a serial structure, from front to back is a deconvolution block - multi-scale convolution block - convolution block - interpolation block, the decoder and the encoder are connected in series to form a complete crack identification large model SAM-Crack; Step S5: Use the training set obtained in step S1 to train the large crack identification model SAM-Crack. The input of the large crack identification model SAM-Crack is a three-channel RGB image or a single-channel grayscale image, and the output is a single-channel binary image. The large crack identification model SAM-Crack predicts that the pixels at the crack are 1, and the pixels at other positions are 0. The training loss function is a combination of DICE loss + cross entropy loss, and the weight of the two losses is 1:

1. For the large crack identification model SAM-crack encoder, only the weight of the crack feature low-rank adapter is updated during training, and the weights of other network layer parameters are retained as the original SAM and are not updated. For the large crack identification model SAM-Crack decoder, the weights are initialized first and then updated. Step S6: Input the bridge apparent crack image to be identified or the test set obtained in step S1 into the trained crack identification large model SAM-Crack. The output is the pixel-level identification result of the bridge apparent crack. The identification result is saved in a single-channel 8-bit binary image, where the pixel where the crack is located is 1 and the pixel where the background is located is 0.

2. According to claim 1, a method for pixel-level identification of bridge apparent cracks based on visual large model SAM is characterized in that: The specific structure of the crack pixel-level identification and decoding module described in step S4 is as follows: the module first comprises two deconvolution layers connected in series, and the output size of the deconvolution layer is twice the input size; the deconvolution layer is followed by a multi-scale convolution layer, and the hyperparameter required to be set for the multi-scale convolution layer is the number of multi-scales, which is set to an integer of 4-8 according to the complexity of the image; the multi-scale convolution layer is followed by a 1×1 convolution layer, and the output channel is 1; and finally, there are two interpolation layers, and the output size of each interpolation layer is twice the input size, and the interpolation method uses the nearest neighbor or polynomial.

3. According to claim 1, a method for pixel-level identification of bridge apparent cracks based on visual large model SAM is characterized in that: The training hyperparameters of the crack identification large model SAM-Crack described in step S5 are set as follows: during training, the initial learning rate is set to 0.000005-0.00002, and the learning rate variation strategy should adopt the cosine annealing method or the multi-step descent method.

4. According to claim 1, a method for pixel-level identification of bridge apparent cracks based on visual large model SAM is characterized in that: The training process stopping condition of the crack identification large model SAM-Crack described in step S5 is: when the training loss curve of SAM-Crack gradually stabilizes, that is, the change in the loss value between each two Epochs in 15 consecutive Epochs is no more than 10%, and the Dice score on the validation set no longer increases for 15 consecutive Epochs.

5. A computer system comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium stores computer instructions, characterized in that: When the computer instructions are executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Citation Information

Cited By

  • Bridge damage assessment method and system based on visual segmentation model, and medium

    CN122368795A

  • A bridge damage assessment method and system based on a visual segmentation model and a medium

    CN122368795B