Bridge apparent crack extraction method based on visual large model distillation

Through visual large-scale distillation technology, the problems of poor generalization ability and large parameters of the bridge apparent fracture extraction model are solved, and the rapid and accurate extraction of apparent fractures of the bridge is achieved, which improves the accuracy of the evaluation of apparent disease of the bridge and the simplified Rodability provides technical support for the mobile deployment of bridge apparent crack extraction models.

CN118365582BActive Publication Date: 2024-11-15HARBIN INST OF TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410289947.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-14
Publication Date
2024-11-15
Estimated Expiration
2044-03-14

AI Technical Summary

Technical Problem

The existing bridge apparent fracture extraction model has poor generalization capabilities, large parameters and slow running speed, resulting in low accuracy and low efficiency in crack extraction on the new data set.

Method used

Using a method based on visual big model distillation, a lightweight crack extraction encoder and decoder is established by randomly extracting the data set of SAM models, and a lightweight crack extraction encoder and decoder are carried out to perform model distillation, reducing the amount of parameters and improving the running speed. At the same time, DICE loss and cross-entropy loss are used. The combined function with image similarity loss is trained to ensure the accuracy of the extraction results.

Benefits of technology

While ensuring generalization capabilities, the parameter quantity and running time of the model are significantly reduced, the accurate and rapid extraction of apparent cracks of the bridge is achieved, and the accuracy and robustness of the apparent disease extraction of bridges is improved, making it apparent to the bridge. The mobile deployment of crack extraction models provides a technical basis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118365582B_ABST
    Figure CN118365582B_ABST
Patent Text Reader

Abstract

The present invention proposes a bridge apparent crack extraction method based on visual large model distillation, which belongs to the field of bridge apparent disease intelligent extraction, including: collecting a general target segmentation data set and a bridge crack segmentation data set to form a distillation data set and an apparent crack extraction data set; establishing a lightweight crack extraction encoder for distillation with a visual large model SAM; using the distillation data set for distillation, the distillation target is to minimize the difference in image embedding generated by the visual large model SAM and the lightweight crack extraction encoder; adding a crack extraction decoder after the lightweight crack extraction encoder to form a bridge apparent crack extraction model; using a combined loss function to train the bridge apparent crack extraction model; using the trained model to extract cracks from the bridge apparent crack image. The present invention significantly reduces the number of parameters and running time of the model, that is, it can accurately and quickly extract cracks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of intelligent extraction of bridge apparent defects, and in particular relates to a method for extracting bridge apparent cracks based on visual large model distillation. Background Art

[0002] With the increase of service time and increasingly complex vehicle loads, bridge structures will inevitably experience performance degradation, which greatly threatens the safe operation of bridges in my country and the smooth flow of major transportation arteries. The most direct manifestation of the degradation of bridge structure performance is the occurrence and development of apparent diseases, among which apparent cracks are the most important form of disease. Accurately extracting bridge apparent cracks is of great significance for evaluating the apparent state of bridges and then assessing the technical conditions of bridge components and the whole. The traditional bridge apparent crack extraction method based on bridge maintenance specifications requires maintenance engineers to depict and measure cracks based on practical experience, which is susceptible to subjectivity and low efficiency. Therefore, researchers at home and abroad began to use advanced computer vision and deep learning technologies to simulate the thinking process of maintenance engineers, taking bridge apparent crack images as input, and outputting crack extraction results by establishing an end-to-end deep learning model.

[0003] Currently, a large number of crack extraction models based on deep convolutional neural networks and Transformer structures have achieved good results on specific data sets, but the models generally have poor generalization ability (poor crack extraction accuracy in new data sets). The emergence of the visual large model Segment Anything Model (SAM) has revolutionized the field of general target segmentation and demonstrated strong generalization ability. However, SAM is developed for general targets. For crack extraction tasks, its parameters are extremely redundant, resulting in huge number of parameters and slow running speed when SAM is directly used for crack extraction. How to propose a method that can retain the powerful generalization ability of SAM, reduce the number of model parameters, and improve the running speed of the model in response to existing problems is a research frontier and technical problem in the field of bridge surface disease extraction, which needs to be solved urgently. Summary of the invention

[0004] Based on the above shortcomings, the present invention provides a bridge apparent crack extraction method based on visual large model distillation, which solves the problems of poor generalization ability, large number of parameters and slow running speed of the existing bridge apparent crack extraction model.

[0005] The technical solution adopted by the present invention is as follows: a bridge apparent crack extraction method based on visual large model distillation, the steps are as follows:

[0006] Step S1: randomly extract pictures from the SA-1B data set of the SAM model at a ratio of 1%-5% of the number of pictures to form a distillation data set applied by this method, collect open source bridge crack segmentation data sets on the Internet and unify the annotation form to form an apparent crack extraction data set applied by this method;

[0007] Step S2: Establish a lightweight crack extraction encoder for distillation with the visual large model SAM, including upsampling layer, patch embedding layer, mobile convolution layer and Transformer layer. The image embedding generated by the encoder is recorded as The image embedding generated by the visual large model SAM encoder is denoted as

[0008] Step S3: Use the distilled dataset in step S1 to perform model distillation. During distillation, only the image data in the distilled dataset is needed, and no labeled data is needed. The same image is input into the visual large model SAM and the lightweight crack extraction encoder to obtain image embeddings. and The goal of distillation is to make and The difference is small enough to be within the allowable error, and the weights in the lightweight crack extraction encoder are saved when the distillation is completed;

[0009] Step S4: adding a crack extraction decoder after the lightweight crack extraction encoder established in step S2, including a deconvolution layer, a multi-scale convolution layer, a common convolution layer and an upsampling layer, and connecting the lightweight crack extraction encoder and the crack extraction decoder in series to form a bridge appearance crack extraction model completed by visual large model distillation;

[0010] Step S5: Use the apparent crack extraction data set of step S1 to train the bridge apparent crack extraction model. The training loss function is a combination of DICE loss, cross entropy loss and image similarity loss. During training, the weights of the lightweight crack extraction encoder retain the weights when the distillation is completed in step S2 without updating. The weights of the lightweight crack extraction decoder are initialized with He before being updated.

[0011] Step S6: Input the bridge apparent crack image to be identified in the RGB three-channel 24-bit image format into the bridge apparent crack extraction model trained in step S5, so as to generate the corresponding apparent crack extraction result, which is stored in the form of a binary image, in which the pixel value at the crack corresponds to 255 and the pixel value at other positions is 0.

[0012] Furthermore, the specific structure of the lightweight crack extraction encoder described in step S2 is: from front to back, it is 2x upsampling layer-mobile convolution layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer.

[0013] Furthermore, the specific situation of the distillation target described in step S3 is: using the mean square error indicator to evaluate and The similarity of each image is and When the mean square error is less than 0.1, the distillation is considered complete.

[0014] Furthermore, the distillation objective function is:

[0015]

[0016] Where ω is the weight in the lightweight crack extraction encoder; N is the number of images in the distillation dataset.

[0017] Furthermore, the distillation process described in step S3 is specifically as follows: the number of images in the distillation data set required for model distillation is within 100,000. During model distillation, the visual large model SAM retains the original weights unchanged, while the weights of the lightweight crack extraction encoder are updated after being initialized with He.

[0018] Furthermore, the specific structure of the crack extraction decoder described in step S4 is: from front to back, it is 2x deconvolution layer-2x deconvolution layer-multi-scale convolution layer-1×1 convolution layer-1×1 convolution layer-2x interpolation upsampling layer, where the scale of the multi-scale convolution layer is set to 6.

[0019] Furthermore, the training loss function described in step S5 is: the weights of DICE loss, cross entropy loss and image similarity loss in the combined loss function are 1:1:0.5 respectively.

[0020] On the other hand, the object of the present invention is implemented by a computer system, which includes a processor and a computer-readable storage medium communicatively connected to the processor, wherein computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed by the processor, the steps of the method described above are implemented.

[0021] The beneficial effects of the present invention are as follows: the present method adopts a new idea of ​​model distillation, distilling the most space-consuming and time-consuming encoder in SAM into a lightweight crack extraction encoder, while ensuring the powerful segmentation and generalization capabilities of SAM, significantly reducing the number of model parameters and running time, that is, accurately and quickly extracting cracks. The present invention improves the accuracy and robustness of bridge apparent disease extraction, and provides a basic technical solution for the mobile deployment of bridge apparent crack extraction models based on large visual models. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The framework diagram of the algorithm proposed in the present invention;

[0023] Figure 2 Comparison diagram of bridge apparent crack extraction results of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below in conjunction with the embodiments and drawings, but the present invention is not limited thereto. Those skilled in the art may modify the present invention in form and detail, and these modifications are within the scope of protection of the present invention.

[0025] Example 1

[0026] like Figure 1 As shown in FIG. 1 , a bridge apparent crack extraction method based on visual large model distillation is shown in FIG. 1 . The steps are as follows:

[0027] Step S1: In this embodiment, pictures from the SA-1B data set in SAM are used, and a distillation data set applied in this embodiment is formed by random sampling at a ratio of 1%-5% of the number of pictures, and open source bridge crack segmentation data sets on the Internet are collected and the annotation forms are unified. In this embodiment, GAPs, Rissbilder and Volker data sets are used to form the apparent crack extraction data set applied in this embodiment;

[0028] Step S2: Establish a lightweight crack extraction encoder for distillation with the visual large model SAM, including upsampling layer, patch embedding layer, mobile convolution layer and Transformer layer. The image embedding generated by the encoder is recorded as The image embedding generated by the visual large model SAM encoder is denoted as Among them, the specific structure of the lightweight crack extraction encoder is: from front to back, it is 2x upsampling layer-mobile convolution layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer;

[0029] Step S3: Model distillation is performed using the distilled dataset in step S1. During distillation, only the image data in the distilled dataset is needed, and no labeled data is needed. The graphics card used in this embodiment is Nvidia GTX 3090Ti, and the video memory is 24GB. Therefore, 8 images are input into the visual large model SAM and the lightweight crack extraction encoder at the same time to obtain image embeddings. and The goal of distillation is to make and The difference is small enough to be within the allowable error. When the distillation is completed, the weights in the lightweight crack extraction encoder are saved. The specific distillation goal is: the mean square error (MSE) indicator is used to evaluate and The similarity of each image is and When the MSE is less than 0.1, the distillation is considered complete, and the distillation objective function is: Wherein, ω is the weight in the lightweight crack extraction encoder; N is the number of images in the distillation dataset; the specific distillation process is as follows: the number of images in the distillation dataset required for model distillation is within 100,000. During model distillation, the visual large model SAM retains the original weights unchanged, while the weights of the lightweight crack extraction encoder are updated after being initialized with He;

[0030] Step S4: adding a crack extraction decoder after the lightweight crack extraction encoder established in step S2, including a deconvolution layer, a multi-scale convolution layer, a normal convolution layer and an upsampling layer, and connecting the lightweight crack extraction encoder and the crack extraction decoder in series to form a bridge appearance crack extraction model completed by visual large model distillation; wherein the specific structure of the crack extraction decoder is: from front to back, 2x deconvolution layer-2x deconvolution layer-multi-scale convolution layer-1×1 convolution layer-1×1 convolution layer-2x interpolation upsampling layer, wherein the scale of the multi-scale convolution layer is set to 6;

[0031] Step S5: Use the apparent crack extraction data set of step S1 to train the bridge apparent crack extraction model. The training loss function is a combination of DICE loss, cross entropy loss and image similarity loss. During training, the weights of the lightweight crack extraction encoder retain the weights when the distillation is completed in step S2 without updating, and the weights of the crack extraction decoder are initialized with He before being updated; wherein, the weights of DICE loss, cross entropy loss and image similarity loss in the combined loss function are 1:1:0.5 respectively, and the loss function is:

[0032] Step S6: Input the bridge apparent crack image to be identified into the bridge apparent crack extraction model trained in step S5 in RGB three-channel 24-bit image format to generate the corresponding apparent crack extraction result, which is stored in the form of a binary image, in which the pixel value at the crack corresponds to 255 and the pixel value at other positions is 0.

[0033] Example 2

[0034] The bridge apparent crack extraction model based on visual large model distillation trained in Example 1 is used to test the test set of the data set, wherein Deeplab v3+ is one of the most advanced CNN models commonly used for crack extraction, Segformer is one of the most advanced Transformer models commonly used for crack extraction, and SAM is a visual large model for crack extraction. Table 1 is a comparison of the crack extraction accuracy (Dice score × 100%), parameter quantity, video memory occupancy, and running speed of the four methods. It can be seen that the accuracy of the present invention on the three data sets is greatly improved compared with Deeplab v3+ and Segformer, and the accuracy on the three data sets is not greatly reduced compared with SAM, while the parameter quantity and video memory occupancy are significantly reduced. The present invention is more than 10 times faster than SAM in running speed (the running hardware is Nvidia GTX 3090Ti).

[0035] Table 1 Comparison of crack extraction accuracy, parameter quantity, video memory usage and running speed of four methods

[0036]

[0037] like Figure 2 The test result comparison diagram shown reflects the accuracy of the method of the present invention in identifying cracks. The first row is the original crack image, and the second row is the extracted crack area, which verifies the accuracy of the method of the present invention.

Claims

1. A bridge apparent crack extraction method based on visual large model distillation, It is characterized in that The steps are as follows: Step S1: randomly extract pictures from the SA-1B data set of the SAM model at a ratio of 1%-5% of the number of pictures to form a distillation data set applied by this method, collect open source bridge crack segmentation data sets on the Internet and unify the annotation form to form an apparent crack extraction data set applied by this method; Step S2: Establish a lightweight crack extraction encoder for distillation with the visual large model SAM, including upsampling layer, patch embedding layer, mobile convolution layer and Transformer layer. The image embedding generated by the encoder is recorded as The image embedding generated by the visual large model SAM encoder is denoted as Step S3: Use the distilled dataset in step S1 to perform model distillation. During distillation, only the image data in the distilled dataset is needed, and no labeled data is needed. The same image is input into the visual large model SAM and the lightweight crack extraction encoder to obtain image embeddings. and The goal of distillation is to make and The difference is small enough to be within the allowable error, and the weights in the lightweight crack extraction encoder are saved when the distillation is completed; The specific goal of the distillation is to use the mean square error indicator to evaluate and The similarity of each image is and When the mean square error is less than 0.1, the distillation is considered to be completed; the objective function of the distillation is: Where ω is the weight in the lightweight crack extraction encoder; N is the number of images in the distillation dataset; The distillation process is specifically as follows: the number of images in the distillation data set required for model distillation is within 100,000. During model distillation, the visual large model SAM retains the original weights unchanged, while the weights of the lightweight crack extraction encoder are updated after being initialized with He; Step S4: adding a crack extraction decoder after the lightweight crack extraction encoder established in step S2, including a deconvolution layer, a multi-scale convolution layer, a common convolution layer and an upsampling layer, and connecting the lightweight crack extraction encoder and the crack extraction decoder in series to form a bridge appearance crack extraction model completed by visual large model distillation; Step S5: Use the apparent crack extraction data set of step S1 to train the bridge apparent crack extraction model. The training loss function is a combination of DICE loss, cross entropy loss and image similarity loss. During training, the weights of the lightweight crack extraction encoder retain the weights when the distillation is completed in step S2 without updating. The weights of the lightweight crack extraction decoder are initialized with He before being updated. Step S6: Input the bridge apparent crack image to be identified in the RGB three-channel 24-bit image format into the bridge apparent crack extraction model trained in step S5, so as to generate the corresponding apparent crack extraction result, which is stored in the form of a binary image, in which the pixel value at the crack corresponds to 255 and the pixel value at other positions is 0.

2. According to claim 1, a bridge apparent crack extraction method based on visual large model distillation, It is characterized in that The specific structure of the lightweight crack extraction encoder described in step S2 is: from front to back, it is 2x upsampling layer-mobile convolution layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer-Transformer layer-mobile convolution layer.

3. According to claim 1, a bridge apparent crack extraction method based on visual large model distillation, It is characterized in that The specific structure of the crack extraction decoder described in step S4 is: from front to back, it is 2x deconvolution layer-2x deconvolution layer-multi-scale convolution layer-1×1 convolution layer-1×1 convolution layer-2x interpolation upsampling layer, where the scale of the multi-scale convolution layer is set to 6.

4. According to claim 1, a bridge apparent crack extraction method based on visual large model distillation, It is characterized in that The training loss function described in step S5 is: the weights of DICE loss, cross entropy loss and image similarity loss in the combined loss function are 1:1:0.5 respectively.

5. A computer system comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium stores computer instructions, It is characterized in that When the computer instructions are executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Citation Information

Cited By

  • Bridge damage assessment method and system based on visual segmentation model, and medium

    CN122368795A

  • A bridge damage assessment method and system based on a visual segmentation model and a medium

    CN122368795B