A multi-modal crack detection method, medium and device

Abstract

The present invention discloses a multimodal crack detection method, medium, and device. The multimodal method includes intensity and range modes. The method comprises extracting intensity crack features and range crack features simultaneously, introducing two sets of decoupling adapter modules to fine-tune the SAM2 encoder; inputting the intensity crack features, range crack features, and their fusion features into the SAM2 decoder; employing a gradient conflict resolution method to mitigate gradient conflicts, thereby obtaining three updated gradients. Finally, all updated gradients are summed to generate a final gradient; and the updated gradients are used to update the parameters of the intensity and range encoders and decoder, ultimately generating a crack detection map.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a multimodal crack detection method, medium and device that utilizes a gradient conflict resolution method to balance the introduction of single-modal losses. Background Art

[0002] In recent years, the widespread application of deep learning techniques in computer vision, particularly convolutional neural networks (CNNs), has significantly improved the performance of image processing tasks. In crack detection, deep learning methods can achieve more accurate crack location by automatically extracting image features. However, single-modality image feature extraction methods have certain limitations. In particular, image information from different modalities often provides complementary information in crack detection. Therefore, combining multimodal information for crack detection is of great significance.

[0003] Traditional multimodal fusion methods rely primarily on simple feature concatenation or weighted fusion, ignoring the differences in features between different modalities. This can lead to inaccurate feature learning, especially when processing features with different activation levels, thus affecting model performance.

[0004] To address these issues, the deep learning-based Decoupling Adapter module has emerged in recent years. This technology automatically selects effective features from multimodal inputs and optimizes low- and high-activation features differently. By applying different optimization strategies to channels with high and low activation values, the Decoupling Adapter effectively reduces the risk of overfitting and improves the model's generalization capabilities.

[0005] However, in multimodal crack detection, introducing multiple single-modal loss functions can lead to gradient conflicts. This can disrupt the model's optimization process, especially when the gradient directions are inconsistent. Therefore, gradient conflicts have become a key factor affecting the performance of multimodal crack detection. Existing methods typically employ simple gradient accumulation or merging strategies, ignoring the interactions between different loss functions. This can lead to unstable training and even prevent the full learning of features from a single modality.

[0006] Therefore, resolving the gradient conflict problem and improving the stability and accuracy of multimodal crack detection methods has become a key challenge in current research. By introducing a gradient conflict resolution strategy, the problem of inconsistent gradient directions can be effectively alleviated, making the model more stable during training while improving the utilization efficiency of each modal feature, thereby enhancing the accuracy and robustness of crack detection. Summary of the Invention

[0007] The present invention proposes a multi-modal crack detection method that can solve at least one of the technical problems in the background technology.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A multimodal crack detection method comprises the following steps:

[0010] S1, extracts the intensity crack feature intensity and the distance crack feature range from the test image, and introduces two sets of decoupling adapter modules to fine-tune the SAM2 encoder;

[0011] S2, input the intensity crack feature intensity and the distance crack feature range and the fusion feature into the SAM2 decoder to obtain crack detection information;

[0012] S3, using the gradient conflict resolution method to deal with the conflict between the intensity crack feature and the range crack feature, and obtain a new gradient;

[0013] S4. Use the new gradient to update the parameters of the intensity encoder, range encoder, and decoder to determine the crack detection information and finally generate a crack detection map.

[0014] Furthermore, the method for extracting the intensity crack feature and the range crack feature in step S1 of the present invention includes:

[0015] The intensity crack feature and the range crack feature are obtained through the dual-mode encoder:

[0016] F R =E R (I R ,θ R )

[0017] F T =E T (I T ,θ T )

[0018] Where, F R is the intensity crack characteristic, E R is the intensity encoder, F T is the crack characteristic range, E T It is the range distance encoder;

[0019] First, freeze and share the SAM2 encoder and fine-tune the two sets of adapter parameters θ R and θ TIn order to adapt the SAM model to intensity-range crack detection, an adapter module is usually used in practice. The standard adapter adopts a bottleneck structure design:

[0020] Given input features It contains the lower projection layer parameters GeLU activation function σ(·) and up-projection layer parameters The bottleneck intermediate dimension The calculation formula of the bottleneck structure is:

[0021]

[0022] The highly activated area after GeLU activation highlights the contour features of the salient target, and the foreground adapter selects the top P through the TopK operation. for Ratio of highly activated channels to background adapter selection after P back Proportional low activation channels:

[0023]

[0024] Among them, TopK(input,k,largest) retains the largest k channels when largest=True, and retains the smallest k channels when False. for and P back Controls the proportion of foreground / background activations retained respectively.

[0025] Furthermore, the step S2 of the present invention of obtaining crack detection information includes:

[0026] The SAM2 decoder first performs deep fusion of the input intensity crack feature and range crack feature through a multi-layer convolutional neural network (CNN). Assuming that the input single-modal features are intensity crack feature I and range crack feature R, the deep fusion is expressed as:

[0027] F=CNN(I,R)

[0028] Where F is the fused feature;

[0029] The global context information is captured by a multi-layer convolutional neural network. The convolution operation in the neural network is expressed as:

[0030]

[0031] Among them F in is the input feature map, F out is the output feature map, K is the convolution kernel;

[0032] At the same time, the attention mechanism is used to automatically focus on the local area according to the significance of the crack and enhance the detailed information of the crack edge. Assuming that F is the fused feature, the attention weight A is expressed as:

[0033]

[0034] Where Q and K are the query and key matrices, d k is the dimension of the key vector, and the output of the attention mechanism is expressed as:

[0035] F att =A·V

[0036] Where V is the value matrix;

[0037] Through this joint processing of multimodal features, the decoder can further optimize the recognition of local areas based on global features, accurately locate the location and morphology of cracks, and achieve the following through weighted fusion:

[0038] F final =α·F global +(1-α)·F local

[0039] Among them, F global is the global feature, F local is a local feature, α is a weight coefficient;

[0040] In order to accurately locate the position and morphology of the crack, cross entropy loss or Dice loss is used. For example, cross entropy loss is expressed as:

[0041]

[0042] Among them, y i is the true label, is the predicted probability.

[0043] Furthermore, the method for using the gradient conflict resolution method in step S3 of the present invention to handle the conflict between two single modes includes:

[0044] According to the chain rule, the gradient of the two encoders is expressed as:

[0045]

[0046] Observe the gradient relationship, G in most areas R and G T If the ratio is greater than 1, update the formula according to the following parameters:

[0047]

[0048] Conclusion: The parameter update speed of intensity encoder is significantly faster than that of range encoder. The difference comes from G R The gradient value is continuously greater than G T The gradient value causes the intensity encoder to obtain a larger parameter update amplitude in each iteration;

[0049] Where η represents the learning rate and t represents the optimization step;

[0050] A shared decoder is used to decode the intensity feature and range feature to obtain a single-modal prediction P R and P T ;

[0051] P R =D(F R ,θ D ),P T =D(F T ,θ D )

[0052] Compute two unimodal losses:

[0053]

[0054] By introducing two auxiliary constraints of unimodal loss, the gradient of the intensity encoder tends to be equal to the gradient of the range encoder;

[0055] In encoder E R Part, comes from the single-modal loss L R Gradient and the fusion loss L F Gradient Jointly affect parameter update; in encoder E T Part, comes from the single-modal loss L T Gradient and the fusion loss L F Gradient Jointly affect parameter updates; in the decoder D part, the gradient from the intensity flow loss Gradients from range flow loss and the gradients from the fused stream Jointly affect parameter update.

[0056]

[0057] The cosine angles of different gradients are not always greater than 0, which indicates that there are some opposite gradient directions, which is harmful to model training;

[0058]

[0059] Among them, GradDeConflict is a gradient conflict resolution method using PCGrad strategy. For a given gradient G i ∈G, the current gradient after conflict resolution Initialized to G j ; Then, estimate the current gradient With another gradient G i ∈G(j≠i) cosine similarity;

[0060]

[0061] If the cosine similarity is less than 0, it means that the optimization directions of the two gradients conflict. Will subtract it from G j The projection onto the normal plane of , otherwise remains unchanged;

[0062]

[0063] Then, eliminate the potential conflicts with all other gradients;

[0064] This process is repeated between all gradients;

[0065] Finally, all updated gradients are added together to generate the final gradient G:

[0066]

[0067] After eliminating all gradient conflicts in the encoder and decoder, the parameters of the two encoders and decoders are updated as follows:

[0068]

[0069] Furthermore, the crack detection map generation method in step S4 of the present invention includes:

[0070] The two features F R and F T The fusion is input into the decoder to generate the prediction result P F , its calculation process can be expressed as:

[0071] P F =D(P R +F T ,θ D )

[0072] Where D represents the parameter θ D decoder;

[0073] Calculate the prediction result Pf The loss function between the real label GT:

[0074]

[0075] Where, and They represent the weighted intersection-over-union loss function IoU and the weighted binary cross entropy loss function BCE respectively.

[0076] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0077] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0078] From the above technical solution, it can be seen that the present invention proposes a significant crack detection method based on Intensity-Range multimodal fusion, which has higher detection accuracy and robustness than traditional methods. Through multimodal feature fusion, the intensity and range information are effectively combined to enhance the recognition ability of crack areas; the introduction of the decoupling adapter module enables the model to distinguish high and low activation features of different modalities, adaptively optimize feature expression, and improve detection performance; in addition, the present invention adopts a gradient conflict resolution strategy to alleviate the gradient conflict problem in the multimodal learning process and improve the training stability and generalization ability of the model. The present invention can effectively improve the accuracy, stability and adaptability of crack detection in complex scenarios, and is suitable for various types of engineering detection and intelligent detection systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a flow chart of a multi-modal crack detection method of the present invention;

[0080] Figure 2 This is a neural network structure diagram of the multimodal crack detection method of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0082] like Figure 1 As shown, the multi-modal crack detection method described in this embodiment performs the following steps by a computer device:

[0083] S1, extract the intensity crack feature and the distance crack feature range from the image to be tested;

[0084] S2, input the intensity crack feature intensity and the distance crack feature range and the fusion feature into the SAM2 decoder to obtain crack detection information;

[0085] S3, using the gradient conflict resolution method to deal with the conflict between the intensity crack feature and the range crack feature, and obtain a new gradient;

[0086] S4. Use the new gradient to update the parameters of the intensity and range distance encoder and decoder, and finally generate a crack detection map.

[0087] The following is a detailed description of each step:

[0088] S1, extracts the intensity crack feature intensity and the distance crack feature range from the test image, and introduces two sets of decoupling adapter modules to fine-tune the SAM2 encoder;

[0089] When extracting intensity crack features and range crack features, there is a problem of indistinguishable learning between high activation and low activation. Two sets of decoupling adapter modules are introduced to fine-tune the SAM2 encoder. The decoupling adapter modules decouple the learning processes of intensity mode and range mode, enabling the model to process features of different activation levels more effectively.

[0090] Methods for extracting intensity crack characteristics and range crack characteristics include:

[0091] Obtain feature representation through a bimodal encoder:

[0092] F R =E R (I R ,θ R )

[0093] F T =E T (I T ,θ T )

[0094] Where, F R is the intensity crack characteristic, E R is the intensity encoder, F T is the crack characteristic range, E T It is the range distance encoder;

[0095] Following common practice, we freeze and share the SAM2 encoder and fine-tune both sets of adapter parameters θ R (intensity branch) and θ T (range branch). To adapt the SAM model to intensity-range crack detection, an adapter module is usually used in practice. The standard adapter adopts a bottleneck structure design:

[0096] Given input features It contains the lower projection layer parameters GeLU activation function σ(·) and up-projection layer parameters The bottleneck intermediate dimension The calculation formula of this structure is:

[0097]

[0098] However, traditional adapters use the same gradient optimization for both high- and low-activation neurons. High-activation regions activated by GeLU highlight the contours of salient objects, while low-activation regions primarily correspond to task-irrelevant background information. Over-optimization of low-activation regions can easily introduce noise and increase the risk of overfitting. To address this, we propose decoupling the adapter into foreground and background adapters.

[0099] In specific implementation, the foreground adapter selects the top P through the TopK operation for Ratio of highly activated channels to background adapter selection after P back Proportional low activation channels:

[0100]

[0101] Among them, TopK(input,k,largest) retains the largest k channels when largest=True, and retains the smallest k channels when False. for and P back Controls the proportion of foreground / background activations retained respectively.

[0102] S2, input the intensity crack feature intensity and the distance crack feature range and the fusion feature into the SAM2 decoder to obtain crack detection information;

[0103] The SAM2 decoder first performs deep fusion on the input unimodal features (including intensity crack features and range crack features), usually achieved through a multi-layer convolutional neural network (CNN). Assuming that the input unimodal features are I (intensity crack features) and R (range crack features), deep fusion can be expressed as:

[0104] F=CNN(I,R)

[0105] Where F is the fused feature.

[0106] The global context information is captured by multi-layer convolutional neural networks. The convolution operation in the neural network can be expressed as:

[0107]

[0108] Among them F in is the input feature map, F out is the output feature map, and K is the convolution kernel.

[0109] At the same time, the attention mechanism is used to automatically focus on the local area according to the significance of the crack, and enhance the detailed information of the crack edge. Assuming F is the fused feature, the attention weight A can be expressed as:

[0110]

[0111] Where Q and K are the query and key matrices, d k is the dimension of the key vector. The output of the attention mechanism can be expressed as:

[0112] F att =A·V

[0113] Where V is the value matrix.

[0114] By combining this multimodal feature processing, the decoder can further optimize the recognition of local areas based on global features and accurately locate the location and shape of cracks. This can be achieved through weighted fusion:

[0115] F final =α·F global +(1-α)·F local

[0116] Among them, F global is the global feature, F local is a local feature, and α is a weight coefficient.

[0117] In order to accurately locate the location and morphology of the crack, cross entropy loss or Dice loss may be used. For example, cross entropy loss can be expressed as:

[0118]

[0119] Among them, y i is the true label, is the predicted probability.

[0120] S3, using the gradient conflict resolution method to deal with the conflict between the intensity crack feature and the range crack feature, and obtain a new gradient;

[0121] Since the introduction of two single-mode (intensity crack feature and range crack feature) losses leads to gradient conflicts, a gradient conflict resolution method is used to alleviate the gradient conflicts, thereby obtaining three updated gradients. Finally, all updated gradients are added to generate the final gradient;

[0122] According to the chain rule, the gradient of the two encoders can be expressed as:

[0123]

[0124] Observe the gradient relationship, G in most areas R and G T The ratio is greater than 1. Update the formula according to the following parameters:

[0125]

[0126] It can be concluded that the parameter update speed of the intensity encoder is significantly faster than that of the range encoder. This difference is due to the fact that G R The gradient value is continuously greater than G T The gradient value results in a larger parameter update amplitude for the intensity encoder in each iteration.

[0127] Here, η represents the learning rate and t represents the optimization step. The speed comparison of parameter updates shows that the intensity mode dominates the training of the entire model, while the range mode is not fully optimized. Therefore, the two modes are not well utilized, which reduces the performance.

[0128] To address this challenge, we introduce unimodal supervision to enhance the optimization of range modality. Specifically, a shared decoder is used to decode the intensity feature and the range feature to obtain the unimodal prediction P R and P T .

[0129] P R =D(F R ,θ D ),P T =D(F T ,θ D )

[0130] Therefore, two unimodal losses are calculated:

[0131]

[0132]

[0133] By introducing two auxiliary constraints of unimodal losses, the gradient of the intensity encoder tends to be equal to the gradient of the range encoder.

[0134] However, the introduction of two unimodal losses will lead to gradient conflicts. R Part, comes from the single-modal loss L R Gradient and the fusion loss L F Gradient Jointly affect parameter update; in encoder E T Part, comes from the single-modal loss L T Gradient and the fusion loss L F Gradient Jointly affect parameter updates; in the decoder D part, the gradient from the intensity flow loss Gradients from range flow loss and the gradients from the fused stream Jointly affect parameter update.

[0135]

[0136] The cosine angles of different gradients are not always greater than 0, indicating that there are some opposing gradient directions that are detrimental to model training. In this case, gradient conflict resolution is applied to mitigate these gradient conflicts, resulting in three updated gradients. This approach allows the model to avoid conflicts between gradient directions during training, improving optimization stability and efficiency.

[0137]

[0138] Among them, GradDeConflict is a gradient conflict resolution method using the PCGrad strategy. Specifically, for a given gradient G i ∈G, the current gradient after conflict resolution Initialized to G j Then, estimate the current gradient With another gradient G i ∈G(j≠i) is the cosine similarity between them.

[0139]

[0140] If the cosine similarity is less than 0, it means that there is a conflict in the optimization directions of the two gradients. Will subtract it from Gj The projection onto the normal plane of , otherwise remains unchanged.

[0141]

[0142] Then, eliminate the Potential conflicts with all other gradients.

[0143] This process is repeated for all gradients. Finally, all updated gradients are added together to generate the final gradient G:

[0144]

[0145] After eliminating all gradient conflicts in the encoder and decoder, the parameters of the two encoders and decoders are updated as follows:

[0146]

[0147] S4. Use the new gradient to update the parameters of the intensity encoder, range encoder, and decoder to determine the crack detection information and finally generate a crack detection map.

[0148] The two features F R and F T The fusion is input into the decoder to generate the prediction result P F The calculation process can be expressed as:

[0149] P F =D(P R +F T ,θ D )

[0150] Where D represents the parameter θ D decoder.

[0151] Furthermore, the prediction result P is calculated F The loss function between the real label GT:

[0152]

[0153] Where, and These represent the weighted Intersection-over-Union (IoU) loss and weighted Binary Cross-Entropy (BCE) loss, respectively. These two loss functions, through a weight adjustment mechanism, effectively improve the model's learning ability for difficult examples. The weighted IoU loss enhances boundary localization accuracy, while the weighted cross-entropy loss enhances the robustness of pixel-level classification.

[0154] The following is an explanation using specific examples:

[0155] like Figure 2 As shown, the multimodal crack detection method described in this embodiment is verified on the FIND dataset. For the FIND dataset, the same settings as the paper "A Dual-Stream End-to-end Pavement Crack Segmentation Network based on Transformer and Multi-modal Fusion" are adopted, including a training set of 2000 pairs of intensity-range images and a test set of 500 pairs of intensity-range images.

[0156] During training and testing, the input intensity-range images were resized to 512x512. The model was trained using the AdamW optimizer, an initial learning rate of 1e-3, a batch size of 4, SAM2 pre-trained parameters, and default PyTorch settings. An NVIDIA GTX 3090 GPU was used. The warmup and joint training phases each lasted 45 epochs.

[0157] The method is compared with seven RGBD detection methods, including Crack Transformer, SegFormer, Swin-UperNet, VGG19-FCN, UNet-FCN, DSEN-uno, and CrackFusionNet. The results are shown in Table 1.

[0158]

[0159] Table 1 Experimental results are as follows:

[0160] [1] W.Wang, Q.Lai, H.Fu, J.Shen, H.Ling, and R.Yang, "Salient object detection in the deep learning era: An in-depth survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44, no.6, pp.3239-3259, 2021.

[0161] [2]Z.Tu,Y.Ma,Z.Li,C.Li,J.Xu,and Y.Liu,“RGBT salient object detection:A large-scaledataset and benchmark,”IEEE Transactions on Multimedia,vol.25,pp.4163-4176,2022.

[0162] [3]A.Kirillov,E.Mintun,N.Ravi,H.Mao,C.Rolland,L.Gustafson,T.Xiao,S.Whitehead,A.C.Berg,W.-Y.Lo et al.,“Segment anything,”in Proceedings of theIEEE / CVF International Conference on ComputerVision,2023,pp.4015-4026.

[0163] [4]N.Ravi,V.Gabeur,Y.-T.Hu,R.Hu,C.Ryali,T.Ma,H.Khedr,R.Ra¨dle,C.Rolland,L.Gustafson et al.,“SAM 2:Segment anything in images and videos,”arXiv preprint arXiv:2408.00714,2024.

[0164] [5]S.Gao,P.Zhang,T.Yan,and H.Lu,“Multi-scale and detail-enhancedsegment anything model for salient object detection,”in Proceedings of the32nd ACM International Conference on Multimedia,2024,pp.9894-9903.

[0165] [6]P.Zhang,T.Yan,Y.Liu,and H.Lu,“Fantastic animals and where to findthem:Segment any marine animal with dual SAM,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2024,pp.2578-2587.

[0166] [7]S.Lian,Z.Zhang,H.Li,W.Li,L.T.Yang,S.Kwong,and R.Cong,“Diving intounderwater:Segment anything model guided underwater salientinstancesegmentation and a large-scale dataset,”in Forty-first InternationalConference on Machine Learning,2024,pp.29 545-29 559.

[0167] [8]K.Wang,D.Lin,C.Li,Z.Tu,and B.Luo,“Adapting segment anything modelto multi-modal salient object detection with semantic feature fusionguidance,”arXiv preprint arXiv:2408.15063,2024.

[0168] [9]Y.Liu,P.Wu,M.Wang,and J.Liu,“CPAL:Cross-prompting adapter withLoRAs for RGB+X semantic segmentation,”IEEE Transactions on Circuits andSystems for Video Technology,pp.1-14,2025.

[0169]

[10] S.Chen,C.Ge,Z.Tong,J.Wang,Y.Song,J.Wang,and P.Luo,“AdaptFormer:Adapting vision transformers for scalable visual recognition,”Advances inNeural Information Processing Systems,vol.35,pp.16 664-16 678,2022.

[0170]

[11] T.Chen,L.Zhu,C.Deng,R.Cao,Y.Wang,S.Zhang,Z.Li,L.Sun,Y.Zang,andP.Mao,“SAM-Adapter:Adapting segment anything inunderperformed scenes,”inProceedings of the IEEE / CVF International Conference on Computer Vision,2023,pp.3367-3375.

[0171]

[12] T.Chen,A.Lu,L.Zhu,C.Ding,C.Yu,D.Ji,Z.Li,L.Sun,P.Mao,and Y.Zang,“SAM2-Adapter:Evaluating&adapting segment anything 2in downstream tasks:Camouflage,shadow,medical image segmentation,and more,”arXiv preprint arXiv:2408.04579,2024.

[0172]

[13] X.Peng,Y.Wei,A.Deng,D.Wang,and D.Hu,“Balanced multimodal learningvia on-the-fly gradient modulation,”in Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition,2022,pp.8238-8247.

[0173]

[14] Y.Wei and D.Hu, "MMPareto: Boosting multimodal learning within nocent unimodal assistance," in Forty-first International Conference onMachine Learning, 2024, pp.52 559-52 572.

[0174]

[15] T.Li, Z.Wen, Y.Li, and TSLee, "Emergence of shape bias inconvolutional neural networks through activation sparsity," Advances in NeuralInformation Processing Systems, vol.36, pp.71 755-71 766, 2023.

[0175]

[16] Y.Pang,

[0176] As shown in Table 1, the proposed method achieves the best results in terms of IoU, F1, Precision, and Recall evaluation indicators, proving the effectiveness of the method.

[0177] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0178] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0179] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the multimodal crack detection methods in the above embodiments.

[0180] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above methods.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0182] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0183] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0184] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal crack detection method, characterized in that: The following steps are involved: S1, extract the intensity crack feature and the distance crack feature range from the image to be tested; S2, input the intensity crack feature intensity and the distance crack feature range and the fusion feature into the SAM2 decoder to obtain crack detection information; S3, using the gradient conflict resolution method to deal with the conflict between the intensity crack feature and the range crack feature, and obtain a new gradient; S4. Use the new gradient to update the parameters of the intensity encoder, range encoder, and decoder to determine the crack detection information and finally generate a crack detection map. The method for using the gradient conflict resolution method to handle the conflict between two single modes in step S3 includes: According to the chain rule, the gradient of the two encoders is expressed as: Observe the gradient relationship, most of the areas and If the ratio is greater than 1, update the formula according to the following parameters: Conclusion: The parameter update speed of the intensity encoder is significantly faster than that of the range encoder. The difference comes from Gradient value continuously greater than The gradient value causes the intensity encoder to obtain a larger parameter update amplitude in each iteration; Where η represents the learning rate and t represents the optimization step; Use a shared decoder to decode intensity features and range features to obtain unimodal predictions and ; Compute two unimodal losses: By introducing two auxiliary constraints of unimodal loss, the gradient of the intensity encoder tends to be equal to the gradient of the range encoder; In the encoder Part, from the unimodal loss Gradient With the fusion loss Gradient Jointly affect parameter update; in the encoder Part, from the unimodal loss Gradient With the fusion loss Gradient Jointly affect parameter updates; in the decoder D part, the gradient from the intensity flow loss , gradient from range flow loss and the gradients from the fused stream Jointly influence parameter updates; The cosine angles of different gradients are not always greater than 0, which indicates that there are some opposite gradient directions, which is harmful to model training; in, It is adopted The policy gradient conflict resolution method, for a given gradient , the current gradient after conflict resolution Initialized to ; Then, estimate the current gradient With another gradient The cosine similarity between If the cosine similarity is less than 0, it means that the optimization directions of the two gradients conflict. It will be subtracted The projection onto the normal plane of , otherwise remains unchanged; Then, eliminate the potential conflicts with all other gradients; This process is repeated between all gradients; Finally, all updated gradients are added together to generate the final gradient : After eliminating all gradient conflicts in the encoder and decoder, the parameters of the two encoders and decoders are updated as follows: 。 2. A multimodal crack detection method according to claim 1, characterized in that: The method for extracting the intensity crack feature and the range crack feature in step S1 includes: The intensity crack feature and the range crack feature are obtained through the dual-mode encoder: Where, F R is the intensity crack characteristic, E R is the intensity encoder, F T is the crack characteristic range, E T It is the range distance encoder; First freeze and share the SAM2 encoder, while fine-tuning both sets of adapter parameters and In order to adapt the SAM model to intensity-range crack detection, an adapter module is usually used in practice. The standard adapter adopts a bottleneck structure design: Given input features , which contains the lower projection layer parameters , GeLU activation function and the upper projection layer parameters , where the bottleneck intermediate dimension is , the calculation formula of the bottleneck structure is: The highly activated area after GeLU activation highlights the contour features of the salient target, and the foreground adapter selects the foreground through the TopK operation. Ratio of highly activated channels to background adapter selection Proportional low activation channels: + in When largest=True, the largest k channels are retained; when False, the smallest k channels are retained. and Controls the proportion of foreground / background activations retained respectively.

3. The multimodal crack detection method according to claim 1, characterized in that: Acquiring crack detection information in step S2 includes: The SAM2 decoder first performs deep fusion of the input intensity crack feature and range crack feature through a multi-layer convolutional neural network (CNN). Assuming that the input single-modal features are intensity crack feature I and range crack feature R, the deep fusion is expressed as: Where F is the fused feature; The global context information is captured by a multi-layer convolutional neural network. The convolution operation in the neural network is expressed as: in is the input feature map, is the output feature map, K is the convolution kernel; At the same time, the attention mechanism is used to automatically focus on the local area according to the significance of the crack and enhance the detailed information of the crack edge. Assuming that F is the fused feature, the attention weight A is expressed as: Where Q and K are the query and key matrices, is the dimension of the key vector, and the output of the attention mechanism is expressed as: Where V is the value matrix; Through the joint processing of this multimodal feature, the decoder further optimizes the recognition of local areas based on global features, accurately locates the position and morphology of the crack, and achieves the following through weighted fusion: in, is a global feature, is a local feature, is the weight coefficient; Use cross entropy loss or Dice loss to accurately locate the location and morphology of the crack, where the cross entropy loss is expressed as: Where, is the true label, is the predicted probability.

4. The multimodal crack detection method according to claim 1, characterized in that: The method for generating a crack detection map in step S4 includes: The two features and Fusion is performed and input into the decoder to generate prediction results , its calculation process can be expressed as: in, Indicates that the parameter is decoder; Calculate prediction results and the true label The loss function between: Where, and They represent the weighted intersection-over-union loss function IoU and the weighted binary cross entropy loss function BCE respectively.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 4.

6. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 4.

Images