Multi-modal fusion defect detection method and system

Abstract

The invention discloses a multi-modal fusion defect detection method and system, and belongs to the technical field of intelligent detection and machine vision, and the system comprises a visible light sensor, a thermal infrared sensor, a hyperspectral sensor, a multi-band light source trigger control system, a modal preprocessing and alignment module, a cross-modal feature fusion module, and a defect detection and output module. According to the invention, three sensors are used to construct a multi-modal visual perception system, and a multi-band light source triggers a control system to complete image acquisition; after multi-modal information is subjected to preprocessing and cross-modal alignment through the modal preprocessing and alignment module, multi-modal fusion is achieved through the cross-modal attention module and the multi-scale feature fusion pyramid structure, and finally defect recognition and output are conducted through the defect detection and output module. According to the method, the defect identification precision can be obviously improved, and the method has obvious advantages especially for low-contrast and early-stage hidden crack defects, and has good expandability and deployment suitability at the same time.

Description

Technical Field

[0001] This invention relates to the fields of intelligent inspection and machine vision technology, specifically to a multimodal fusion defect detection method and system. Background Technology

[0002] Currently, there are many defect detection systems, but most are based on single-modal visual signals, such as visible light imaging or thermal infrared imaging. Due to the diversity, concealment, and variability of material spectral responses of defects, single-modal imaging mechanisms are significantly limited in their application.

[0003] (i) Defects such as uneven coating thickness, local pinholes or microcracks in lithium battery electrodes make it difficult for conventional RGB images to distinguish between background noise and real anomalies.

[0004] (ii) Hidden cracks in photovoltaic panels have no obvious visual characteristics in the early stages and can only be seen in stress concentration or thermal images.

[0005] Therefore, there is an urgent need for a detection system that integrates multiple imaging mechanisms, multimodal physical characteristics, and deep learning semantic understanding capabilities. Based on this, the present invention designs a multimodal fusion defect detection method and system to solve the aforementioned problems. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a multimodal fusion defect detection method and system.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A multimodal fusion defect detection system includes:

[0009] Visible light sensor, used to acquire visible light images, is represented as Where H represents the image height and W represents the image width;

[0010] A thermal infrared sensor is used to acquire thermal infrared images, represented as...

[0011] A hyperspectral sensor, used to acquire hyperspectral images in the B-band, is represented as... Where B represents the number of bands;

[0012] A multi-band light source triggering control system, including a main control chip, is used to control three sensors to acquire images of the same field of view within a unit of time.

[0013] The modal preprocessing and alignment module is used to preprocess and align the original images of the three modalities across modalities.

[0014] A cross-modal feature fusion module, containing a cross-modal attention module and a multi-scale feature fusion pyramid structure, is used to achieve multimodal fusion.

[0015] The cross-modal attention module is based on a self-attention structure and introduces an intermodal interaction modeling unit to achieve the correlation weight allocation of the three modalities. The weight calculation formula is as follows:

[0016]

[0017] in, f represents a learnable matrix; i For the embedding features of mode i;

[0018] The fusion output features of the cross-modal attention module are:

[0019]

[0020] in, W is used to output features for multimodal fusion. v Value transformation matrix: Maps the input features to learnable parameters in the fusion space. j : The embedding features of modality j, generated by the feature extractor, serve as input to the attention mechanism;

[0021] A multi-scale feature fusion pyramid structure is used to extract contextual features from different receptive fields. l represents the l-th layer in the multi-scale feature fusion pyramid structure. Each layer contains a convolutional module and a channel attention module. The fusion path is formed by downsampling features from the upper layer and upsampling features from the lower layer. The fusion output of the multi-scale feature fusion pyramid structure is defined as:

[0022]

[0023] Where CA represents the channel attention structure, w l The weights of the fusion features of the l-th layer are optimized inversely using boundary regression loss;

[0024] The defect detection and output module is used for defect identification and output.

[0025] Furthermore, the control signals emitted by the main control chip are defined as follows:

[0026]

[0027] Where, d i Let τ be the delayed trigger time of the i-th modal camera. i Let t be the exposure duration of the i-th modal camera. k Unit of time.

[0028] Furthermore, by adjusting d i With τ i Adjust the synchronization strategy to minimize the intermodal mismatch error ∈, ∈=∑ i＜j ||x i -T ij (x j )||, where T ij Let x be the intermodal spatial mapping function. i and x j Let i and j represent the input features of mode i and mode j, respectively.

[0029] Furthermore, preprocessing includes: uniform resolution, grayscale distribution, and band response.

[0030] Furthermore, the cross-modal alignment step is as follows: Define a modal feature extractor. Each modality is projected onto a shared embedding space, and the extracted features are as follows: Among them, f vis For the extracted visible light features, f ir For the extracted infrared features, f hsi To extract hyperspectral features, d i H' represents the delayed trigger time of the i-th modal camera; W' represents the height of the processed feature map; C represents the width of the feature map after processing by the modal feature extractor (i.e., the number of columns in the spatial dimension); and C represents the number of channels in the feature map, i.e., the dimension of the feature vector at each spatial location.

[0031] Furthermore, the visible light sensor uses a visible light camera; the thermal infrared sensor uses a thermal infrared imaging module; and the hyperspectral sensor uses a hyperspectral camera.

[0032] Furthermore, the main control chip is an STM32H7.

[0033] To better achieve the objectives of this invention, this invention also provides a multimodal fusion defect detection method, comprising the following steps:

[0034] First, the multi-band light source triggering control system is centered on a main control chip. The main control chip outputs multiple PWM signals, each with an adjustable delay trigger time and exposure duration, controlling the exposure within a unit time t. k The visible light sensor, thermal infrared sensor, and hyperspectral sensor each acquire images under the same field of view;

[0035] Secondly, the original images of the three modalities are preprocessed and cross-modal aligned through the modal preprocessing and alignment module;

[0036] Next, the extracted features are input into the cross-modal attention module within the cross-modal feature fusion module. The cross-modal attention module is based on a self-attention structure and introduces an inter-modal interaction modeling unit to achieve the correlation weight allocation among the three modalities. The weight calculation formula is as follows:

[0037]

[0038] in, f represents a learnable matrix; i For the embedding features of mode i;

[0039] The fusion output features of the cross-modal attention module are:

[0040]

[0041] in, W is used to output features for multimodal fusion. v Value transformation matrix: Maps the input features to learnable parameters in the fusion space. j : The embedding features of modality j, generated by the feature extractor, serve as input to the attention mechanism;

[0042] Subsequently, the fused output features of the cross-modal attention module are input into a multi-scale feature fusion pyramid structure, which extracts contextual features from different receptive fields. l represents the l-th layer in the multi-scale feature fusion pyramid structure. Each layer uses convolutional modules and channel attention modules to enhance sensitivity to minute defects. The fusion output of the multi-scale feature fusion pyramid structure is defined as:

[0043]

[0044] Where CA represents the channel attention structure, w l The weights of the fusion features of the l-th layer are optimized inversely using boundary regression loss;

[0045] Next, during training, a combined loss function is used to optimize the detection network. The formula for the combined loss function is:

[0046]

[0047] in, For the total loss, Cross-entropy classification loss, For bounding box regression loss, This is a modal consistency regularization term;

[0048]

[0049] Among them, fi : Represents the embedding feature of the i-th modality. For example, in a cross-modal attention module, f i These can be feature vectors from visible light (VIS), thermal infrared (IR), or hyperspectral (HSI) modes. These features are extracted from the original image using a modal feature extractor. j : Represents the embedding feature of the j-th modality. (and f) i Similarly, f j It is also a feature vector extracted from another modality (such as IR or HSI). In cross-modal interactions, f j Used with f i Perform correlation calculations to achieve dynamic weight allocation between modalities.

[0050] Finally, the defect detection and output module is used to identify and output defects.

[0051] Furthermore, the control signals emitted by the main control chip are defined as follows:

[0052]

[0053] Where, d i Let τ be the delayed trigger time of the i-th modal camera. i Let d be the exposure duration of the i-th modal camera; by adjusting d i With τ i Adjust the synchronization strategy to minimize the intermodal mismatch error ∈, ∈=∑ i＜j ||x i -T ij (x j )||, where T ij Let x be the intermodal spatial mapping function. i and x j Let i and j represent the input features of mode i and mode j, respectively.

[0054] Furthermore, preprocessing includes: uniform resolution, grayscale distribution, and band response;

[0055] The cross-modal alignment steps are as follows: Define the modal feature extractor. Each modality is projected onto a shared embedding space, and the extracted features are as follows:

[0056] Among them, f vis For the extracted visible light features, f ir For the extracted infrared features, f hsi To extract hyperspectral features, d iH' represents the delayed trigger time of the i-th modal camera; W' represents the height of the processed feature map; C represents the width of the feature map after processing by the modal feature extractor (i.e., the number of columns in the spatial dimension); and C represents the number of channels in the feature map, i.e., the dimension of the feature vector at each spatial location.

[0057] Compared with the prior art, the beneficial effects of this invention are: (1) It proposes a multi-band light source triggering control system based on the main control chip, so as to realize that the three sensors can obtain the same field of view image under the condition of mutual non-interference within a unit time.

[0058] (2) A cross-modal attention mechanism based on self-attention structure and introducing inter-modal interaction modeling unit was constructed, which realizes dynamic adjustment of information value weight in different modalities, thereby improving the discriminative ability of fusion expression;

[0059] (3) By combining the pyramid structure with the channel attention mechanism and utilizing the multi-scale feature fusion, the system’s ability to respond to minor defects (such as cracks) is enhanced.

[0060] (4) The combined loss function introduces a modality consistency regularization term to construct a shared semantic space, which encourages the feature spaces learned by each modality to maintain structural consistency, thus preventing modality drift while ensuring expressive power.

[0061] (5) A scalable and deployable robust industrial defect detection system framework has been formed, which is suitable for complex manufacturing scenarios such as lithium battery electrodes and photovoltaic panels. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0063] Figure 1 This is a structural block diagram of the multimodal fusion defect detection system of the present invention;

[0064] Figure 2 This is a schematic diagram of the multi-band light source triggering control system of the present invention;

[0065] Figure 3 This is a schematic diagram of the cross-modal attention module of the present invention;

[0066] Figure 4 This is a schematic diagram of the multi-scale feature fusion pyramid structure of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0068] Example 1: Please refer to the accompanying drawings in the instruction manual. Figure 1 A multimodal fusion defect detection system, including:

[0069] Visible light sensor, using a visible light camera; used to acquire visible light images, represented as... Where H represents the image height and W represents the image width;

[0070] A thermal infrared sensor, employing a thermal infrared imaging module, is used to acquire thermal infrared images, represented as follows:

[0071] The hyperspectral sensor employs a hyperspectral camera; it is used to acquire hyperspectral images in the B-band, denoted as... Where B represents the number of bands, specifically the spectral dimension of the hyperspectral image;

[0072] Multi-band light source triggering control system, such as Figure 2 As shown, a main control chip (such as STM32H7) is used as the core to control three sensors in a unit time t. k The images are obtained from the same field of view under conditions where they do not interfere with each other.

[0073] The control signals emitted by the main control chip are defined as follows:

[0074]

[0075] Where, d i Let τ be the delayed trigger time of the i-th modal camera. i Let d be the exposure duration of the i-th mode camera. This is achieved by adjusting d... i With τ i It can implement various synchronization strategies such as "infrared first, then visible" and "simultaneous acquisition of visible and hyperspectral data". The key to its implementation lies in minimizing the intermodal mismatch error ∈, where ∈ = ∑ i＜j ||x i -T ij (x i )||, where T ij Let x be the intermodal spatial mapping function. i and x j Let i and j represent the input features of mode i and mode j, respectively.

[0076] The modal preprocessing and alignment module is used to preprocess and align the original images of the three modalities across modalities.

[0077] The preprocessing includes: uniform resolution, grayscale distribution, and band response;

[0078] The cross-modal alignment steps are as follows: Define the modal feature extractor. Each modality is projected onto a shared embedding space, and the extracted features are as follows: Among them, f vis For the extracted visible light features, f ir For the extracted infrared features, f hsi To extract hyperspectral features, d i H' represents the delayed trigger time of the i-th modal camera; W' represents the height of the processed feature map; C represents the width of the feature map after processing by the modal feature extractor (i.e., the number of columns in the spatial dimension); and C represents the number of channels in the feature map, i.e., the dimension of the feature vector at each spatial location.

[0079] A cross-modal feature fusion module, containing a cross-modal attention module and a multi-scale feature fusion pyramid structure, is used to achieve multimodal fusion.

[0080] like Figure 3 As shown, the cross-modal attention module is based on a self-attention structure and introduces an intermodal interaction modeling unit to achieve the correlation weight allocation of the three modalities. The weight calculation formula is as follows:

[0081]

[0082] in, f represents a learnable matrix; i For the embedding features of mode i;

[0083] The fusion output features of the cross-modal attention module are:

[0084]

[0085] in, W is used to output features for multimodal fusion. v Value transformation matrix: Maps the input features to learnable parameters in the fusion space. j : The embedding features of modality j, generated by the feature extractor, serve as input to the attention mechanism;

[0086] like Figure 4As shown, the Multi-scale Cross-Modality Feature Pyramid (MCM-FPN) structure is used to extract contextual features from different receptive fields. l represents the l-th layer in the multi-scale feature fusion pyramid structure. Each layer uses deformable convolution and channel attention mechanisms to enhance sensitivity to minute defects. The fusion output of the multi-scale feature fusion pyramid structure is defined as:

[0087]

[0088] Where CA represents the channel attention structure, w l The weights of the fusion features of the l-th layer are optimized inversely using boundary regression loss.

[0089] like Figure 4 As shown, the multi-scale feature fusion pyramid structure stacks feature layers horizontally or vertically. Each layer contains a convolutional module and a channel attention module. The upper layer features are downsampled and the lower layer features are upsampled to form a fusion path, outputting a fused feature map of uniform size.

[0090] The defect detection and output module is used for defect identification and output.

[0091] This invention utilizes three sensors—visible light imaging (VIS), thermal infrared imaging (IR), and hyperspectral imaging (HSI)—to construct a multimodal visual perception system. A multi-band light source triggering control system coordinates the different sensors to acquire images at a unified spatiotemporal scale. Multimodal information is preprocessed and aligned across modalities by a modal preprocessing and alignment module before being input into a cross-modal feature fusion module. Multimodal fusion is achieved through a cross-modal attention module and a multi-scale feature fusion pyramid structure. Finally, a defect detection and output module identifies and outputs defects, thus achieving end-to-end defect recognition from the image layer to the semantic layer. This system significantly improves defect recognition accuracy, especially for low-contrast and early-stage latent crack defects, while also possessing good scalability and deployment adaptability.

[0092] Example 2: Multimodal fusion defect detection method, the specific steps are as follows:

[0093] First, to achieve synchronous control of image acquisition, this invention constructs a multi-band light source triggering control system, with a main control chip (such as STM32H7) as the core. The main control chip outputs multiple PWM (Pulse Width Modulation) signals, each with an adjustable delay trigger time and exposure duration, controlled within a unit time t. k The internal visible light sensor, thermal infrared sensor, and hyperspectral sensor each acquire images in the same field of view under conditions that do not interfere with each other.

[0094] The control signals emitted by the main control chip are defined as follows:

[0095]

[0096] Where, d i Let τ be the delayed trigger time of the i-th modal camera. i Let d be the exposure duration of the i-th mode camera. This is achieved by adjusting d... i With τ i It can implement various synchronization strategies such as "infrared first, then visible" and "simultaneous acquisition of visible and hyperspectral data". The key to its implementation lies in minimizing the intermodal mismatch error ∈, where ∈ = ∑ i＜j ||x i -T ij (x j )||, where T ij Let x be the intermodal spatial mapping function. i and x j Let i and j represent the input features of mode i and mode j, respectively.

[0097] Secondly, the original images of the three modalities are preprocessed and cross-modal aligned through the modal preprocessing and alignment module;

[0098] Visible light images are represented as Thermal infrared images are represented as The hyperspectral image of the B band is represented as follows Where H represents the image height, W represents the image width, and B represents the number of bands, specifically referring to the spectral dimension of the hyperspectral image;

[0099] Preprocessing includes: uniform resolution, grayscale distribution, and band response;

[0100] The cross-modal alignment steps are as follows: Define the modal feature extractor. Each modality is projected onto a shared embedding space, and the extracted features are as follows:

[0101] Among them, f vis For the extracted visible light features, f ir For the extracted infrared features, f hsi To extract hyperspectral features, d i H' represents the delayed trigger time of the i-th modal camera; W' represents the height of the processed feature map; C represents the width of the feature map after processing by the modal feature extractor (i.e., the number of columns in the spatial dimension); and C represents the number of channels in the feature map, i.e., the dimension of the feature vector at each spatial location.

[0102] Next, the extracted features are input into the Cross-Modal Attention Fusion (CMAF) module within the cross-modal feature fusion module. The CMAF module is based on a self-attention structure and introduces an inter-modal interaction modeling unit to achieve the weight allocation of the correlation between the three modalities. The weight calculation formula is as follows:

[0103]

[0104] in, f represents a learnable matrix; i For the embedding features of mode i;

[0105] The fusion output features of the cross-modal attention module are:

[0106]

[0107] in, W is used to output features for multimodal fusion. v Value transformation matrix: Maps the input features to learnable parameters in the fusion space. j : The embedding features of modality j, generated by the feature extractor, serve as input to the attention mechanism;

[0108] Through the aforementioned attention mechanism, the information value weights in different modalities are dynamically adjusted, thereby improving the discriminative ability of the fused representation. In early hot spot detection tasks, infrared features dominate in shallow layers, while in obvious crack recognition, visible light texture features and hyperspectral boundary gradients enhance each other.

[0109] Subsequently, in order to further enhance the system's multi-scale detection capability, the fused output features of the cross-modal attention module are input into the multi-scale cross-modality feature pyramid (MCM-FPN) structure.

[0110] Multi-scale feature fusion pyramid structure extracts contextual features from different receptive fields. l represents the l-th layer in the multi-scale feature fusion pyramid structure. Each layer uses deformable convolution and channel attention mechanisms to enhance sensitivity to minute defects. The fusion output of the multi-scale feature fusion pyramid structure is defined as:

[0111]

[0112] Where CA represents the channel attention structure, w l The weights of the fusion features of the l-th layer are optimized inversely using boundary regression loss.

[0113] Next, during training, a combined loss function is used to optimize the detection network. The formula for the combined loss function is:

[0114]

[0115] in, For the total loss, Cross-entropy classification loss, For bounding box regression loss (such as IoU loss), The modality consistency regularization term aims to encourage the feature spaces learned by each modality to maintain structural consistency, thereby preventing modality drift from affecting the discrimination boundary.

[0116]

[0117] Among them, f i : Represents the embedding feature of the i-th modality. For example, in a cross-modal attention module, f i These can be feature vectors from visible light (VIS), thermal infrared (IR), or hyperspectral (HSI) modes. These features are extracted from the original image using a modal feature extractor. j : Represents the embedding feature of the j-th modality. (and f) i Similarly, f j It is also a feature vector extracted from another modality (such as IR or HSI). In cross-modal interactions, f j Used with f i Perform correlation calculations to achieve dynamic weight allocation between modalities.

[0118] Finally, the defect detection and output module is used to identify and output defects.

[0119] Experimental example:

[0120] When this invention was applied to the inspection of lithium battery electrodes, the results showed that compared with the single-mode inspection system (YOLOv5), the overall inspection accuracy of the fusion system was improved from 88.3% to 97.4%, especially the recall index was improved by nearly 20% for pinhole defects.

[0121] When this invention was applied to the crack detection of photovoltaic panels, the results showed that the method of this invention can detect potential hidden cracks when there are no obvious hot spots in the infrared image but the spectrum is abnormally weakly deformed. Its advantage is that the hyperspectral mode can capture subtle changes in the material, the infrared mode can detect the thermal diffusion process, and the visible light mode can identify obvious structural defects. The three complement each other and work together to significantly enhance the robustness of the system.

[0122] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal fusion defect detection system, characterized in that, include: Visible light sensor, used to acquire visible light images, is represented as... Where H represents the image height and W represents the image width; A thermal infrared sensor is used to acquire thermal infrared images, represented as... A hyperspectral sensor, used to acquire hyperspectral images in the B-band, is represented as... Where B represents the number of bands; A multi-band light source triggering control system, including a main control chip, is used to control three sensors to acquire images of the same field of view within a unit of time. The modal preprocessing and alignment module is used to preprocess and align the original images of the three modalities across modalities. A cross-modal feature fusion module, containing a cross-modal attention module and a multi-scale feature fusion pyramid structure, is used to achieve multimodal fusion. The cross-modal attention module is based on a self-attention structure and introduces an intermodal interaction modeling unit to achieve the correlation weight allocation of the three modalities. The weight calculation formula is as follows: in, f represents a learnable matrix; i For the embedding features of mode i; The fusion output features of the cross-modal attention module are: in, W is used to output features for multimodal fusion. v Value transformation matrix; f j : Embedding features of mode j; A multi-scale feature fusion pyramid structure is used to extract contextual features from different receptive fields. l represents the l-th layer in the multi-scale feature fusion pyramid structure. Each layer contains a convolutional module and a channel attention module. The fusion path is formed by downsampling features from the upper layer and upsampling features from the lower layer. The fusion output of the multi-scale feature fusion pyramid structure is defined as: Where CA represents the channel attention structure, w l The weights of the fusion features of the l-th layer are optimized inversely using boundary regression loss; The defect detection and output module is used for defect identification and output.

2. The multimodal fusion defect detection system according to claim 1, characterized in that, The control signals emitted by the main control chip are defined as follows: Where, d i Let τ be the delayed trigger time of the i-th modal camera. i Let t be the exposure duration of the i-th modal camera. k Unit of time.

3. The multimodal fusion defect detection system according to claim 2, characterized in that, By adjusting d i With τ i Adjust the synchronization strategy to minimize the intermodal mismatch error ∈, ∈=∑ i＜j ||x i -T ij (x j )||, where T ij Let x be the intermodal spatial mapping function. i and x j Let i and j represent the input features of mode i and mode j, respectively.

4. The multimodal fusion defect detection system according to claim 1, characterized in that, Preprocessing includes: uniform resolution, grayscale distribution, and band response.

5. The multimodal fusion defect detection system according to claim 1, characterized in that, The cross-modal alignment steps are as follows: Define the modal feature extractor. Each modality is projected onto a shared embedding space, and the extracted features are as follows: Among them, f vis For the extracted visible light features, f ir For the extracted infrared features, f hsi To extract hyperspectral features, d i H' represents the delayed trigger time of the i-th modal camera; W' represents the height of the processed feature map; C represents the width of the feature map after processing by the modal feature extractor; and C represents the number of channels in the feature map.

6. The multimodal fusion defect detection system according to claim 5, characterized in that, The visible light sensor uses a visible light camera; the thermal infrared sensor uses a thermal infrared imaging module; and the hyperspectral sensor uses a hyperspectral camera.

7. The multimodal fusion defect detection system according to claim 6, characterized in that, The main control chip is STM32H7.

8. A multimodal fusion defect detection method, characterized in that, Includes the following steps: First, the multi-band light source triggering control system is centered on a main control chip. The main control chip outputs multiple PWM signals, each with an adjustable delay trigger time and exposure duration, controlling the exposure within a unit time t. k The visible light sensor, thermal infrared sensor, and hyperspectral sensor each acquire images under the same field of view; Secondly, the original images of the three modalities are preprocessed and cross-modal aligned through the modal preprocessing and alignment module; Next, the extracted features are input into the cross-modal attention module within the cross-modal feature fusion module. The cross-modal attention module is based on a self-attention structure and introduces an inter-modal interaction modeling unit to achieve the weight allocation for the correlation between the three modalities. The weight calculation formula is as follows: in, f represents a learnable matrix; i For the embedding features of mode i; The fusion output features of the cross-modal attention module are: in, W is used to output features for multimodal fusion. v Value transformation matrix: Maps the input features to learnable parameters in the fusion space. j : The embedding features of modality j, generated by the feature extractor, serve as input to the attention mechanism; Subsequently, the fused output features of the cross-modal attention module are input into a multi-scale feature fusion pyramid structure, which extracts contextual features from different receptive fields. l represents the l-th layer in the multi-scale feature fusion pyramid structure. Each layer uses convolutional modules and channel attention modules to enhance sensitivity to minute defects. The fusion output of the multi-scale feature fusion pyramid structure is defined as: Where CA represents the channel attention structure, w l The weights of the fusion features of the l-th layer are optimized inversely using boundary regression loss; Next, during training, a combined loss function is used to optimize the detection network. The formula for the combined loss function is: in, For the total loss, Cross-entropy classification loss, For bounding box regression loss, This is a modal consistency regularization term; Among them, f i : Represents the embedding feature of the i-th modality, f j : Represents the embedding feature of the j-th modality; Finally, the defect detection and output module is used to identify and output defects.

9. The multimodal fusion defect detection method according to claim 8, characterized in that, The control signals emitted by the main control chip are defined as follows: Where, d i Let τ be the delayed trigger time of the i-th modal camera. i Let d be the exposure duration of the i-th modal camera; by adjusting d i With τ i Adjust the synchronization strategy to minimize the intermodal mismatch error ∈, ∈=∑ i＜j ||x i -T ij (x j )||, where T ij Let x be the intermodal spatial mapping function. i and x j Let i and j represent the input features of mode i and mode j, respectively.

10. The multimodal fusion defect detection method according to claim 8, characterized in that, Preprocessing includes: uniform resolution, grayscale distribution, and band response; The cross-modal alignment steps are as follows: Define the modal feature extractor. Each modality is projected onto a shared embedding space, and the extracted features are as follows: Among them, f vis For the extracted visible light features, f ir For the extracted infrared features, f hsi To extract hyperspectral features, d i H' represents the delayed trigger time of the i-th modal camera; W' represents the height of the processed feature map; C represents the width of the feature map after processing by the modal feature extractor; and C represents the number of channels in the feature map.

Images