Multi-mode integrated PCB defect detection method and system

Abstract

The invention discloses a multi-mode fused PCB defect detection method and system, and relates to the technical field of defect detection.The multi-mode fused PCB defect detection method comprises the steps that multi-mode data of a PCB is collected, and registration is conducted on the multi-mode data; inputting the registered multi-modal data into a pre-trained defect detection model, and extracting and adaptively weighting and fusing multi-modal features through the defect detection model to obtain a defect detection result; the defect detection model comprises a multi-branch feature extraction module, a cross-modal attention fusion module and a defect classification module. By adopting the embodiment of the invention, through registration of the multi-modal data and automatic feature extraction fusion, the multi-modal features of the PCB can be adaptively fused, the key information extraction capability is enhanced, the joint detection precision is remarkably improved, and meanwhile, the lightweight design is adapted to the industrial real-time detection requirement.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and in particular, to a multi-modal fusion-based PCB defect detection method and system. Background Art

[0002] Current PCB defect detection mainly relies on single-modal data of optical images or infrared images. Optical images can capture surface defects, but it is difficult to detect internal structure defects; infrared images reflect the internal state through thermal distribution, but have low resolution and are easily affected by the environment.

[0003] Existing multi-modal fusion methods usually adopt simple feature stitching or weighted averaging, and cannot dynamically balance the importance of features of different modalities, resulting in limited fusion effects and difficulty in adapting to complex defect scenarios. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a multi-modal fusion-based PCB defect detection method and system, which can adaptively fuse multi-modal features of the PCB, enhance the key information extraction ability, significantly improve the joint detection accuracy, and at the same time, the lightweight design meets the requirements of industrial real-time detection.

[0005] The embodiments of the present invention provide a multi-modal fusion-based PCB defect detection method, including:

[0006] Collect multi-modal data of the PCB and register the multi-modal data;

[0007] Input the registered multi-modal data into a pre-trained defect detection model, and extract and adaptively weight and fuse multi-modal features through the defect detection model to obtain a defect detection result; the defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module.

[0008] As an improvement of the above solution, the multi-modal data includes optical images and infrared images.

[0009] As an improvement of the above solution, the step of collecting multi-modal data of the PCB and registering the multi-modal data includes:

[0010] Collect optical images and infrared images of the PCB;

[0011] Use the SIFT algorithm to match the feature points of the optical image and the infrared image, and mark the successfully matched feature points to obtain the registered multi-modal data.

[0012] As an improvement of the above solution, the method for constructing the defect detection model includes:

[0013] Extract the features of multi-modal data through a convolutional network to form a multi-branch feature extraction module;

[0014] Calculate the channel attention and spatial attention of the features in each channel, and perform weighted fusion on the channel attention and spatial attention to obtain fused features, forming a cross-modal attention fusion module;

[0015] Connect the fused features of each channel through a fully connected layer, output the defect classification result, and form a defect classification module;

[0016] Obtain a defect detection model according to the multi-branch feature extraction module, the cross-modal attention fusion module, and the defect classification module.

[0017] As an improvement of the above solution, when the model input includes an optical image and an infrared image, the step of extracting the features of multi-modal data through a convolutional network to form a multi-branch feature extraction module includes:

[0018] Use the first-branch convolutional network to extract the first features of the optical image; the first-branch convolutional network is a RestNet-50 convolutional network;

[0019] Use the second-branch convolutional network to extract the first features of the infrared image; the second-branch convolutional network is a MobileNetV3 convolutional network;

[0020] Form a multi-branch feature extraction module according to the first-branch convolutional network and the second-branch convolutional network.

[0021] As an improvement of the above solution, the step of calculating the channel attention and spatial attention of the features in each channel, and performing weighted fusion on the channel attention and spatial attention to obtain fused features, forming a cross-modal attention fusion module includes:

[0022] Perform global average pooling on the first features of each channel through a squeeze-and-excitation network to calculate the first weights of the channel attention;

[0023] Perform feature transformation on the second features of each channel through a convolutional network to calculate the second weights of the spatial attention;

[0024] According to the first weights and the second weights, perform weighted fusion on the channel attention and spatial attention to obtain fused features, forming a cross-modal attention fusion module.

[0025] As an improvement of the above solution, the method for training the defect detection model includes:

[0026] Obtain a PCB multi-modal data set, and divide the PCB multi-modal data set into a training set and a test set;

[0027] Train the defect detection model using the training set. During the training process, optimize the model parameters by combining the classification loss and the alignment loss to obtain the defect category and confidence level.

[0028] Test the trained defect detection model using the test set to obtain the accuracy of defect detection. When the accuracy is not less than the preset accuracy threshold, stop the iterative training of the defect detection model.

[0029] As an improvement to the above solution, the step of training the defect detection model using the training set and optimizing the model parameters by combining the classification loss and the alignment loss during the training process to obtain the defect category and confidence level includes:

[0030] Construct a cross-entropy loss function as the classification loss function according to the output result of each training of the defect detection model and the actual defect category of the training set.

[0031] Calculate the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function.

[0032] Generate a total loss function according to the classification loss function and the alignment loss function.

[0033] Train the defect detection model using the training set and calculate the loss result of the defect detection model according to the total loss function.

[0034] Iteratively optimize the parameters of the defect detection model according to the loss result until the loss result is less than the preset loss to obtain the defect category and confidence level.

[0035] As an improvement to the above solution, the step of calculating the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function includes:

[0036] Convert the features extracted by the multi-branch feature extraction module into vectors.

[0037] Calculate the cosine similarity between vectors under different branches to obtain the alignment loss function.

[0038] An embodiment of the present invention also provides a PCB defect detection system integrating multi-modalities, including:

[0039] A data registration module for collecting multi-modal data of the PCB and registering the multi-modal data.

[0040] A defect detection module, which is used to input the registered multi-modal data into a pre-trained defect detection model, extract and adaptively weighted fuse multi-modal features through the defect detection model to obtain a defect detection result; the defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module.

[0041] Compared with the prior art, a multi-modal fusion PCB defect detection method and system disclosed by the present invention collect multi-modal data of a PCB and register the multi-modal data; input the registered multi-modal data into a pre-trained defect detection model, extract and adaptively weighted fuse multi-modal features through the defect detection model to obtain a defect detection result; the defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module. By adopting the embodiment of the present invention, it is possible to adaptively fuse the multi-modal features of the PCB, enhance the key information extraction ability, significantly improve the joint detection accuracy, and at the same time, the lightweight design meets the requirements of industrial real-time detection. Description of the Drawings

[0042] Figure 1 is a schematic flowchart of the steps of a multi-modal fusion PCB defect detection method provided by an embodiment of the present invention;

[0043] Figure 2 is a schematic flowchart of the steps of the application of the defect detection model in the multi-modal fusion PCB defect detection method provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic structural diagram of a multi-modal fusion PCB defect detection system provided by an embodiment of the present invention. Detailed Embodiments

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0046] In the description of the specification and the claims, it should be understood that the terms first, second, etc. in the specification and the claims are only used for the purpose of distinguishing the description of the same technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features, nor necessarily describing the order or time sequence. The terms can be interchanged under appropriate circumstances. Thus, the features defined with "first", "second" may explicitly or implicitly include at least one of the features.

[0047] An embodiment of the present invention provides a method for detecting PCB defects by integrating multi-modalities. Please refer to Figure 1 In this embodiment, the method for detecting PCB defects by integrating multi-modalities is specifically executed through steps S11 to S12:

[0048] S11. Collect multi-modal data of the PCB and register the multi-modal data.

[0049] Data of different modalities often contain different types of information. Multi-modal data registration refers to spatially aligning data from different modalities, which can verify their spatial consistency and further improve information complementarity and enhance the accuracy of image analysis.

[0050] S12. Input the registered multi-modal data into a pre-trained defect detection model, extract and adaptively weighted fuse multi-modal features through the defect detection model to obtain a defect detection result; the defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module.

[0051] The defect detection model provided by the embodiment of the present invention can adaptively weighted fuse multi-modal features, assign weights according to the importance of different modal features, make the model pay more attention to the key information related to defect detection, and further improve the detection effect.

[0052] In the above solution, through the registration and automatic feature extraction and fusion of multi-modal data, the multi-modal features of the PCB can be adaptively fused, the key information extraction ability can be enhanced, the joint detection accuracy can be significantly improved, and at the same time, the lightweight design meets the requirements of industrial real-time detection.

[0053] In a preferred embodiment of the present invention, the multi-modal data includes an optical image and an infrared image.

[0054] It can be understood that an optical image is obtained by an optical imaging system using the principles of light reflection and refraction, which can reflect the surface defects of the PCB; while an infrared image is obtained by an infrared detector receiving the infrared rays emitted by an object, converting them into electrical signals, and then generating them through signal processing and conversion, which can reflect the internal state of the PCB through the thermal distribution.

[0055] As a preferred implementation manner, step S11. Collect multi-modal data of the PCB and register the multi-modal data includes:

[0056] Collect the optical image and infrared image of the PCB;

[0057] Use the SIFT algorithm to match the feature points of the optical image and the infrared image, and mark the successfully matched feature points to obtain the registered multi-modal data.

[0058] It should be noted that when collecting optical images and infrared images, the stability of the collection device should be maintained to avoid blurred images caused by jitter.

[0059] In this preferred embodiment, registration is performed through the SIFT (Scale Invariant Feature Transform) algorithm. The SIFT algorithm has scale, rotation, and illumination invariance. It can extract representative feature points in the optical image and the infrared image and find the corresponding relationships between them. By matching the feature points, the relative position and orientation of the two images in space can be determined.

[0060] In some other preferred embodiments, when performing image registration, the internal and external parameters of the collection device can also be combined, and other registration algorithms can also be used. Exemplarily, the SIFT algorithm performs registration through feature point matching. Those skilled in the art can use algorithms such as the SURF algorithm and the ORB algorithm for feature point matching, and can also use the affine transformation algorithm for registration.

[0061] In the embodiments of the present invention, before performing image registration, the multi-modal data is uniformly scaled to a preset pixel size and normalized. Exemplarily, the preset pixel size is 256*256 pixels.

[0062] Please refer to Figure 2 , Figure 2 , which is a schematic diagram of the step flow of the application of the defect detection model provided by the embodiments of the present invention in the multi-modal PCB defect detection method.

[0063] As a preferred implementation manner, constructing the defect detection model is performed through steps S21 - S24:

[0064] S21: Respectively extract the features of the multi-modal data through a convolutional network to form a multi-branch feature extraction module.

[0065] The convolutional network has a powerful feature extraction ability and can automatically extract local features and global features in the image. The multi-branches run independently, and the convolutional networks used in each branch can be the same or different. It can independently extract features for different modal data, retain the unique information of each modal data, and provide a basis for subsequent fusion.

[0066] S22: Calculate the channel attention and spatial attention of the features in each channel, and perform weighted fusion on the channel attention and spatial attention to obtain a fusion feature, forming a cross-modal attention fusion module.

[0067] Channel attention is used to measure the importance of features in each channel, enabling the model to focus on the channels that are more critical for defect detection. Spatial attention can locate the key spatial positions of features. The combination of the two can enhance features from both the channel and spatial dimensions, enabling better fusion of different modality features and highlighting information related to defects.

[0068] In some preferred embodiments of the present invention, it is not necessary to calculate the channel attention and the spatial attention for each branch convolution module. Instead, different branches correspond to different attention patterns, and when performing feature fusion, different attention weights for different features are considered.

[0069] S23. Connect the fused features of each channel through a fully connected layer, output the defect classification result, and form a defect classification module.

[0070] The fully connected layer can integrate the information of each channel of the fused features, map them to different classification categories, and realize the judgment of whether there are defects on the PCB and the types of defects.

[0071] S24. Obtain a defect detection model according to the multi-branch feature extraction module, the cross-modal attention fusion module, and the defect classification module.

[0072] Further, preferably, when the model input includes an optical image and an infrared image, in step S21, extract the features of the multi-modal data through a convolutional network respectively to form a multi-branch feature extraction module, including:

[0073] Use the first branch convolutional network to extract the first features of the optical image; the first branch convolutional network is a ResNet-50 convolutional network;

[0074] Use the second branch convolutional network to extract the first features of the infrared image; the second branch convolutional network is a MobileNetV3 convolutional network;

[0075] Form a multi-branch feature extraction module according to the first branch convolutional network and the second branch convolutional network.

[0076] Optical images and infrared images have different characteristics. Using different convolutional network architectures can better adapt to these characteristics. Taking the embodiments of the present invention as an example, ResNet-50 has strong feature extraction ability and depth, and is suitable for processing optical images with rich information, and can capture complex texture and detail information in optical images; while MobileNetV3 is a lightweight network architecture with high computational efficiency. For infrared images with relatively low resolution and relatively less information, it can not only meet the requirements of feature extraction, but also reduce the consumption of computing resources and improve the overall operation efficiency of the model.

[0077] It can be understood that the first branch convolutional network and the second branch convolutional network can also select other convolutional networks, and the selection criterion is the amount of data that the convolutional network can process. That is, the amount of data of the first branch convolutional network is not less than that of the second branch convolutional network.

[0078] Furthermore, preferably, in step S22, calculating the channel attention and spatial attention of the feature in each channel, and performing weighted fusion on the channel attention and spatial attention to obtain a fusion feature, forming a cross-modal attention fusion module, including:

[0079] Performing global average pooling on the first feature of each channel through a squeeze-and-excitation network to calculate the first weight of the channel attention;

[0080] Performing feature transformation on the second feature of each channel through a convolutional network to calculate the second weight of the spatial attention;

[0081] According to the first weight and the second weight, performing weighted fusion on the channel attention and spatial attention to obtain a fusion feature, forming a cross-modal attention fusion module.

[0082] It should be noted that optical images mainly reflect the appearance information of objects such as surface texture, color, and shape. Different channels contain different visual features, and the importance of each channel information for defect detection is different. The channel attention mechanism can automatically adjust the weights of each channel through learning, highlight the channel features that are important for defect detection, suppress irrelevant channel information, and thus more effectively extract the defect features in optical images.

[0083] Infrared images are mainly based on the thermal radiation characteristics of objects for imaging, reflecting the temperature distribution of objects, and are used to detect internal defects or thermal anomalies of objects. Their defects often manifest as temperature anomalies in specific regions, and spatial position information is crucial. For example, a short circuit defect inside a PCB board will cause local temperature rise. Through the spatial attention mechanism, the model can focus on the spatial regions with temperature anomalies in infrared images, accurately identify the defect position and range. Relatively speaking, the differences between channels in infrared images contribute less significantly to defect detection than spatial information, so it is more suitable to use spatial attention to strengthen key spatial features.

[0084] In some preferred embodiments, when performing weighted fusion on the channel attention and spatial attention to obtain a fusion feature, the weighted feature is also dimension-reduced through a 1×1 convolution.

[0085] As a preferred implementation manner, training the defect detection model is performed through steps S31-S33:

[0086] S31. Obtain the PCB multi-modal dataset, and divide the PCB multi-modal dataset into a training set and a test set.

[0087] By dividing the dataset, data can be effectively utilized. And by learning features and patterns on the training set and evaluating the generalization ability on the test set, overfitting of the model can be avoided, enabling the model to adapt to unseen data and improving the accuracy and reliability of defect detection.

[0088] S32. Use the training set to train the defect detection model. During the training process, optimize the model parameters by combining the classification loss and the alignment loss to obtain the defect category and confidence level.

[0089] It should be noted that in the embodiments of the present invention, optimizing the model parameters by combining the classification loss and the alignment loss can enhance the consistency of multi-modal features. There are differences in the feature distributions and representation methods of different modal data. The alignment loss helps to make different modal features closer in the feature space, and the classification loss guides the model to classify correctly. The combination of the two can improve the model performance. At the same time, considering that there may be errors in multi-modal data registration, the alignment loss can mitigate the impact of such errors on the model, making the model not overly dependent on the registration accuracy and enhancing the robustness of the model.

[0090] S33. Use the test set to test the trained defect detection model to obtain the accuracy of defect detection. When the accuracy is not less than the preset accuracy threshold, stop the iterative training of the defect detection model.

[0091] In some preferred embodiments, it can also be to divide the PCB multi-modal dataset into several subsets and perform cross-validation according to the subsets.

[0092] Further, step S32. Use the training set to train the defect detection model. During the training process, optimize the model parameters by combining the classification loss and the alignment loss to obtain the defect category and confidence level, includes:

[0093] Construct a cross-entropy loss function as the classification loss function according to the output result of each training of the defect detection model and the actual defect category of the training set.

[0094] Calculate the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function.

[0095] Generate a total loss function according to the classification loss function and the alignment loss function.

[0096] Use the training set to train the defect detection model, and calculate the loss result of the defect detection model according to the total loss function.

[0097] Iteratively optimize the parameters of the defect detection model according to the loss result until the loss result is less than a preset loss to obtain the defect category and confidence level.

[0098] Exemplarily, the total loss function is expressed as L total = L cls + λL align ; where L cls is the classification loss function, and L align is the alignment loss function; λ is the balance coefficient.

[0099] It should be noted that the balance coefficient is used to adjust the weights of the classification loss and the alignment loss. In some preferred embodiments, an adaptive method is used to adjust the balance coefficient so that the model dynamically balances the influence of the two according to the training situation.

[0100] Furthermore, preferably, calculating the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function includes:

[0101] Convert the features extracted by the multi-branch feature extraction module into vectors;

[0102] Calculate the cosine similarity between the vectors under different branches to obtain the alignment loss function.

[0103] After converting the features into vectors, the relevant theories and methods of the vector space can be used to analyze and process the features of different modalities.

[0104] In some preferred embodiments, the features are flattened to be converted into vectors.

[0105] In the embodiments of the present invention, taking the cosine similarity as the basis of the alignment loss function can minimize the alignment loss during the model training process, so that the features of different modalities are more aligned in the feature space.

[0106] Using the multi-modal fusion PCB defect detection method provided by the embodiments of the present invention can adaptively fuse the multi-modal features of the PCB, enhance the key information extraction ability, significantly improve the joint detection accuracy, and at the same time, the lightweight design meets the requirements of industrial real-time detection.

[0107] The embodiments of the present invention provide a multi-modal fusion PCB defect detection system. Please refer to Figure 3 , the multi-modal fusion PCB defect detection system includes a data registration module 11 and a defect detection module 12, where:

[0108] The data registration module 11 is used to collect the multi-modal data of the PCB and register the multi-modal data;

[0109] The defect detection module 12 is configured to input the registered multi-modal data into a pre-trained defect detection model, extract and adaptively weighted-fuse multi-modal features through the defect detection model to obtain a defect detection result; the defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module.

[0110] As a preferred implementation, the multi-modal data includes an optical image and an infrared image.

[0111] As a preferred implementation, the data registration module 11 includes:

[0112] An image acquisition unit for acquiring the optical image and the infrared image of the PCB;

[0113] A feature point matching unit for matching the feature points of the optical image and the infrared image using the SIFT algorithm, and marking the successfully matched feature points to obtain the registered multi-modal data.

[0114] As a preferred implementation, the method for constructing the defect detection model includes:

[0115] Extract the features of the multi-modal data through a convolutional network respectively to form a multi-branch feature extraction module;

[0116] Calculate the channel attention and spatial attention of the features in each channel, and perform weighted fusion on the channel attention and spatial attention to obtain a fusion feature, forming a cross-modal attention fusion module;

[0117] Connect the fusion features of each channel through a fully connected layer, output a defect classification result, and form a defect classification module;

[0118] Obtain a defect detection model according to the multi-branch feature extraction module, the cross-modal attention fusion module, and the defect classification module.

[0119] Further, preferably, when the model input includes an optical image and an infrared image, the extracting the features of the multi-modal data through a convolutional network respectively to form a multi-branch feature extraction module includes:

[0120] Use the first-branch convolutional network to extract the first feature of the optical image; the first-branch convolutional network is a ResNet-50 convolutional network;

[0121] Use the second-branch convolutional network to extract the first feature of the infrared image; the second-branch convolutional network is a MobileNetV3 convolutional network;

[0122] Form a multi-branch feature extraction module according to the first-branch convolutional network and the second-branch convolutional network.

[0123] Furthermore, as a preferred implementation, calculating the channel attention and spatial attention of the feature in each channel, and performing weighted fusion on the channel attention and spatial attention to obtain a fused feature, forming a cross-modal attention fusion module, including:

[0124] Performing global average pooling on the first feature of each channel through a squeeze-and-excitation network to calculate the first weight of the channel attention;

[0125] Performing feature transformation on the second feature of each channel through a convolutional network to calculate the second weight of the spatial attention;

[0126] According to the first weight and the second weight, performing weighted fusion on the channel attention and spatial attention to obtain a fused feature, forming a cross-modal attention fusion module.

[0127] As a preferred implementation, the method for training the defect detection model includes:

[0128] Obtaining a PCB multi-modal dataset, and dividing the PCB multi-modal dataset into a training set and a test set;

[0129] Training the defect detection model with the training set, and optimizing the model parameters by combining the classification loss and the alignment loss during the training process to obtain the defect category and confidence;

[0130] Testing the trained defect detection model with the test set to obtain the accuracy of defect detection. When the accuracy is not less than a preset accuracy threshold, stop the iterative training of the defect detection model.

[0131] Further preferably, the training the defect detection model with the training set, and optimizing the model parameters by combining the classification loss and the alignment loss during the training process to obtain the defect category and confidence includes:

[0132] Constructing a cross-entropy loss function according to the output result of each training of the defect detection model and the actual defect category of the training set as the classification loss function;

[0133] Calculating the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function;

[0134] Generating a total loss function according to the classification loss function and the alignment loss function;

[0135] Training the defect detection model with the training set, and calculating the loss result of the defect detection model according to the total loss function;

[0136] Iteratively optimize the parameters of the defect detection model according to the loss result until the loss result is less than a preset loss, and obtain the defect category and confidence level.

[0137] Further preferably, calculating the cosine similarity of the features extracted by the multi-branch feature extraction module and establishing an alignment loss function includes:

[0138] Convert the features extracted by the multi-branch feature extraction module into vectors;

[0139] Calculate the cosine similarity between vectors under different branches to obtain the alignment loss function.

[0140] By using a PCB defect detection system that fuses multi-modalities provided by the embodiments of the present invention, it can adaptively fuse the multi-modal features of the PCB, enhance the key information extraction ability, significantly improve the joint detection accuracy, and at the same time, the lightweight design adapts to the requirements of industrial real-time detection.

[0141] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0142] The above is the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A PCB defect detection method integrating multi-modalities, characterized in that, Including: Collecting multi-modal data of the PCB and registering the multi-modal data; Inputting the registered multi-modal data into a pre-trained defect detection model, and extracting and adaptively weighted fusing multi-modal features through the defect detection model to obtain a defect detection result; The defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module.

2. The multimodal fusion PCB defect detection method according to claim 1, wherein The multi-modal data includes an optical image and an infrared image.

3. The multimodal fusion PCB defect detection method according to claim 2, wherein The collecting multi-modal data of the PCB and registering the multi-modal data includes: Collecting the optical image and the infrared image of the PCB; Using the SIFT algorithm to match the feature points of the optical image and the infrared image, and marking the successfully matched feature points to obtain the registered multi-modal data.

4. A multi-modal integrated PCB defect detection method according to claim 1, characterized in that, The method for constructing the defect detection model includes: Respectively extracting the features of the multi-modal data through a convolutional network to form a multi-branch feature extraction module; Calculating the channel attention and spatial attention of the features in each channel, and weighted fusing the channel attention and spatial attention to obtain a fused feature, thereby forming a cross-modal attention fusion module; Connecting the fused features of each channel through a fully connected layer to output a defect classification result, thereby forming a defect classification module; Obtaining a defect detection model according to the multi-branch feature extraction module, the cross-modal attention fusion module, and the defect classification module.

5. The multimodal fusion PCB defect detection method according to claim 4, characterized in that When the model input includes an optical image and an infrared image, the respectively extracting the features of the multi-modal data through a convolutional network to form a multi-branch feature extraction module includes: Using a first-branch convolutional network to extract the first feature of the optical image; the first-branch convolutional network is a RestNet-50 convolutional network; Using a second-branch convolutional network to extract the first feature of the infrared image; the second-branch convolutional network is a MobileNetV3 convolutional network; Forming a multi-branch feature extraction module according to the first-branch convolutional network and the second-branch convolutional network.

6. The multimodal fusion PCB defect detection method according to claim 5, wherein The calculating the channel attention and spatial attention of the features in each channel, and weighted fusing the channel attention and spatial attention to obtain a fused feature, thereby forming a cross-modal attention fusion module includes: Performing global average pooling on the first feature of each channel through a squeeze-and-excitation network to calculate the first weight of the channel attention; Performing feature transformation on the second feature of each channel through a convolutional network to calculate the second weight of the spatial attention; According to the first weight and the second weight, weighted fusing the channel attention and spatial attention to obtain a fused feature, thereby forming a cross-modal attention fusion module.

7. A method for detecting PCB defects integrating multi-modalities according to claim 1 or 4, characterized in that The method for training the defect detection model includes: Obtaining a PCB multi-modal data set, and dividing the PCB multi-modal data set into a training set and a test set; Training the defect detection model using the training set, and optimizing the model parameters by combining a classification loss and an alignment loss during the training process to obtain a defect category and a confidence level; Use the test set to test the trained defect detection model to obtain the accuracy of defect detection. When the accuracy is not less than the preset accuracy threshold, stop the iterative training of the defect detection model.

8. The method for detecting PCB defects integrating multi-modalities according to claim 7, wherein, Training the defect detection model using the training set, and optimizing the model parameters by combining classification loss and alignment loss during the training process to obtain the defect category and confidence, including: Construct a cross-entropy loss function as the classification loss function according to the output result of each training of the defect detection model and the actual defect category of the training set; Calculate the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function; Generate a total loss function according to the classification loss function and the alignment loss function; Train the defect detection model using the training set, and calculate the loss result of the defect detection model according to the total loss function; Iteratively optimize the parameters of the defect detection model according to the loss result until the loss result is less than the preset loss to obtain the defect category and confidence.

9. The method for detecting PCB defects by fusing multi-modalities according to claim 8, wherein, The calculating the cosine similarity of the features extracted by the multi-branch feature extraction module to establish an alignment loss function includes: Convert the features extracted by the multi-branch feature extraction module into vectors; Calculate the cosine similarity between vectors under different branches to obtain the alignment loss function.

10. A PCB defect detection system integrating multi-modalities, characterized in that, Including: A data registration module for collecting multi-modal data of the PCB and registering the multi-modal data; A defect detection module for inputting the registered multi-modal data into a pre-trained defect detection model, and extracting and adaptively weighted fusing multi-modal features through the defect detection model to obtain a defect detection result; The defect detection model includes a multi-branch feature extraction module, a cross-modal attention fusion module, and a defect classification module.

Images