PCB defect segmentation method and system based on multi-modal fusion

Abstract

The invention discloses a PCB defect segmentation method and system based on multi-modal fusion, and relates to the field of image processing, and the method comprises the steps: inputting a visible light image, an infrared image and a polarized light image of the same PCB sample, forming a multi-modal PCB defect segmentation data set, extracting a feature sequence of the multi-modal PCB defect segmentation data set through an independent Stem block, and obtaining a feature sequence of the multi-modal PCB defect segmentation data set; and dynamic intelligent fusion is carried out by using a modal perception selective fusion module. And the fused features are then input into a Transform and Mamba double-branch decoder in parallel, global context and local details are captured respectively, pixel-level adaptive integration is carried out through a space-channel gating fusion module, and finally a defect segmentation mask of the PCB sample is output by a pixel decoder. According to the method, the single-mode defect is effectively overcome, and the accuracy and robustness of PCB defect segmentation are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a PCB defect segmentation method and system based on multimodal fusion. Background Technology

[0002] Printed Circuit Board (PCB) defect detection is of significant practical importance in industrial manufacturing and quality control. However, the subtlety of defects leads to extremely high error rates and low efficiency in traditional manual inspection methods. Therefore, automated inspection technology based on machine vision has gradually become the mainstream development direction. Early PCB inspection relied heavily on traditional image processing algorithms, such as image differencing, edge detection, template matching, and morphological analysis. While these algorithms can detect obvious defects in ideal environments, their robustness and accuracy are significantly insufficient when facing complex backgrounds, lighting variations, and minute defects. With the continuous development of machine learning, researchers began to introduce handcrafted features such as texture, geometry, and edges, and combine them with classifiers such as Support Vector Machines (SVMs) for defect identification. Although this stage achieved automation of inspection, the overall performance improvement was limited by the expressive power of handcrafted features and the algorithm's adaptability to high-dimensional data.

[0003] In recent years, the rise of deep learning technology has brought new breakthroughs to PCB defect detection. However, these methods are mostly object detection-oriented, only providing the bounding box location of defects, making it difficult to achieve fine-grained characterization of defect boundaries and morphology, thus failing to meet the precise quality assessment requirements of high-reliability manufacturing scenarios. In contrast, image segmentation-based PCB defect detection methods achieve finer-grained structural recognition through pixel-level classification. Meanwhile, the Transformer architecture, with its advantages in modeling long-range dependencies and global contextual relationships, has been gradually introduced into the PCB segmentation field. However, most existing segmentation methods rely primarily on visible light images, which often fail to capture subtle or potential defect information under complex lighting, low contrast, or surface reflection conditions, leading to insufficient feature representation and decreased generalization performance. Therefore, multimodal fusion technology has been introduced into PCB detection tasks, jointly utilizing complementary information from different modalities such as visible light, infrared, and polarization to enhance the robustness and completeness of feature representation. However, the high-dimensional heterogeneity of multimodal features and the challenge of cross-modal semantic alignment make efficient fusion and decoding of multi-scale information a new research bottleneck. Summary of the Invention

[0004] The purpose of this invention is to provide a PCB defect segmentation method and system based on multimodal fusion, so as to alleviate the problem of insufficient detection accuracy in existing PCB defect detection due to changes in illumination, complex textures and differences in the scale of small defects.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: In a first aspect, the present invention provides a PCB defect segmentation method based on multimodal fusion, characterized by comprising the following steps: The visible light image, infrared image, and polarized light image of the same PCB sample are input to form a multimodal PCB defect segmentation dataset; Each modal image in the multimodal PCB defect segmentation dataset is input into an independent Stem block for processing to obtain the downsampled feature sequence of each modality; The downsampled feature sequences of each modality are input into the modality-aware selective fusion module, and fused features are generated through splicing, dimensionality reduction, gated hybrid expert network processing and channel reweighting. The fused features are input in parallel to the Transformer branch and the Mamba branch for decoding, and the first branch decoding features and the second branch decoding features are output respectively. The first branch decoding features and the second branch decoding features are input into the spatial-channel gating fusion module. The final decoding features are obtained by pixel-level fusion of channel gating and spatial gating. The final decoded features are input into the pixel decoder for multi-scale feature aggregation and upsampling, and the defect segmentation mask of the PCB sample is output.

[0006] Furthermore, each independent Stem block includes a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer connected in sequence. The input of each Stem block is a single-modal image, and the output is a feature map with a spatial size of 1 / 4 of the input image and C channels. The feature map is then flattened into a sequence to obtain the downsampled feature sequence of each modality.

[0007] Furthermore, the specific process of inputting the downsampled feature sequences of each modality into the modality-aware selective fusion module, and generating fused features through splicing, dimensionality reduction, gated hybrid expert network processing, and channel reweighting is as follows: The downsampled feature sequences of visible light, infrared light and polarized light are concatenated along the channel dimension to obtain the concatenated features; The spliced ​​features are subjected to layer normalization and linear projection dimensionality reduction to obtain dimensionality-reduced features; The dimensionality reduction features are input into a gated hybrid expert module containing four expert networks. A noise gating mechanism is used to dynamically select the two experts with the highest weights for each token and perform weighted processing to obtain the expert fusion output. The expert fusion output is divided into three modal scaling vectors corresponding to visible light, infrared light, and polarized light, respectively; Global average pooling and sigmoid activation are performed on each modality scaling vector to generate the channel reweighting factor for each modality; Using the channel reweighting factor, the original visible light, infrared and polarized light downsampled feature sequences are calibrated at the channel level, and the learnable bias terms corresponding to each mode are superimposed to obtain the calibrated modal features. The fused features are generated by weighting and summing the calibrated modal features using a set of learnable weights.

[0008] Furthermore, the Transformer branch includes the following steps: using a set of learnable query vectors and the fused features to perform multi-head cross-attention calculation to obtain context-aware query vectors; then performing self-attention calculation and feedforward network processing on the context-aware query vectors to output the first branch decoding features.

[0009] Furthermore, the Mamba branch includes the following steps: reshaping the set of learnable query vectors into a two-dimensional spatial grid; superimposing learnable two-dimensional positional codes onto the two-dimensional spatial grid; inputting the superimposed features into a state space model for sequence modeling, and outputting the second branch decoded features.

[0010] Furthermore, the first branch decoding features and the second branch decoding features are input into the spatial-channel gating fusion module. The specific process of obtaining the final decoding features through pixel-level fusion of channel gating and spatial gating is as follows: The first branch decoding features and the second branch decoding features are concatenated along the channel dimension to obtain the concatenated decoding features; Perform a full connection operation and Sigmoid activation on the splicing and decoding features to generate the channel gating coefficient α; The channel gating coefficient α is reshaped into a two-dimensional spatial mapping diagram; Perform a 3×3 convolution operation on the two-dimensional spatial mapping graph to generate spatial gating weights β; Based on the channel gating coefficient α and the spatial gating weight β, the first branch decoding features and the second branch decoding features are weighted and fused at the pixel level to obtain the final decoding features.

[0011] Furthermore, the calculation formula for the weighted fusion is as follows: O = β ⊙ (α ⊙ T) + (1 - β) ⊙ ((1 - α) ⊙ M); Where O represents the final decoding feature, T represents the first branch decoding feature, M represents the second branch decoding feature, and ⊙ represents element-wise multiplication.

[0012] Furthermore, the final decoded features are input into the pixel decoder for multi-scale feature aggregation and upsampling, outputting a defect segmentation mask for the PCB sample. The pixel decoder, based on the Mask2Former architecture, aggregates features of different scales through a multi-scale deformable attention mechanism to obtain aggregated features. The aggregated features are fused using a top-down path in the form of a feature pyramid network, and combined with mask query to generate the defect segmentation mask.

[0013] Furthermore, when the method is applied to model training, it uses a multimodal PCB defect segmentation dataset containing visible light images, infrared images, and polarized light images and their corresponding pixel-level defect annotations; the multimodal PCB defect segmentation dataset is augmented through random scaling, random cropping, and random horizontal flipping.

[0014] Secondly, the present invention also provides a PCB defect segmentation system based on multimodal fusion, comprising: The image input module is used to input visible light images, infrared images, and polarized light images of the same PCB sample to form a multimodal PCB defect segmentation dataset. The feature extraction module, connected to the image input module, is used to process each modal image in the multimodal PCB defect segmentation dataset through multiple independent Stem blocks to obtain the downsampled feature sequence of each modality. The modality fusion module, connected to the feature extraction module, is used to receive the downsampled feature sequences of each modality and generate fused features through splicing, dimensionality reduction, gated hybrid expert network processing and channel reweighting. A dual-branch decoding module, connected to the modality fusion module, includes a parallel Transformer branch and a Mamba branch, used to receive the fused features and perform parallel decoding, and output the first branch decoding features and the second branch decoding features respectively; A branch fusion module, connected to the dual-branch decoding module, is used to receive the first branch decoding features and the second branch decoding features, and obtain the final decoding features by pixel-level fusion through channel gating and spatial gating; The mask generation module, connected to the branch fusion module, is used to receive the final decoded features, perform multi-scale feature aggregation and upsampling, and output the defect segmentation mask of the PCB sample.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects: This invention provides a PCB defect segmentation method based on multimodal fusion. A PCB defect segmentation dataset is constructed, and visible light, infrared, and polarized light modes are dynamically integrated through a modality-aware selective fusion module. By utilizing a hybrid expert mechanism and gated weighting, the method effectively overcomes the sensitivity of single modes to illumination changes and noise, enhancing the robustness and generalization ability of feature representation. This invention balances global and local modeling, using Transformer and Mamba branches to capture long-range semantic dependencies and local details respectively. The parallel design of the two branches takes into account both global context and fine-grained boundary information. Furthermore, a spatial-channel gated fusion module achieves pixel-level adaptive fusion of channel and spatial gating, reducing feature distortion and ensuring clear and accurate segmentation boundaries, making it particularly suitable for small-sized and ambiguous defects. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the PCB defect segmentation method based on multimodal fusion in an embodiment of the present invention.

[0017] Figure 2 This is an overall framework diagram of the PCB defect segmentation method based on multimodal fusion in an embodiment of the present invention.

[0018] Figure 3 This is a diagram of the modal perception selective fusion module of the PCB defect segmentation method based on multimodal fusion in an embodiment of the present invention.

[0019] Figure 4 This is a spatial-channel gating fusion module diagram of the PCB defect segmentation method based on multimodal fusion in an embodiment of the present invention.

[0020] Figure 5 The figure shows the qualitative experimental results provided in the embodiments of the present invention. Detailed Implementation

[0021] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0022] This invention provides a PCB defect segmentation method based on multimodal fusion, such as... Figure 1 As shown, the specific steps of this method include: Step 1: Construct a multimodal PCB defect segmentation dataset for model training and evaluation. This multimodal PCB defect segmentation dataset includes three modal images of the same PCB sample: visible light image, infrared image and polarized light image, and is equipped with pixel-level defect annotation labels, covering 7 common defects such as exposed copper, missing holes, rodent bites, open circuits, short circuits, burrs and stray copper.

[0023] Based on 540 original images, the images were segmented into small blocks using a sliding window cropping strategy (window size 512×512 pixels, stride 256 pixels) to obtain the initial training set. To further improve the model's generalization ability, data augmentation operations were performed on the multimodal PCB defect segmentation dataset, including random scaling, random cropping, and random horizontal flipping, expanding the multimodal PCB defect segmentation dataset to 5400 images. Subsequently, the augmented dataset was randomly divided into training, validation, and test sets in an 8:1:1 ratio.

[0024] Step 2: Construct the MAMFormer network. The backbone of the MAMFormer network is divided into three independent Stem blocks (corresponding to...). Figure 2 The input and Stem part of the dataset process three modalities of images respectively. The input multimodal PCB defect segmentation dataset X={X vis ,X ir ,X pol (Each modality has a size of H×W×3). Each modality is first processed through an independent stem block: each stem block contains a 7×7 convolutional layer, batch normalization (BatchNorm), ReLU activation function, and max pooling layer, with the output downsampled to... × Feature map F of ×C vis ,F ir ,F pol These feature maps are flattened into sequence form R. N×C (where N= × The downsampled feature sequences of each modality are obtained for subsequent fusion. like Figure 3 As shown, the downsampled feature sequences of the three modalities obtained in step 2 (denoted as X, Y, and Z, respectively) are input into the Modal-Aware Selective Fusion Module (MASF). The specific process is as follows: Concatenation: The feature sequences of the three modalities are concatenated along the channel dimension to obtain joint features. .

[0025] Dimensionality reduction: Layer normalization is performed on the concatenated features, and the feature dimension is reduced from 3C to C′ by layer normalization and linear projection, generating dimensionality-reduced features. , where B is the batch size Gated Hybrid Expert Network Processing: Dimensionality Reduction Features The input is fed into a gated mixture-of-experts (MoE) network. In this embodiment, the MoE network includes independent expert networks. A noisy gating network is used to calculate the weight of each token in the sequence for each expert, and the top two (top-2) experts with the highest weights are dynamically selected for weighted summation to obtain the expert fusion output. This process can be represented as:

[0026] The output O is designed to contain three modal scaling vectors.

[0027] Channel reweighting: The expert fusion output O is divided into three parts, corresponding to the scaling vectors of visible light, infrared, and polarized light, respectively. Global average pooling (GAP) and sigmoid activation function are applied to each scaling vector to generate the channel reweighting factor for each modality.

[0028] Modal feature calibration and aggregation: Using the generated channel reweighting factor, the original visible light, infrared and polarized light downsampled feature sequences are calibrated at the channel level, and the learnable bias terms corresponding to each mode are superimposed to obtain the calibrated modal features; The three modal features after calibration are weighted and summed using a set of learnable weights to generate the final fused features.

[0029] Step 4: Input the fused features in parallel into the Transformer branch and the Mamba branch for decoding, and output the first branch decoded features and the second branch decoded features respectively, such as... Figure 2 and Figure 4 As shown, the Transformer branch includes the following steps: A set of learnable query vectors are used to perform multi-head cross-attention (MHCA) computation with the fused features, enabling the query to aggregate relevant information from the fused features and obtain a context-aware query vector. This query vector is then subjected to multi-head self-attention (MHSA) computation and a feed-forward network (FFN) layer for processing, outputting the first branch decoded features.

[0030] The Mamba branch includes the following steps: using the same set of learnable query vectors as the Transformer branch, reshaping the set of learnable query vectors into a two-dimensional spatial grid, superimposing learnable two-dimensional positional encodings onto the two-dimensional spatial grid to inject spatial positional information, inputting the superimposed features into the state space model for sequence modeling, and outputting the decoded features of the second branch.

[0031] like Figure 4 As shown, the decoded features of the first branch and the decoded features of the second branch are input into the Spatial-Channel Gated Fusion Module (SCGF) for dynamic fusion. The specific process is as follows: The first branch decoding features and the second branch decoding features are concatenated along the channel dimension to obtain the concatenated decoding features; The spliced ​​decoding features are subjected to a fully connected operation and Sigmoid activation to generate channel gating coefficients α. The channel gating coefficient α is between 0 and 1, reflecting the relative importance of the Transformer branch on each feature channel. The channel gating coefficient α is reshaped into a two-dimensional spatial mapping diagram; A 3×3 convolution operation is performed on the two-dimensional spatial mapping to generate a spatial gating weight β. The value of the spatial gating weight β at each pixel position is between 0 and 1, reflecting the reliability of the channel gating result at that spatial position or the degree of adjustment required.

[0032] According to the weighted fusion calculation formula: O = β ⊙ (α ⊙ T) + (1 - β) ⊙ ((1 - α) ⊙ M); Where O represents the final decoded feature, T represents the first branch decoded feature, M represents the second branch decoded feature, and ⊙ represents element-wise multiplication. The final decoded feature O is obtained. This formula enables fine-tuning of the two branch outputs in both channel and spatial dimensions.

[0033] Step 5: Input the final decoded features into the pixel decoder, perform multi-scale feature aggregation and upsampling, and output the defect segmentation mask of the PCB sample.

[0034] In this embodiment, the final decoded features are input into the pixel decoder for multi-scale feature aggregation and upsampling, outputting the defect segmentation mask of the PCB sample. The pixel decoder is based on the Mask2Former architecture and aggregates features of different scales through a multi-scale deformable attention mechanism to obtain aggregated features. The aggregated features are fused using a top-down path in the form of a feature pyramid network, and combined with mask query to generate the defect segmentation mask.

[0035] When applied to model training, the method uses the multimodal PCB defect segmentation dataset prepared in step 1, sets the optimizer to AdamW, the initial learning rate to 1e-4, the weight decay to 0.05, and the batch size to 4, and trains for a total of 80,000 iterations. A multinomial learning rate decay strategy is adopted, with a decay power of 0.9 and a minimum learning rate of 0. The loss function is a weighted sum of cross-entropy loss and Dice loss to simultaneously optimize pixel classification accuracy and region overlap. After training, the model weights with the best performance on the validation set are saved. During the inference phase, these weights are loaded, and the visible light, infrared, and polarized light three-modal images of the test set are input. The MAMFormer network will automatically execute steps 2 to 5 above, outputting the defect segmentation mask of the PCB sample, thus completing the defect segmentation task.

[0036] This invention introduces a Modality-Aware Selective Fusion (MASF) module in the initial stage of the network to adaptively fuse visible light, infrared, and polarization modal features, achieving dynamic weighting and redundancy suppression of cross-modal features, thereby generating multimodal feature inputs with unified representation capabilities. Subsequently, a hybrid Transformer-Mamba decoder is designed in the segmentation head, combining the global context modeling capability of Transformer with the local continuity modeling advantage of Mamba to achieve fine segmentation of multimodal features and enhanced semantic consistency, thereby improving the accuracy and robustness of PCB defect segmentation.

[0037] In another embodiment of the present invention, a PCB defect segmentation system based on multimodal fusion includes: The image input module is used to input visible light images, infrared images, and polarized light images of the same PCB sample to form a multimodal PCB defect segmentation dataset. The feature extraction module, connected to the image input module, is used to process each modal image in the multimodal PCB defect segmentation dataset through multiple independent Stem blocks to obtain the downsampled feature sequence of each modality. The modality fusion module, connected to the feature extraction module, is used to receive the downsampled feature sequences of each modality and generate fused features through splicing, dimensionality reduction, gated hybrid expert network processing and channel reweighting. A dual-branch decoding module, connected to the modality fusion module, includes a parallel Transformer branch and a Mamba branch, used to receive the fused features and perform parallel decoding, and output the first branch decoding features and the second branch decoding features respectively; A branch fusion module, connected to the dual-branch decoding module, is used to receive the first branch decoding features and the second branch decoding features, and obtain the final decoding features by pixel-level fusion through channel gating and spatial gating; The mask generation module, connected to the branch fusion module, is used to receive the final decoded features, perform multi-scale feature aggregation and upsampling, and output the defect segmentation mask of the PCB sample.

[0038] The effects of the present invention will be further described below with reference to simulation experiments.

[0039] Simulation experimental conditions: The hardware platform for the simulation experiment of the invention is: Intel(R) Core(TM) i7-11700X CPU with a main frequency of 2.5GHz, 32.0GB of memory, and a 3090Ti graphics card.

[0040] Software platform: Win10 operating system, PyTorch 1.12.0.

[0041] The simulation experiment of this invention uses a self-constructed multimodal PCB defect segmentation dataset, containing 540 images covering seven common defects: exposed copper, missing vias, rodent bites, open circuits, short circuits, burrs, and stray copper. Each sample includes three-modal images: visible light (vi), infrared (ir), and polarized light (pol). The images are then cropped into 512×512 pixel blocks with a stride of 256 pixels to ensure complete defect coverage. Random scaling, cropping, and horizontal flipping are performed to complete image preprocessing and enhancement. The dataset is randomly divided into training, validation, and test sets in an 8:1:1 ratio. All images are labeled with pixel-level defect masks manually using the X-AnyLabeling tool to ensure boundary accuracy.

[0042] Simulation content and result analysis: The simulation experiments of this invention aim to verify the effectiveness of the MAMFormer network in PCB defect segmentation. The experiments compare the proposed method with various state-of-the-art techniques, including CNN-based, Transformer-based, and SSM-based models, to comprehensively evaluate its performance.

[0043] To verify the simulation results of this invention, we calculated the average performance of segmenting seven types of defects. We compared the results of this invention and three existing technologies on the test samples using three evaluation metrics: accuracy (Acc), intersection over union (IoU), and F1-score, combined with floating point operations per second (FLOPs) and the number of parameters (Params). Acc reflects the proportion of correctly classified pixels out of the total number of pixels; IoU measures the overlap between the predicted ground truth and the ground truth; and F1-score balances the model's precision and recall, comprehensively evaluating the model's overall performance in classifying positive and negative samples. Higher values ​​for these three metrics indicate better detection and classification performance. Meanwhile, FLOPs and Params are used to measure the model's computational complexity and parameter size. Lower FLOPs and Params indicate that the model maintains high accuracy while achieving better computational efficiency and storage performance.

[0044] The comparison results of the average values ​​of all objective evaluation indicators in the present invention and existing advanced methods are shown in Table 1: Table 1

[0045] As can be seen from Table 1, the three metrics mIoU, mAcc and mFscore of this invention outperform existing methods in segmentation on the PCB defect dataset.

[0046] Figure 5 This is a qualitative result diagram of the quality improvement obtained from the experiment of this invention. Figure 5 As can be seen, the method proposed in this invention can generate more accurate defect boundaries, effectively suppress false detections, and achieve better segmentation quality across multiple defect categories.

[0047] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

[0048] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A PCB defect segmentation method based on multimodal fusion, characterized in that, Includes the following steps: The visible light image, infrared image, and polarized light image of the same PCB sample are input to form a multimodal PCB defect segmentation dataset; Each modal image in the multimodal PCB defect segmentation dataset is input into an independent Stem block for processing to obtain the downsampled feature sequence of each modality; The downsampled feature sequences of each modality are input into the modality-aware selective fusion module, and fused features are generated through splicing, dimensionality reduction, gated hybrid expert network processing and channel reweighting. The fused features are input in parallel to the Transformer branch and the Mamba branch for decoding, and the first branch decoding features and the second branch decoding features are output respectively. The first branch decoding features and the second branch decoding features are input into the spatial-channel gating fusion module. The final decoding features are obtained by pixel-level fusion of channel gating and spatial gating. The final decoded features are input into the pixel decoder for multi-scale feature aggregation and upsampling, and the defect segmentation mask of the PCB sample is output.

2. The PCB defect segmentation method based on multimodal fusion according to claim 1, characterized in that, Each independent Stem block consists of a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer connected in sequence. The input of each Stem block is a single-modal image, and the output is a feature map. The feature map is flattened into a sequence to obtain the downsampled feature sequence of each modality.

3. The PCB defect segmentation method based on multimodal fusion according to claim 1, characterized in that, The specific process of inputting the downsampled feature sequences of each modality into the modality-aware selective fusion module, and generating fused features through splicing, dimensionality reduction, gated hybrid expert network processing, and channel reweighting is as follows: The downsampled feature sequences of visible light, infrared light and polarized light are concatenated along the channel dimension to obtain the concatenated features; The spliced ​​features are subjected to layer normalization and linear projection dimensionality reduction to obtain dimensionality-reduced features; The dimensionality reduction features are input into a gated hybrid expert module containing four expert networks. A noise gating mechanism is used to dynamically select the two experts with the highest weights for each token and perform weighted processing to obtain the expert fusion output. The expert fusion output is divided into three modal scaling vectors corresponding to visible light, infrared light, and polarized light, respectively; Global average pooling and sigmoid activation are performed on each modality scaling vector to generate the channel reweighting factor for each modality; Using the channel reweighting factor, the original visible light, infrared and polarized light downsampled feature sequences are calibrated at the channel level, and the learnable bias terms corresponding to each mode are superimposed to obtain the calibrated modal features. The fused features are generated by weighting and summing the calibrated modal features using a set of learnable weights.

4. The PCB defect segmentation method based on multimodal fusion according to claim 1, characterized in that, The Transformer branch includes the following steps: using a set of learnable query vectors and the fused features to perform multi-head cross-attention calculation to obtain context-aware query vectors; then performing self-attention calculation and feedforward network processing on the context-aware query vectors to output the first branch decoding features.

5. The PCB defect segmentation method based on multimodal fusion according to claim 4, characterized in that, The Mamba branch includes the following steps: reshaping the set of learnable query vectors into a two-dimensional spatial grid; superimposing learnable two-dimensional positional codes onto the two-dimensional spatial grid; inputting the superimposed features into a state space model for sequence modeling, and outputting the decoded features of the second branch.

6. The PCB defect segmentation method based on multimodal fusion according to claim 1, characterized in that, The first branch decoding features and the second branch decoding features are input into the spatial-channel gating fusion module. The pixel-level fusion of channel gating and spatial gating is used to obtain the final decoding features. The specific process is as follows: The first branch decoding features and the second branch decoding features are concatenated along the channel dimension to obtain the concatenated decoding features; Perform a full connection operation and Sigmoid activation on the splicing and decoding features to generate the channel gating coefficient α; The channel gating coefficient α is reshaped into a two-dimensional spatial mapping diagram; Perform a 3×3 convolution operation on the two-dimensional spatial mapping graph to generate spatial gating weights β; Based on the channel gating coefficient α and the spatial gating weight β, the first branch decoding features and the second branch decoding features are weighted and fused at the pixel level to obtain the final decoding features.

7. The PCB defect segmentation method based on multimodal fusion according to claim 6, characterized in that, The formula for calculating the weighted fusion is as follows: O = β ⊙ (α ⊙ T) + (1 - β) ⊙ ((1 - α) ⊙ M); Where O represents the final decoding feature, T represents the first branch decoding feature, M represents the second branch decoding feature, and ⊙ represents element-wise multiplication.

8. The PCB defect segmentation method based on multimodal fusion according to claim 1, characterized in that, The final decoded features are input into the pixel decoder for multi-scale feature aggregation and upsampling, outputting a defect segmentation mask for the PCB sample. The pixel decoder is based on the Mask2Former architecture and aggregates features of different scales through a multi-scale deformable attention mechanism to obtain aggregated features. The aggregated features are fused using a top-down path in the form of a feature pyramid network, and combined with mask query to generate the defect segmentation mask.

9. The PCB defect segmentation method based on multimodal fusion according to claim 1, characterized in that, When applied to model training, the method uses a multimodal PCB defect segmentation dataset containing visible light images, infrared images, and polarized light images, along with their corresponding pixel-level defect annotations. The multimodal PCB defect segmentation dataset is augmented through random scaling, random cropping, and random horizontal flipping.

10. A PCB defect segmentation system based on multimodal fusion, characterized in that, include: The image input module is used to input visible light images, infrared images, and polarized light images of the same PCB sample to form a multimodal PCB defect segmentation dataset. The feature extraction module, connected to the image input module, is used to process each modal image in the multimodal PCB defect segmentation dataset through multiple independent Stem blocks to obtain the downsampled feature sequence of each modality. The modality fusion module, connected to the feature extraction module, is used to receive the downsampled feature sequences of each modality and generate fused features through splicing, dimensionality reduction, gated hybrid expert network processing and channel reweighting. A dual-branch decoding module, connected to the modality fusion module, includes a parallel Transformer branch and a Mamba branch, used to receive the fused features and perform parallel decoding, and output the first branch decoding features and the second branch decoding features respectively; A branch fusion module, connected to the dual-branch decoding module, is used to receive the first branch decoding features and the second branch decoding features, and obtain the final decoding features by pixel-level fusion through channel gating and spatial gating; The mask generation module, connected to the branch fusion module, is used to receive the final decoded features, perform multi-scale feature aggregation and upsampling, and output the defect segmentation mask of the PCB sample.

Images