A PCB defect cross-domain collaborative detection method based on a multi-modal model

Abstract

The application provides a PCB defect cross-domain cooperative detection method based on a multi-modal model, which comprises collecting X-ray images and optical images, processing the X-ray images and the optical images by using a Vision Transformer model to obtain visual features, processing production log texts by using a BERT model to obtain text features, performing feature alignment on the visual features and the text features by using a cross-attention mechanism to obtain fusion features, and performing defect classification on the fusion features to obtain PCB defect types. The Vision Transformer model is used to process the X-ray images and the optical images, which can simultaneously capture internal features of PCB defects and surface features of PCB defects. The BERT model is used to understand and analyze the production log texts, and extract process parameter information related to defects. The cross-attention mechanism is used to perform feature alignment on the visual features and the text features, so that feature fusion between multi-modal data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of SMT processing, and particularly relates to a PCB defect cross-domain collaborative detection method based on a multi-modal model. BACKGROUND

[0002] In the PCB (Printed Circuit Board) manufacturing process, the detection of welding quality is a key link to ensure product performance and reliability. Traditional PCB defect detection methods mainly rely on optical imaging technology or X-ray imaging technology, but these two technologies have their own limitations in different application scenarios. Optical imaging can effectively capture surface defects, but it has weak detection capability for internal solder joint defects, while X-ray imaging can detect internal defects, but due to its high cost and complex operation, it is difficult to be widely applied in production lines.

[0003] In addition, production log texts, as an important data source for recording the production process, usually contain process parameter information related to defects, but traditional detection methods cannot fully exploit the correlation between text data and image data, resulting in limited accuracy and robustness of defect detection. Existing methods are usually single-modal image analysis or manual rule application, lacking cross-modal feature fusion and intelligent reasoning capabilities.

[0004] Therefore, there is an urgent need for a method that can combine Vision Transformer model and BERT model to realize efficient information interaction and feature fusion, and perform cross-modal data feature alignment and collaborative reasoning, to improve the accuracy and robustness of PCB defect type detection, especially in defect data scarce and small sample scenarios, thereby realizing cross-line migration. SUMMARY

[0005] To overcome the problems in the related art, the purpose of the present application is to provide a PCB defect cross-domain collaborative detection method based on a multi-modal model, which can combine Vision Transformer model and BERT model to realize efficient information interaction and feature fusion, and perform cross-modal data feature alignment and collaborative reasoning, to improve the accuracy and robustness of PCB defect type detection, especially in defect data scarce and small sample scenarios, thereby realizing cross-line migration.

[0006] A PCB defect cross-domain collaborative detection method based on a multi-modal model, comprising:

[0007] X-ray images and optical images are collected;

[0008] The Vision Transformer model is used to process the X-ray images and optical images to obtain visual features;

[0009] The BERT model is used to process the production log text to obtain text features.

[0010] A cross-attention mechanism is used to align the visual features and the text features to obtain fusion features.

[0011] The fusion features are classified to obtain the PCB defect type.

[0012] In the preferred technical solution of the present application, the Vision Transformer model is used to process the X-ray image and the optical image to obtain visual features, which includes:

[0013] The visual features are extracted by the following formula:

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] Wherein, PatchEmbed is image block embedding, CLS is classification mark, represents splicing operation, PosEmb is position coding, MSA is multi-head self-attention mechanism, LN is layer normalization, MLP is multi-layer perception, l is the number of Transformer layers, I is the original image, X patch is the image block, X cls is the classification mark block, Z0 is the initial visual feature matrix, Z l is the lth visual feature matrix, is the mapped feature matrix, V feat is the visual feature.

[0021] In the preferred technical solution of the present application, the BERT model is used to process the production log text to obtain text features, which includes:

[0022] The text features are extracted by the following formula:

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] wherein TokenEmbed is a word embedding, SegEmbed is a segment embedding, FFN is a feed-forward neural network, T emb is a post-encoding text feature, H0 is an initial text feature matrix, H l-1 is an (l-1)th text feature matrix, H l is an lth text feature matrix; MultiHeadAttn is a multi-head attention, LN is a layer normalization, is an lth layer-normalized text matrix, T feat is the text feature.

[0029] In the preferred technical solution of the present application, the Vision Transformer model is used to process the X-ray image and the optical image to obtain visual features, which comprises:

[0030] The Vision Transformer model is used to process the X-ray image to obtain image features of internal defects of the PCB.

[0031] The Vision Transformer model is used to process the optical image to obtain image features of surface defects of the PCB.

[0032] The image features of the internal defects of the PCB and the image features of the surface defects of the PCB constitute the visual features.

[0033] In the preferred technical solution of the present application, the cross-attention mechanism is used to align the visual features and the text features to obtain fusion features, which comprises:

[0034] The following formula is used to extract the fusion features:

[0035] ;

[0036] wherein Q is a query matrix of the text features, K is a key matrix of the visual features, V is a value matrix of the visual features, softmax is a normalized exponential function, d k is a vector length of the key matrix of the visual features, and CrossAttn(Q, K, V) is the fusion features.

[0037] In the preferred technical solution of the present application, before the fusion features are classified to obtain the PCB defect type, the method further comprises:

[0038] A contrast learning loss function is constructed.

[0039] Construct a masked autoencoder loss function;

[0040] The final loss function is obtained by adding the contrastive learning loss function and the mask autoencoder loss function.

[0041] The final loss function is used to train the classification model to be trained, resulting in a trained classification model.

[0042] In a preferred embodiment of the present invention, constructing the contrastive learning loss function includes:

[0043] The contrastive learning loss function is constructed using the following formula:

[0044] ;

[0045] Among them, L contrast Let x be the contrastive learning loss function. i For the i-th input sample, For the i-th positive sample, Let f be the j-th negative sample, f be the feature extraction function, sim be the similarity function, τ be the temperature parameter, N be the total number of input samples, and M be the total number of negative samples.

[0046] In a preferred embodiment of the present invention, the construction of the mask autoencoder loss function includes:

[0047] The masked autoencoder loss function is constructed using the following formula:

[0048] ;

[0049] Where L mae The mask autoencoding loss function is... x is the total number of masked samples. i For the i-th input sample, Let be the expected value of the i-th input sample.

[0050] In a preferred embodiment of the present invention, before constructing the mask autoencoder loss function, the method further includes:

[0051] Determine the total number of mask samples;

[0052] In each mask sample, generate rp masks with random shapes and positions, where rp ≥ 1.

[0053] In a preferred embodiment of the present invention, generating rp masks of random shape and position in each mask sample includes:

[0054] A first random number is randomly generated within a first range. The first random number is a positive integer. The minimum value of the first range is 1, and the maximum value of the first range is 5.

[0055] Within the width and height range of the mask sample, a second random number and a third random number are randomly generated; the second random number is a positive integer or a decimal, and the third random number is a positive integer or a decimal.

[0056] Using the second random number as the center x-coordinate and the third random number as the center y-coordinate, a mask region corresponding to the first random number is generated; the mask region can be circular, elliptical, rectangular, triangular, or star-shaped.

[0057] The beneficial effects of this invention are as follows:

[0058] This invention provides a cross-domain collaborative detection method for PCB defects based on a multimodal model. The method involves acquiring X-ray and optical images, processing them using a Vision Transformer model to obtain visual features, and processing production log text using a BERT model to obtain text features. Both visual and text features are characteristics of PCB defects and complement each other, jointly reflecting the type of PCB defect. A cross-attention mechanism is used to align the visual and text features, resulting in fused features. These fused features are then used for defect classification to determine the PCB defect type. Using the Vision Transformer model to process X-ray and optical images allows for the simultaneous capture of both internal and surface features of PCB defects. These internal and surface features together constitute the visual features, enabling efficient visual feature extraction. The BERT model is used to understand and analyze the production log text, extracting process parameter information related to defects. The BERT model uses a self-attention mechanism to capture long-distance dependencies in the text, ultimately obtaining text features. This combination of textual and visual information significantly improves the accuracy of PCB defect detection. By aligning visual and textual features through a cross-attention mechanism, feature fusion between multimodal data is achieved, further improving the accuracy and robustness of PCB defect types. Attached Figure Description

[0059] Figure 1 This is a flowchart of the cross-domain collaborative detection method for PCB defects based on a multimodal model according to the present invention;

[0060] Figure 2 This is a flowchart of the process of training the classification model to be trained according to the present invention;

[0061] Figure 3 This is the PCB defect type of the present invention. Detailed Implementation

[0062] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides a cross-domain collaborative detection method for PCB defects based on a multimodal model, including:

[0065] S1: Acquire X-ray and optical images.

[0066] S2: The Vision Transformer model is used to process X-ray and optical images to obtain visual features.

[0067] S3: Use the BERT model to process production log text and obtain text features.

[0068] S4: Employ a cross-attention mechanism to align visual and textual features, resulting in fused features.

[0069] S5: Classify the fused features to obtain PCB defect types.

[0070] The X-ray imaging detection principle utilizes the difference in X-ray absorption at different locations when X-rays penetrate a PCB. A detector receives the remaining X-rays and converts them into electrical signals, ultimately forming a grayscale image reflecting the internal structure of the PCB. X-ray images are used to detect internal defects in the PCB. The optical imaging detection principle involves capturing reflected light signals from the PCB surface using a high-resolution optical imaging system. Photoelectric sensors such as CMOS / CCD convert the light signals into electrical signals and digitize them, ultimately forming a digital image. Optical images are used to detect defects on the PCB surface. X-ray and optical images complement each other to better reflect the overall defects of the PCB.

[0071] The process of using the Vision Transformer model to process X-ray and optical images to obtain visual features includes:

[0072] Image features are extracted using the following formula:

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] Where PatchEmbed is image patch embedding, CLS is classification label, ⊕ indicates concatenation operation, PosEmb is positional encoding, MSA is multi-head self-attention mechanism, LN is layer normalization, MLP is multi-layer perceptron, l is the number of Transformer layers, I is the original image, and X is the image structure. patch For image patches, X cls For classification label blocks, Z0 is the initial visual feature matrix, Z l For the l-th visual feature matrix, V is the mapped feature matrix. feat It is a visual feature.

[0080] Vision Transformer (ViT) is an image recognition model based on a pure Transformer architecture. Its core idea is to split an image into sequential image patches and model the global context relationship through a self-attention mechanism to achieve efficient feature extraction and classification.

[0081] The specific steps for the Vision Transformer model to extract visual features are as follows:

[0082] Step 1: Image Input and Patch Division

[0083] ;

[0084] Where I represents the original image, PatchEmbed represents image patch embedding, and X represents... patch For image blocks.

[0085] First, input the original image I. The original image can be an X-ray image. It can also be an optical image. Where H is the height of the original image and W is the width of the original image. 3 represents the number of channels in the image, i.e., RGB three channels; here, we take an input X-ray image as an example. Next, the original image I is divided into multiple patches of a fixed size, such as 16×16. Each patch is flattened and transformed into a vector through a linear projection layer, i.e., Patch Embedding. Finally, a series of image patches X are output. patch Xpatch It is a matrix.

[0086] Step 2: Add category tags and location codes

[0087] ;

[0088] ;

[0089] Where CLS is the classification label, ⊕ is the concatenation operation, PosEmb is the positional encoding, and X... cls Z0 is the classification label block, and Z0 is the initial visual feature matrix, which is the vector matrix after adding position encoding.

[0090] First in X patch A learnable classification label CLS is added to the beginning of the sequence, and then concatenated with all patch vectors to output the classification label block X. cls That is, the matrix after adding the classification labels. Next, X... cls Adding the positional encoding to each patch, including CLS, adds learnable spatial location information and outputs the initial visual feature matrix Z0, which is the matrix after adding the positional encoding.

[0091] Step 3: Transformer encoder processing

[0092] ;

[0093] ;

[0094] Multi-head attention mechanisms employ multiple attention heads, each focusing on different patterns such as image texture, edges, or color. The features extracted by these multiple attention heads are then concatenated and fused, with attention computation performed in parallel across different subspaces, thereby capturing rich contextual information from the input data. Multilayer perceptrons (MLPs) are used to map the features extracted by the multi-head attention mechanism to the desired results for the task. MLPs use multiple fully connected layers and one activation layer to approximate arbitrary continuous functions, and are simple to implement and can be processed in parallel.

[0095] The Transformer encoder consists of L identical layers, each layer comprising two sub-modules. The first sub-module contains a multi-head self-attention (MSA) layer, a multi-mode auto-attention (MLP) layer, and a residual connection network. The second sub-module contains an MLP layer, a multi-head self-attention (MLP) layer, and a residual connection network. MSA calculates the dependencies between all elements in the input sequence, capturing global contextual information. LN normalizes the feature dimensions of each sample, stabilizing the training process. MLP performs non-linear transformations on the features at each location, enhancing the model's expressive power. The residual connection network preserves original information while preventing gradient vanishing.

[0096] First, Z0 is processed sequentially using LN, MSA, and residual join. Then, LN, MLP, and residual join are performed sequentially. This process is repeated multiple times to output the mapped feature matrix. .

[0097] Step 4: Extract Image Features

[0098] ;

[0099] After processing by multiple Transformer encoders, the output is... The 0th dimension corresponds to [CLS], and the vector V corresponding to [CLS] is extracted directly by slicing. feat As a global image feature, V feat Used for image feature classification.

[0100] The process of using the BERT model to process production log text yields text features, including:

[0101] The following formula is used to extract text features:

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] Where TokenEmbed is word embedding, SegEmbed is segment embedding, FFN is feedforward neural network, and T emb H0 represents the encoded text features, and H0 represents the initial text feature matrix. l-1 For the (l-1)th text feature matrix, H lLet be the l-th text feature matrix; MultiHeadAttn represents multi-head attention, and LN represents layer normalization. Let T be the text matrix after normalization of the l-th layer. feat The text features are described above.

[0108] BERT is a pre-trained language model based on the Transformer architecture that significantly improves the performance of Natural Language Processing (NLP) tasks through bidirectional context modeling. The specific steps of the BERT model in extracting text features are as follows:

[0109] Step 1: Input the embedding layer T emb

[0110] ;

[0111] The input text T is transformed into a vector using word embedding, segment embedding, and positional encoding. Word embedding maps input words to vectors, segment embedding identifies sentence categories, and positional encoding injects word position information. Finally, the word embedding matrix TokenEmbed(T), the segment embedding matrix SegEmbed(T), and the positional encoding PosEmbed(T) are summed to obtain the encoded text features T. emb .

[0112] Step 2: Encoder Stacking

[0113] ;

[0114] ;

[0115] ;

[0116] First, regarding T emb Perform initialization, set T emb As the initial text feature matrix H0, H0 is sequentially subjected to LN, MultiHeadAttn, and residual concatenation, followed by LN, FFN, and residual concatenation. This process is repeated l times to obtain the normalized text matrix of the l-th layer. .

[0117] Step 3: Text Feature Extraction

[0118] ;

[0119] Take the last layer The vector at position 0, i.e., the feature corresponding to the [CLS] tag, serves as the global semantic feature T for the entire sentence. feat T feat Used for text feature classification.

[0120] This embodiment provides a cross-domain collaborative detection method for PCB defects based on a multimodal model. The method involves acquiring X-ray and optical images, processing them using a Vision Transformer model to obtain visual features, and processing production log text using a BERT model to obtain text features. Both visual and text features are characteristics of PCB defects and complement each other, jointly reflecting the type of PCB defect. A cross-attention mechanism is used to align the visual and text features, resulting in fused features. These fused features are then used for defect classification to determine the PCB defect type. Using the Vision Transformer model to process X-ray and optical images allows for the simultaneous capture of both internal and surface features of PCB defects. These internal and surface features together constitute the visual features, enabling efficient visual feature extraction. The BERT model is used to understand and analyze the production log text, extracting process parameter information related to defects. The BERT model uses a self-attention mechanism to capture long-distance dependencies in the text, ultimately obtaining text features. This combination of textual and visual information significantly improves the accuracy of PCB defect detection. By aligning visual and textual features through a cross-attention mechanism, feature fusion between multimodal data is achieved, further improving the accuracy and robustness of PCB defect types.

[0121] Example 2

[0122] like Figure 1 As shown, this embodiment provides a cross-domain collaborative detection method for PCB defects based on a multimodal model. This embodiment describes the differences from Embodiment 1, and the method includes:

[0123] S1: Acquire X-ray and optical images.

[0124] S2: The Vision Transformer model is used to process X-ray and optical images to obtain visual features.

[0125] S3: Use the BERT model to process production log text and obtain text features.

[0126] S4: Employ a cross-attention mechanism to align visual and textual features, resulting in fused features.

[0127] S5: Classify the fused features to obtain PCB defect types.

[0128] The process of using the Vision Transformer model to process X-ray and optical images yields visual features, including:

[0129] S21: The Vision Transformer model is used to process X-ray images to obtain image features of internal defects in the PCB.

[0130] S22: The Vision Transformer model is used to process optical images to obtain image features of PCB surface defects.

[0131] S23: The image features of the internal defects of the PCB and the image features of the surface defects of the PCB constitute the visual features.

[0132] The Vision Transformer model has the following advantages: (1) Long-range dependency is captured in one pass, and self-attention directly calculates weights between any two patches, so global relationships can be modeled in one forward pass. (2) Linear complexity scalability. (3) High efficiency of pre-trained transfer. (4) Under the requirements of small sample size, multiple labels and high accuracy, it can be quickly adapted through pre-training and fine-tuning, which is significantly better than traditional CNN.

[0133] In Vision Transformer, different levels of Transformer blocks, such as Z1, Z2, ..., Z, are used. L Features at different abstraction levels are generated, i.e., downsampling is performed continuously using a feature pyramid approach. The smaller the size of the downsampled image patch, the closer the represented feature is to the global semantics; that is, Z1 has the largest size, Z... L The smallest size is used for low-level features. Low-level features retain more detail and are suitable for surface defects in optical images. High-level features focus more on global semantics, which is applicable to the internal structure of X-ray images.

[0134] The method employs a cross-attention mechanism to align visual and textual features to obtain fused features, including:

[0135] The following formula is used to extract the fusion features:

[0136] ;

[0137] Where Q is the query matrix for text features, K is the key matrix for visual features, V is the value matrix for visual features, softmax is the normalization exponential function, and d kLet Q be the vector length of the key matrix of visual features, and CrossAttn(Q,K,V) be the fused feature. The cross-attention mechanism employs a Transformer model. The cross-attention module is the core module in the Transformer architecture for cross-sequence or cross-modal information fusion. It enables a set of query vectors to actively focus on another set of key and value vectors, thus breaking down information silos and achieving a mutually influential interaction mechanism. In the cross-attention mechanism, the query matrix Q of text features comes from the current sequence (text modality), while the key matrix K and value matrix V of visual features come from the external sequence (image modality).

[0138] This embodiment utilizes a cross-attention mechanism to process data from different modalities, which improves detection accuracy while addressing the resolution differences between modalities. Combining visual and textual information enhances the accuracy of defect classification.

[0139] Example 3

[0140] like Figure 1 As shown, this embodiment provides a cross-domain collaborative detection method for PCB defects based on a multimodal model. This embodiment describes the differences from Embodiment 1, and the method includes:

[0141] S1: Acquire X-ray and optical images.

[0142] S2: The Vision Transformer model is used to process X-ray and optical images to obtain visual features.

[0143] S3: Use the BERT model to process production log text and obtain text features.

[0144] S4: Employ a cross-attention mechanism to align visual and textual features, resulting in fused features.

[0145] S5: Classify the fused features to obtain PCB defect types.

[0146] like Figure 2 As shown, before classifying the fused features to obtain the PCB defect type, the process further includes:

[0147] S41': Construct the contrastive learning loss function.

[0148] S42': Construct the masked autoencoder loss function.

[0149] S43': Add the contrastive learning loss function and the mask autoencoder loss function to obtain the final loss function.

[0150] S44': The final loss function is used to train the classification model to be trained, and the trained classification model is obtained.

[0151] Both contrastive learning loss and masked autoencoder loss functions belong to the category of self-supervised learning loss functions. By employing self-supervised learning, the system can mine potential patterns and features from large amounts of unlabeled data, further improving the model's robustness and adaptability to different application scenarios. The contrastive learning loss function enhances the discriminative power of features by narrowing the feature distance between similar samples and widening the feature distance between different samples. The masked autoencoder loss function learns the underlying structure of the data by reconstructing the masked features.

[0152] To improve the generalization ability of the model in scenarios with scarce data, this invention adopts self-supervised learning methods such as contrastive learning and mask autoencoder to learn the features of image and text data through unlabeled data, thereby improving the detection performance in small sample scenarios.

[0153] The construction of the contrastive learning loss function includes:

[0154] The contrastive learning loss function is constructed using the following formula:

[0155] ;

[0156] Among them, L contrast Let x be the contrastive learning loss function. i For the i-th input sample, For the i-th positive sample, Let f be the j-th negative sample, f be the feature extraction function, sim be the similarity function, τ be the temperature parameter, N be the total number of input samples, and M be the total number of negative samples.

[0157] The i-th positive sample With the i-th input sample x i Semantic similarity, such as different enhanced versions of the same image. The j-th negative sample With the i-th input sample x i The semantics are different, for example and x i Each represents a different image.

[0158] This embodiment uses the cosine similarity function as an example. Cosine similarity only calculates the difference in direction between two vectors, which can eliminate the difference in vector length. The temperature parameter τ is used to control the sharpness of the distribution; the smaller τ is, the stronger the penalty for negative samples.

[0159] The construction of the mask autoencoder loss function includes:

[0160] The masked autoencoder loss function is constructed using the following formula:

[0161] ;

[0162] Where L mae The mask autoencoding loss function is... x is the total number of masked samples. i For the i-th input sample, Let be the expected value of the i-th input sample.

[0163] The masked autoencoder loss function only calculates the reconstruction error in the masked region; other regions are not included in the loss calculation. The pixel value of the masked region is 1, and the pixel value of the unmasked region is 0. The mask ratio, i.e., the proportion of the masked region to the total sample area, is inversely proportional to the information density. Since the input samples in this embodiment contain both image and semantic information, the mask ratio ranges from 45% to 65%, preferably 55%. Using the masked autoencoder loss function reduces the amount of information involved in the computation, preventing excessive information in the visible part from dominating the gradient and causing the model to degenerate into an identity mapping.

[0164] like Figure 3 As shown, PCB defect types include short circuits, open circuits, vias, excess copper foil, burrs, scratches, rodent bites, and pinholes. The trained classification model can identify any of the above PCB defect types.

[0165] This embodiment combines contrastive learning loss function and masked autoencoder loss function to construct the final loss function. The final loss function is then used to train the classification model to obtain the trained classification model. The classification model to be trained is generally a lightweight adapter, employing a two-layer MLP. The trained classification model simultaneously possesses the advantages of enhanced feature discriminativeness (reducing the feature distance between samples of the same type and increasing the feature distance between samples of different types), and calculating reconstruction error only in masked regions, reducing the amount of information involved in the computation and improving computational efficiency.

[0166] Example 4

[0167] like Figure 1 As shown, this embodiment provides a cross-domain collaborative detection method for PCB defects based on a multimodal model. This embodiment describes the differences between it and Embodiment 1.

[0168] Before classifying the fused features to obtain the PCB defect type, the method further includes:

[0169] S41': Construct the contrastive learning loss function.

[0170] S42': Construct the masked autoencoder loss function.

[0171] S43': Add the contrastive learning loss function and the mask autoencoder loss function to obtain the final loss function.

[0172] S44': The final loss function is used to train the classification model to be trained, and the trained classification model is obtained.

[0173] Before constructing the masked autoencoder loss function, the following steps are also included:

[0174] S41'': Determine the total number of mask samples.

[0175] S42'': Generate rp masks of random shape and position in each mask sample, where rp≥1.

[0176] The process of generating rp masks of random shape and position in each mask sample includes:

[0177] S421'': Randomly generate a first random number within a first range. The first random number is a positive integer. The minimum value of the first range is 1, and the maximum value of the first range is 5.

[0178] S422'': Within the width and height range of the mask sample, a second random number and a third random number are randomly generated; the second random number is a positive integer or a decimal, and the third random number is a positive integer or a decimal.

[0179] S423'': Using the second random number as the center x-coordinate and the third random number as the center y-coordinate, a mask region corresponding to the first random number is generated; the mask region is circular, elliptical, rectangular, triangular, or star-shaped.

[0180] If the first random number is 1, the masked area is circular, with two parameters: the coordinates of the circle and the radius of the masked area. If the first random number is 2, the masked area is elliptical, with three parameters: the size of the major axis, the size of the minor axis, and the direction of the major axis. If the first random number is 3, the masked area is rectangular, with two parameters: the length of the rectangle and the width of the rectangle. If the first random number is 4, the masked area is triangular, with three parameters: the coordinates of the first vertex, the second vertex, and the third vertex. If the first random number is 5, the masked area is star-shaped, with three parameters: the number of vertices, the density, and the radius.

[0181] Using the second random number as the center x-coordinate and the third mask number as the center y-coordinate, generate circles, ellipses, rectangles, triangles, or stars corresponding to the first random number based on the parameters of each shape. If the second random number is close to the width of the mask sample, and / or the third random number is close to the height of the mask sample, then the portion of the shape exceeding the mask sample size is eliminated, and the portion of the shape within the mask sample is retained. For example, if the first random number is 1, the size of the mask sample is 500×500, and the center of the circular mask region is located at the rightmost side of the mask sample (i.e., the x-coordinate of the center is 500), then only half of the circular mask region is within the mask sample. The mask sample refers to the sample containing the masked region.

[0182] This embodiment uses three random numbers to determine the shape and center coordinates of the mask region. The position and shape of the mask are uncertain, which makes the generated mask more random, thereby improving the training effect of the mask autoencoder loss function on the training classification model and improving the generalization performance of the trained classification model for different application scenarios.

[0183] Example 5

[0184] This embodiment provides a cross-domain collaborative detection device for PCB defects based on a multimodal model, including:

[0185] The data acquisition module is used to acquire X-ray images and optical images;

[0186] The image processing module is used to process X-ray and optical images using the Vision Transformer model to obtain visual features;

[0187] The text processing module is used to process production log text using the BERT model to obtain text features;

[0188] The feature alignment module is used to align visual and textual features using a cross-attention mechanism to obtain fused features.

[0189] The defect classification module is used to classify the fused features to obtain the PCB defect type.

[0190] The PCB defect cross-domain collaborative detection device based on a multimodal model in this embodiment is used to execute the PCB defect cross-domain collaborative detection method based on a multimodal model.

[0191] This embodiment also provides a computer device, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communication with external terminals via a network connection.

[0192] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a cross-domain collaborative detection method for PCB defects based on a multimodal model. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0194] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A PCB defect cross-domain collaborative detection method based on a multi-modal model, characterized in that, The method comprises the following steps: Collecting X-ray images and optical images; Processing the X-ray images and the optical images by using a Vision Transformer model to obtain visual features; Processing production log texts by using a BERT model to obtain text features; Aligning the visual features and the text features by using a cross-attention mechanism to obtain fusion features; Constructing a contrast learning loss function; Constructing a mask auto-encoding loss function; Adding the contrast learning loss function and the mask auto-encoding loss function to obtain a final loss function; Training a to-be-trained classification model by using the final loss function to obtain a trained classification model; Classifying the fusion features to obtain a PCB defect type; The step of aligning the visual features and the text features by using a cross-attention mechanism to obtain fusion features comprises the following steps: The fusion features are extracted by using the following formula: ； wherein Q is a query matrix of text features, K is a key matrix of visual features, V is a value matrix of visual features, softmax is a normalization exponential function, d k is a vector length of the key matrix of visual features, and CrossAttn(Q, K, V) is a fusion feature. The step of constructing a mask auto-encoding loss function comprises the following steps: The mask auto-encoding loss function is constructed by using the following formula: ； wherein L mae is the mask auto-encoding loss function, is the total number of mask samples, x i is the i-th input sample, is the expected value of the i-th input sample; Before the step of constructing the mask auto-encoding loss function, the method further comprises the following steps: Determining a total number of mask samples; Generating rp random masks with random shapes and positions in each mask sample, wherein rp is greater than or equal to 1; the proportion of the mask area to the total sample area is inversely proportional to the information density; The step of generating rp random masks with random shapes and positions in each mask sample comprises the following steps: Randomly generating a first random number in a first range, wherein the first random number is a positive integer, the minimum value of the first range is 1, and the maximum value of the first range is 5; Randomly generating a second random number and a third random number in the width and height ranges of the mask sample; the second random number is a positive integer or a decimal number, and the third random number is a positive integer or a decimal number; Generating a mask area corresponding to the first random number by taking the second random number as a central horizontal coordinate and the third random number as a central vertical coordinate; the mask area is circular, elliptical, rectangular, triangular or star-shaped.

2. The PCB defect cross-domain collaborative detection method based on a multi-modal model according to claim 1, characterized in that, The step of processing the X-ray images and the optical images by using a Vision Transformer model to obtain visual features comprises the following steps: The visual features are extracted by using the following formula: ； ； ； ； ； ； Wherein, PatchEmbed is image patch embedding, CLS is classification label, represents splicing operation, PosEmb is position coding, MSA is multi-head self-attention mechanism, LN is layer normalization, MLP is multi-layer perception, l is the number of Transformer layers, I is the original image, X patch is an image block, X cls is a classification label block, Z0 is an initial visual feature matrix, Z l is the lth visual feature matrix, is a mapped feature matrix, V feat is a visual feature.

3. The multi-modal model based PCB defect cross-domain collaborative detection method according to claim 2, characterized in that, The step of processing production log texts by using a BERT model to obtain text features comprises the following steps: The text features are extracted by using the following formula: ； ； ； ； ； where TokenEmbed is a word embedding, SegEmbed is a segment embedding, FFN is a feed-forward neural network, T emb is an initial text feature matrix, H l-1 is an (l-1)-th text feature matrix, H l is an l-th text feature matrix; MultiHeadAttn is a multi-head attention, LN is a layer normalization, is an l-th layer normalized text matrix, T feat is the text feature.

4. The multi-modal model based PCB defect cross-domain collaborative detection method according to claim 1, characterized in that, The step of processing the X-ray images and the optical images by using a Vision Transformer model to obtain visual features comprises the following steps: Processing the X-ray images by using a Vision Transformer model to obtain image features of internal defects of a PCB; Processing the optical images by using a Vision Transformer model to obtain image features of surface defects of the PCB; The image features of the internal defects of the PCB and the image features of the surface defects of the PCB constitute the visual features.

5. The multi-modal model based PCB defect cross-domain collaborative detection method according to claim 1, characterized in that, The step of constructing a contrast learning loss function comprises the following steps: The contrast learning loss function is constructed by using the following formula: ； wherein L contrast is the contrastive learning loss function, x i is the i-th input sample, is the i-th positive sample, is the j-th negative sample, f is a feature extraction function, sim is a similarity function, τ is a temperature parameter, N is the total number of input samples, and M is the total number of negative samples.

Images