Surface defect detection method of chip ceramic packaging substrate based on multimodal large model

Abstract

The present invention provides a chip ceramic package substrate surface defect detection method based on a multimodal large model, which relates to the technical field of ceramic chip surface defects. Multimodal data fusion and cross-domain knowledge migration are used to realize image and text data fusion of chip ceramic package substrate surface defects. Similarity and segmentation loss functions are used comprehensively for constraint optimization to realize efficient and accurate detection of surface defects of package substrates of different product types, further improve the generalization and robustness of the detection method in different scenarios, and combine visual text prompts to form a complete defect analysis report.

Description

Technical Field

[0001] The invention relates to the technical field of ceramic chip surface defects, in particular to a chip ceramic package substrate surface defect detection method based on a multi-modal large model. Background Art

[0002] The packaging substrate can provide electrical connection, protection, support, heat dissipation and assembly for the chip, so as to achieve multi-pin, reduce the size of the packaged product, improve electrical performance and heat dissipation, ultra-high density or multi-chip modularization. Ceramic packaging substrate is an electronic component packaging material made of ceramic materials. It has the advantages of high thermal conductivity, good heat resistance, high mechanical strength, low thermal expansion coefficient, etc. It is a commonly used substrate material for digital packaging.

[0003] However, due to the low level of manufacturing technology, surface defects such as open circuit, breakage, rat bite, and overflow may appear on the ceramic package substrate, which poses a threat to the quality of subsequent products. In order to achieve efficient and accurate detection of surface defects of chip ceramic package substrates, deep learning models are usually used as the basic technical means, and trained with the help of a large amount of finely labeled image data to ensure the accuracy of the algorithm.

[0004] The technical solution described in the Chinese patent "CN117474928B A method for detecting surface defects of ceramic package substrates based on a meta-learning model" proposes a method for detecting surface defects of chip ceramic package substrates based on a meta-learning model to solve the problem that existing detection methods have large data requirements and are difficult to generalize. This method divides the data into a support set and a query set, extracts image features through a backbone network, uses a matrix decomposition method to filter irrelevant background features, uses a prototype selection module to complete the collection of similar features, and uses a boundary optimization module to complete feature optimization. Finally, a lightweight decoder is used to achieve pixel-level prediction of surface defects. The technical solution described in the Chinese patent "CN118464827A A FCBGA package substrate detection device and detection method" proposes a FCBGA package substrate detection device, which fixes the FCBGA package substrate on the detection device using a vacuum adsorption device so as not to damage the FCBGA package substrate, and uses an infrared spectroscopy device to emit infrared light to the area to be detected of the FCBGA package substrate. By using the different properties of infrared spectral characteristics of different materials, it can be determined whether there is any photosensitive film residue in the window area of ​​the FCBGA package substrate based on the comparison between the infrared spectral characteristics of the material and a preset spectral library.

[0005] The technical solution described in the Chinese patent "CN117474928B A method for detecting surface defects of ceramic packaging substrates based on a meta-learning model" constructs a matrix decomposition mechanism through a meta-learning model to filter irrelevant background features, and uses a prototype selection module to complete the collection of similar features. If the feature distribution of the image background and the defect foreground area is very similar, the method will cause misidentification; if the number of defect categories gradually increases, the generalization and robustness of the method will decrease as the number of categories increases, and multiple repeated training is required.

[0006] The Chinese patent "CN118464827A FCBGA package substrate detection device and detection method" compares the infrared spectrum characteristics with the preset spectrum library to determine whether there is photosensitive film residue on the FCBGA package substrate. If the material and type of the product change, the values ​​of the infrared spectrum and the preset spectrum need to be reset. The algorithm has poor generalization and is difficult to cover multiple categories of detection scenarios. Summary of the invention

[0007] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a chip ceramic package substrate surface defect detection method based on a multi-modal large model, comprising:

[0008] Step 1: Collect multiple image data of the surface of the chip ceramic package substrate, and obtain text data corresponding to each image data. The multiple image data of the surface of the chip ceramic package substrate and the text data corresponding to each image data constitute a data set, which is specifically expressed by the following formula:

[0009] ;

[0010] in, represents a data set, is the total number of image data, Indicates image data, where , Indicates image data, Indicates Text data;

[0011] Step 2: Encode each of the image data to obtain a plurality of image features, and encode each of the text data to obtain a text feature;

[0012] Step 3: Through the multimodal fusion strategy, each image feature is fused with the text feature to obtain multiple fusion features, which is specifically achieved through the following formula:

[0013] ;

[0014] in, Indicates the first The fusion characteristics of each stage, represents a 1×1 convolutional layer, represents the feature concatenation operation, Indicates the first j The image features of each stage, Represents text features;

[0015] Step 4: Decode all fused features through a lightweight decoder to obtain pixel-level prediction results, which are specifically expressed by the following formula:

[0016] ;

[0017] in, This is the pixel-level prediction result of the surface defects of the chip ceramic package substrate. It is a lightweight decoder. It is the feature pixel level addition operation.

[0018] Optionally, step 2 specifically includes:

[0019] Step 2.1: For each of the image data, encode it through the ViT-B encoder to obtain image features, which is specifically implemented by the following formula:

[0020] ;

[0021] in, Indicates the ViT-B encoder, is the image feature space domain, B Indicates batch size, H Indicates the length of the image data. W Indicates the width of the image data, Indicates the number of channels of image data;

[0022] Step 2.2: For each of the text data, encode it through the CLIP encoder to obtain a text feature of a fixed dimension, which is specifically implemented by the following formula:

[0023] = ;

[0024] in, represents the CLIP encoder, is the text feature space domain.

[0025] Optionally, also include:

[0026] According to the pixel-level prediction results of each fused feature, text feature and chip ceramic package substrate surface defect, the parameters in the ViT-B encoder, CLIP encoder and lightweight decoder are updated and optimized, including the following steps:

[0027] Step A1: after obtaining each fusion feature and text feature in step 3, determine a similarity loss function according to each fusion feature and text feature;

[0028] Step A2: after obtaining the pixel-level prediction results of the surface defects of the chip ceramic package substrate in step 4, determining the segmentation loss function according to the pixel-level prediction results;

[0029] Step A3: According to the similarity loss function and the segmentation loss function, a final loss function is obtained, and the final loss function is used to update and optimize the parameters in the ViT-B encoder, the CLIP encoder and the lightweight decoder.

[0030] Optionally, step A1 specifically includes:

[0031] Step A1.1: Through the cross-domain knowledge transfer strategy, calculate the feature distribution similarity between each fusion feature and the text feature in the latent space, which is specifically achieved through the following formula:

[0032] ;

[0033] in, Representing Text Features and Image Encoders The corresponding j The similarity of the feature distribution of image features in the latent space at each stage, represents the dot product between eigenvectors, represents the bi-norm of the feature;

[0034] Step A1.2: Determine a similarity loss function according to the feature distribution similarity, which is specifically implemented by the following formula:

[0035] .

[0036] Optionally, the segmentation loss function in step A2 is expressed by the following formula:

[0037] ;

[0038] in, For the i The pixel-level annotation information corresponding to the image data.

[0039] Optionally, the final loss function in step A3 is expressed by the following formula:

[0040] ;

[0041] in, is the hyperparameter of the similarity loss function.

[0042] Optionally, after step 4, the following steps may be performed:

[0043] Step 5: Perform binary threshold extraction on the pixel-level prediction results to obtain the coordinates and size information of each connected domain, which is specifically achieved through the following formula:

[0044] ;

[0045] in, is the coordinate information of the kth defect connected domain in the pixel-level prediction result, is the size information of the kth defect connected domain in the pixel-level prediction result, It is a binary extraction operation.

[0046] Optionally, after step 5, the following steps may be performed:

[0047] Step 6: Generate a defect analysis report based on the total number of connected domains in the image data, the coordinate information of the kth defect connected domain in the pixel-level prediction results corresponding to the image data, the size information of the kth defect connected domain in the pixel-level prediction results corresponding to the image data, and the pixel-level prediction results.

[0048] The beneficial effects of adopting the above technical solution are:

[0049] The purpose of the present invention is to provide a surface defect detection method for chip ceramic package substrates based on a multimodal large model. By performing multimodal data fusion on image data and text data, establishing associations between cross-modal data, and realizing feature generalization between different domains through cross-domain knowledge migration, the detection speed is effectively improved. Finally, pixel-level segmentation is achieved using a learnable lightweight decoder. The present invention makes full use of multimodal data to solve the problem of poor generalization and robustness of existing detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the structure of the process of a chip ceramic package substrate surface defect detection method based on a multi-modal large model in an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of pixel-level prediction results of a chip ceramic package substrate surface defect detection method based on a multi-modal large model in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a complete defect analysis report in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0054] In view of the problems existing in the prior art, the present invention provides a surface defect detection method for chip ceramic package substrates based on a multimodal large model. By performing multimodal data fusion on image data and text data, establishing associations between cross-modal data, and realizing feature generalization between different domains through cross-domain knowledge transfer, the detection speed is effectively improved. Finally, pixel-level segmentation is achieved using a learnable lightweight decoder. The present invention makes full use of multimodal data to solve the problem of poor generalization and robustness of existing detection methods. Specifically, combined with Figure 1 , which may include the following steps:

[0055] Step 1: Collect multiple image data of the surface of the chip ceramic package substrate, and obtain text data corresponding to each image data. The multiple image data of the surface of the chip ceramic package substrate and the text data corresponding to each image data constitute a data set, which is specifically expressed by the following formula:

[0056] ;

[0057] in, represents a data set, is the total number of image data, Indicates image data, where , Indicates image data, Indicates Text data;

[0058] Step 2: Encode each of the image data to obtain a plurality of image features, and encode each of the text data to obtain a text feature;

[0059] Step 2.1: For each of the image data, encode it through the ViT-B encoder to obtain image features, which is specifically implemented by the following formula:

[0060] ；

[0061] in, Indicates the first j The image features of each stage, Indicates the ViT-B encoder, is the image feature space domain,B Indicates batch size, H Indicates the length of the image data. W Indicates the width of the image data, Indicates the number of channels of image data;

[0062] The stages corresponding to the ViT-B encoder are divided into four equal parts according to the 12 blocks contained in the Encoder in ViT-B. Every three blocks form one stage, that is, the ViT-B encoder has four stages. When the present invention processes each of the image data through the ViT-B encoder, the image features of stage 1 are obtained through the three blocks of stage 1. , and then the image features of stage 1 Input the three blocks of stage 2 to obtain the image features of stage 2 , and then the image features of stage 2 Input the 3 blocks of stage 3 to obtain the image features of stage 3 , and then the image features of stage 3 Input the 3 blocks of stage 4 to obtain the image features of stage 4 .

[0063] Step 2.2: For each of the text data, encode it through the CLIP encoder to obtain a text feature of a fixed dimension, which is specifically implemented by the following formula:

[0064] = ;

[0065] in, represents the CLIP encoder, is the text feature space domain, Represents text features.

[0066] Step 3: Through the multimodal fusion strategy, each image feature is fused with the text feature. The multimodal fusion strategy fuses the features of the image features and text features at each stage by a 1×1 convolution layer to obtain multiple fusion features, which is specifically implemented by the following formula:

[0067] ;

[0068] in, Indicates the first The fusion characteristics of each stage, represents a 1×1 convolutional layer, represents the feature concatenation operation, Indicates the first j The image features of each stage, Represents text features;

[0069] Step 4: Decode all fused features through a learnable lightweight decoder to obtain pixel-level prediction results, which are specifically expressed by the following formula:

[0070] ;

[0071] in, This is the pixel-level prediction result of the surface defects of the chip ceramic package substrate. It is a lightweight decoder. It is the feature pixel level addition operation.

[0072] like Figure 2 , the first and third columns are image data, and the second and fourth columns are pixel-level prediction results.

[0073] According to the pixel-level prediction results of each fused feature, text feature and chip ceramic package substrate surface defect, the parameters in the ViT-B encoder, CLIP encoder and lightweight decoder are updated and optimized, including the following steps:

[0074] Step A1: after obtaining each fusion feature and text feature in step 3, determine a similarity loss function according to each fusion feature and text feature;

[0075] Step A1 specifically includes:

[0076] Step A1.1: Through the cross-domain knowledge transfer strategy, calculate the feature distribution similarity between each fusion feature and the text feature in the latent space, which is specifically achieved through the following formula:

[0077] ;

[0078] in, Representing Text Features and Image Encoders The corresponding j The similarity of the feature distribution of image features in the latent space at each stage, represents the dot product between eigenvectors, represents the bi-norm of the feature;

[0079] Step A1.2: Apply similarity constraints to the feature distribution similarity and use the similarity loss function to Optimize it so that it gradually approaches 1. Specifically, determine the similarity loss function according to the feature distribution similarity, which is specifically implemented by the following formula:

[0080] .

[0081] in, The closer it is to 1, the smaller the angle between the image and text feature vectors, and the smaller the value of the loss function. By minimizing this loss function, paired text and images can be embedded in a high-dimensional space as close as possible to achieve cross-domain knowledge transfer.

[0082] Step A2: after obtaining the pixel-level prediction results of the surface defects of the chip ceramic package substrate in step 4, determining the segmentation loss function according to the pixel-level prediction results;

[0083] ;

[0084] in, For the i The pixel-level annotation information corresponding to the image data.

[0085] Step A3: According to the similarity loss function and the segmentation loss function, a final loss function is obtained, and the final loss function is used to update and optimize the parameters in the ViT-B encoder, the CLIP encoder and the lightweight decoder.

[0086] The final loss function in step A3 is expressed by the following formula: ;

[0087] in, is a hyperparameter of the similarity loss function, and in the present invention, the value of the hyperparameter is set to 0.2.

[0088] Step 5: Pixel-level prediction results , further construct visual prompt words. Specifically, perform binary threshold extraction on the pixel-level prediction results to obtain the coordinates and size information of each connected domain, which is specifically achieved through the following formula:

[0089] ;

[0090] in, is the coordinate information of the kth defect connected domain in the pixel-level prediction result, is the size information of the kth defect connected domain in the pixel-level prediction result, It is a binary extraction operation.

[0091] Step 6: Generate a defect analysis report based on the total number of connected domains in the image data, the coordinate information of the kth defect connected domain in the pixel-level prediction results corresponding to the image data, the size information of the kth defect connected domain in the pixel-level prediction results corresponding to the image data, and the pixel-level prediction results.

[0092] In specific implementation, the present invention can be directed to the coordinate information, size information of the connected domain, and the text data of step one. , construct multiple rounds of prompt words, where the first round of prompt words focuses on the total number of connected domains, such as "There are k defective connected domains in this image", the second round of prompt words combines coordinate information and size information, such as "The coordinates of the kth defective connected domain are located in the upper left corner of the image, and its size is 20 pixels × 20 pixels", and the last round of prompt words combines text data to form a complete defect analysis report, such as "Please help me form a complete defect analysis report based on the above answers, indicating the number, size, and location of the defects", for example Figure 3 Defect analysis report.

[0093] The chip ceramic package substrate surface defect detection method based on multimodal large model described in the present invention was tested multiple times in test data of various product types, and achieved an accuracy rate of 92.64% and an intersection-over-union ratio of 87.83%. The visualization results are shown in Figure 2. Figure 2 As shown, the defect analysis report is as follows Figure 3 As shown in the test results, the detection effect of the method of the present invention has good accuracy, good generalization and robustness in the scenario of various product types, and can be further analyzed through text, which basically meets the application requirements of surface defect detection of chip ceramic packaging substrates in actual production lines.

[0094] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure to form a technical solution.

Claims

1. A chip ceramic package substrate surface defect detection method based on a multi-modal large model, characterized in that: include: Step 1: Collect multiple image data of the surface of the chip ceramic package substrate, and obtain text data corresponding to each image data. The multiple image data of the surface of the chip ceramic package substrate and the text data corresponding to each image data constitute a data set, which is specifically expressed by the following formula: Where D represents the data set, n is the total number of image data, where 1≤i≤n, I i represents the i-th image data, T i Represents the i-th text data; Step 2: Encode each of the image data according to the ViT-B encoder to obtain a plurality of image features, and encode each of the text data according to the CLIP encoder to obtain a text feature; Step 3: Through the multimodal fusion strategy, each image feature is fused with the text feature to obtain multiple fusion features, which is specifically achieved through the following formula: in, represents the fused features of the jth stage corresponding to the ViT-B encoder, Conv1D represents a 1×1 convolutional layer, and Concat represents a feature concatenation operation. represents the image features of the jth stage corresponding to the ViT-B encoder, E txt Represents text features; Step 4: Decode all fused features through a lightweight decoder to obtain pixel-level prediction results, which are specifically expressed by the following formula: Among them, Mask pred,i is the pixel-level prediction result of the surface defects of the chip ceramic package substrate, Decoder is a lightweight decoder, and ADD is a feature pixel-level addition operation; The chip ceramic package substrate surface defect detection method based on the multimodal large model also includes: updating and optimizing the parameters in the ViT-B encoder, the CLIP encoder and the lightweight decoder according to each fusion feature, the text feature and the pixel-level prediction result of the chip ceramic package substrate surface defect, specifically including the following steps: Step A1: After obtaining multiple fusion features in step 3, determine a similarity loss function based on each fusion feature and text feature; Step A2: after obtaining the pixel-level prediction results of the surface defects of the chip ceramic package substrate in step 4, determining the segmentation loss function according to the pixel-level prediction results; Step A3: obtaining a final loss function according to the similarity loss function and the segmentation loss function, wherein the final loss function is used to update and optimize parameters in the ViT-B encoder, the CLIP encoder, and the lightweight decoder; Step A1 specifically includes: Step A1.1: Through the cross-domain knowledge transfer strategy, calculate the feature distribution similarity between each fusion feature and the text feature in the latent space, which is specifically achieved through the following formula: Among them, sim j Representing text features and image encoder I The feature distribution similarity of the corresponding image features of the jth stage in the latent space, <,> represents the dot product between feature vectors, and ||·|| represents the bi-norm of the feature; Step A1.2: Determine a similarity loss function according to the feature distribution similarity, which is specifically implemented by the following formula:

2. The chip ceramic package substrate surface defect detection method based on a multi-modal large model according to claim 1 is characterized in that: Step 2 specifically includes: Step 2.1: For each of the image data, encode it through the ViT-B encoder to obtain image features, which is specifically implemented by the following formula: Among them, Encoder I Indicates ViT-B encoder, R B ×H×W×C is the image feature space domain, B represents the batch size, H represents the length of the image data, W represents the width of the image data, and C represents the number of channels of the image data; Step 2.2: For each of the text data, encode it through the CLIP encoder to obtain a fixed-dimensional text feature, which is specifically implemented by the following formula: E txt ∈R B×512 =Encoder T (T i ), where Encoder T represents the CLIP encoder, R B×512 is the text feature space domain.

3. The chip ceramic package substrate surface defect detection method based on multi-modal large model according to claim 1 is characterized in that: The segmentation loss function in step A2 is expressed by the following formula: Maskg t,i is the pixel-level annotation information corresponding to the i-th image data.

4. The chip ceramic package substrate surface defect detection method based on multi-modal large model according to claim 1 is characterized in that: The final loss function in step A3 is expressed by the following formula: L = L seg +αL sim ; Among them, α is the hyperparameter of the similarity loss function.

5. The chip ceramic package substrate surface defect detection method based on multi-modal large model according to claim 1 is characterized in that: After step 4, also include: Step 5: Binarize the pixel-level prediction results and extract the threshold value to obtain the coordinates and size information of each connected domain. This is achieved by the following formula: k ,S k =θ(Mask pred,i ), where P k is the coordinate information of the kth defect connected domain in the pixel-level prediction result, S k is the size information of the kth defect connected domain in the pixel-level prediction result, and θ is the binarization extraction operation.

6. The chip ceramic package substrate surface defect detection method based on multi-modal large model according to claim 1 is characterized in that: After step 5, also include: Step 6: Generate a defect analysis report based on the total number of connected domains in the image data, the coordinate information of the kth defect connected domain in the pixel-level prediction results corresponding to the image data, the size information of the kth defect connected domain in the pixel-level prediction results corresponding to the image data, and the pixel-level prediction results.

Images