Chip surface defect detection method and device based on weak supervision, and related product

By building the main network and attention network, combining ResNet and feature fusion network, and using pixel loss function, the problem of insufficient labeling samples in chip defect detection is solved, and the accuracy of chip surface defect detection and the effectiveness of model are improved.

CN119067921APending Publication Date: 2024-12-03YIMOON(SHANGHAI) TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411079457.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the field of chip defect detection, due to the lack of labeled samples, it is difficult to build a high-accurate defect detection model. Especially in the process of chip packaging manufacturing, small packaging size and complex technology lead to chip surface defect detection efficiency and accuracy. insufficient.

Method used

Using a weak supervision-based method, by building the main network and attention network, combining the ResNet convolutional network and feature fusion network, using fine-grained annotation and data amplification, introducing pixel loss functions, generating defect detection models, and improving chip surface defect detection accuracy.

Benefits of technology

With a small number of labeled samples, the accuracy of chip defect detection and effective utilization of the model are improved, the expression ability of the convolutional network is enhanced, and the defects on the chip surface can be effectively identified.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067921A_ABST
    Figure CN119067921A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a chip surface defect detection method and device based on weak supervision and a related product. The method comprises the following steps: acquiring sample data of a chip surface; performing fine-grained labeling on the positive sample data, wherein the fine-grained labeling comprises a defect category identifier of the chip, a defect part labeling point, an attribute of the defect part labeling point and labeling box information; randomly selecting a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a verification set and a test set; constructing a network model based on a main network and an attention network; wherein each of the main network and the attention network comprises a ResNet convolutional network and a feature fusion network; the total loss function of the network model comprises main network loss, attention network loss and pixel loss; training the network model through the training set to generate a defect detection model; performing parameter adjustment and test on the defect detection model through the verification set and the test set, and if the performance meets the preset requirement, completing the construction of the defect detection model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of defect detection technology, and in particular to a chip surface defect detection method and device based on weak supervision, and related products. Background Art

[0002] In recent years, the demand for chips has been growing, requiring higher efficiency and accuracy in chip detection. However, in the chip packaging and manufacturing process, the package size is getting smaller and smaller. Many factors such as technical complexity, process parameters, equipment performance, materials, and environment often lead to defects on the chip surface, affecting product yield. The more common defects include package defects, surface printing defects, and pin defects.

[0003] At present, image recognition methods based on convolutional neural networks usually require massive amounts of training data and sufficient iterations to accurately classify specific image categories into subcategories. However, in the field of chip defect detection, it is not possible to obtain a large number of labeled samples, and the existing deep learning network is not sufficient to train a good deep neural network. How to build a network model to achieve high accuracy in chip defect detection using few labeled samples has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide a chip surface defect detection method and device based on weak supervision and related products to improve the accuracy of chip defect detection in view of the defects in the prior art.

[0005] To achieve the above object, the first aspect of the present invention provides a chip surface defect detection method based on weak supervision, the method comprising:

[0006] Acquiring sample data on the surface of the chip, wherein the sample data includes positive sample data and negative sample data;

[0007] Fine-grained annotation is performed on the positive sample data, wherein the fine-grained annotation includes the defect category identification of the chip, the defect location annotation points and their attributes, and the annotation box information;

[0008] Randomly select a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a validation set, and a test set;

[0009] Constructing a network model based on a main network and an attention network; wherein the main network and the attention network both include a ResNet convolutional network and a feature fusion network; the total loss function of the network model includes a main network loss, an attention network loss, and a pixel loss;

[0010] Training the network model using the training set to generate a defect detection model;

[0011] The defect detection model is adjusted and tested using the validation set and the test set respectively. If the performance meets the preset requirements, the defect detection model is constructed.

[0012] Furthermore, before fine-grained labeling of the positive sample data, the method further includes:

[0013] The sample data is preprocessed by data amplification to generate sample data including amplified data.

[0014] Furthermore, the method of randomly selecting a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set is specifically as follows:

[0015] The amplified data and the original data in the sample data are randomly combined, and the amplified data and the original data in the training set generated in each iteration are selected according to a preset ratio, and the preset ratio is different for each iteration, and the total number of images is a fixed value.

[0016] Further, the network model is trained by the training set to generate a defect detection model, specifically:

[0017] The main network uses a ResNet convolutional network to extract the main network feature map of the image, and passes the main network feature map through a feature fusion network to generate overall fine-grained feature data;

[0018] By means of weakly supervised learning, the main network feature graph is sorted and significant features are selected and put into the attention network to generate detailed fine-grained feature data;

[0019] The overall fine-grained feature data and the detailed fine-grained feature data are sent to softmax for joint training to generate a defect detection model.

[0020] Furthermore, the feature fusion network is a dual-channel fusion network; the dual-channel fusion network includes a first convolution kernel and a second convolution kernel, the first convolution kernel is a 1×1×n convolution layer, and the second convolution kernel is a 1×3×m and 3×1×m convolution layer.

[0021] Furthermore, the pixel loss formula is as follows:

[0022]

[0023] Among them, Loss pixel is the pixel loss, A is the feature area obtained by segmentation, and i, j are the image positions of the feature pixels.

[0024] Furthermore, the defect category identification includes chip package damage, pin loss, pin deformation, surface scratches, surface contamination, pin short circuit and abnormal pin spacing.

[0025] A second aspect of the present invention provides a chip surface defect detection device based on weak supervision, comprising:

[0026] A sample acquisition module, used to obtain sample data on the chip surface, wherein the sample data includes positive sample data and negative sample data;

[0027] A first data processing module is used to perform fine-grained annotation on the positive sample data, wherein the fine-grained annotation includes the defect category identification of the chip, the defect location annotation points and their attributes, and the annotation box information;

[0028] A second data processing module is used to randomly select a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a validation set and a test set;

[0029] A model building module, used to build a network model based on a main network and an attention network; wherein the main network and the attention network both include a ResNet convolutional network and a feature fusion network; the total loss function of the network model includes a main network loss, an attention network loss and a pixel loss;

[0030] A model training module, used to train the network model through the training set to generate a defect detection model;

[0031] The model testing and verification module is used to adjust parameters and test the defect detection model through the verification set and the test set respectively. If the performance meets the preset requirements, the defect detection model is constructed.

[0032] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0033] A fourth aspect of the present invention provides a controller, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0034] The chip surface defect detection method and device based on weak supervision, and related products provided by the embodiments of the present invention improve the accuracy of chip defect detection by constructing a main network including a feature fusion network and an attention network for cooperative training and sharing training parameters; pixel loss is added to the total loss function, and a pixel-level comparison detection mechanism is introduced when there are only a small number of labeled samples, so that the feature area of ​​concern can be statistically analyzed with the known feature area data, thereby improving the effective utilization of the training set. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1One of the flow charts of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention;

[0036] Figure 2 The second flowchart of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention;

[0037] Figure 3 The overall structure diagram of the network model of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention;

[0038] Figure 4 A main network structure diagram of a chip surface defect detection method based on weak supervision provided in Embodiment 1 of the present invention;

[0039] Figure 5 (a) is one of the feature fusion network structure diagrams of the chip surface defect detection method based on weak supervision provided in Example 1 of the present invention;

[0040] Figure 5 (b) is a second feature fusion network structure diagram of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention;

[0041] Figure 5 (c) is a third feature fusion network structure diagram of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention;

[0042] Figure 6 A schematic diagram of the structure of a chip surface defect detection device based on weak supervision provided in Embodiment 2 of the present invention;

[0043] Figure 7 This is a schematic diagram of the controller structure provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0046] The chip surface defect detection method and device based on weak supervision, and related products provided by the embodiments of the present invention improve the accuracy of chip defect detection by constructing a main network including a feature fusion network and an attention network for cooperative training and sharing training parameters; adding pixel loss to the total loss function, and introducing a pixel-level contrast detection mechanism when there are only a small number of labeled samples, which can statistically analyze the feature area of ​​concern and the known feature area data, thereby improving the effective utilization of the training set; and modeling the linear correlation between different channels by linearly fusing the features of different channels, thereby enhancing the expression ability of the convolutional network.

[0047] Embodiment 1

[0048] Figure 1 One of the flowcharts of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0049] Step 110 , obtaining sample data on the chip surface, where the sample data includes positive sample data and negative sample data.

[0050] Specifically, a small amount of positive sample data and a large amount of negative sample data are obtained through network collection or manual photography. Network collection can use a web crawler to grab data from the Internet. The positive sample data is sample data with chip surface defects and has been accurately labeled, and the negative sample data is sample data without chip surface defects and does not need to be labeled.

[0051] Step 120, fine-grained annotation is performed on the positive sample data, and the fine-grained annotation includes the chip defect category identification, defect location annotation points and their attributes and annotation box information. Optionally, the defect category identification includes chip package damage, pin loss, pin deformation, surface scratches, surface contamination, pin short circuit and pin spacing abnormality, etc.

[0052] In an alternative approach, Figure 2 The second flowchart of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention is as follows: Figure 2 As shown, before step 120, the following steps are also included:

[0053] Step 210: perform data amplification preprocessing on the sample data to generate sample data including amplified data.

[0054] Among them, data augmentation is to crop, rotate, scale, flip and other processes on the basis of the original sample data. The generated augmented data can increase the total amount of sample data and improve the robustness and generalization ability of the model.

[0055] Step 130 , randomly select a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a validation set, and a test set.

[0056] Specifically, in combination with step 120, a preset number of sample data is randomly selected from the labeled positive sample data and negative sample data to generate a training set, specifically: the amplified data and the original data in the sample data are randomly combined, and the amplified data and the original data in the training set generated in each iteration are selected according to a preset ratio, and the preset ratio is different for each iteration, and the total number of pictures is a fixed value. When the preset ratio of each iteration is different, the data set used for each round of training is not exactly the same, and the total number of pictures is a fixed value to facilitate calculation time, making the model more accurate.

[0057] Step 140, construct a network model based on the main network and the attention network; wherein the main network and the attention network both include a ResNet convolutional network and a feature fusion network; the total loss function of the network model includes the main network loss, the attention network loss and the pixel loss. Among them, the feature fusion network is a dual-channel fusion network, and the dual-channel fusion network includes a first convolution kernel and a second convolution kernel.

[0058] In a possible implementation A, Figure 5 As shown in (a), the first convolution kernel is a 1×1×m convolution layer, and the second convolution kernel is a 1×1×n convolution layer.

[0059] In a possible implementation B, if Figure 5 As shown in (b), the first convolution kernel is a 1×1×m convolution layer, and the second convolution kernel is a 3×3×n convolution layer with a padding value of 1.

[0060] In a possible implementation C, Figure 5 As shown in (c), the first convolution kernel is a 1×1×n convolution layer, and the second convolution kernel is a 1×3×m and 3×1×m convolution layer.

[0061] Among them, the aggregation of method A is too single, and the fused feature expression ability is weak. Method B needs to add additional padding values ​​to maintain the unchanged feature size. The additional padding values ​​will add unnecessary information, which is not conducive to the linear fusion of features. Method C combines methods A and B and has better expression ability for feature fusion. Feature fusion in a single main network branch can achieve the highest accuracy compared with methods A and B, and can also better calculate the response of the feature map in a certain area.

[0062] Step 150: Train the network model using the training set to generate a defect detection model.

[0063] In a possible implementation, Figure 3As shown, step 150 specifically includes steps A1-A3,

[0064] In step A1, the main network uses the ResNet convolutional network to extract the main network feature map of the image, and passes the main network feature map through the feature fusion network to generate overall fine-grained feature data.

[0065] Among them, the feature fusion network is a dual-channel feature fusion network, including two feature extractors, which are respectively composed of Conv1 and Conv2. The main network feature map is multiplied by the outputs of two different feature extractors using external products to obtain the feature representation of the image.

[0066] Figure 4 The main network structure diagram of the chip surface defect detection method based on weak supervision provided in the first embodiment of the present invention is as follows: Figure 4 As shown, the specific training process of the main network is shown in the following formulas (1) and (2):

[0067] X = BaseNet(I) (1)

[0068] Among them, I represents the sample data in the input training set, and X represents the feature map extracted by the convolutional network. Figure 4 The f in the equation represents the feature matrix after the feature map X is expanded through the convolution layer Conv1. m represents the number of channels in the feature map, K represents the number of features contained in the feature map, and f' represents the feature matrix obtained by expanding the input feature map X through the convolution layer Conv2. n represents the number of channels of the feature map.

[0069] The feature representation of the image obtained by performing the outer product operation on f and f' is as follows:

[0070]

[0071] Where G is the outer product matrix, and x i represents the i-th row in f, that is, the i-th channel in the feature map, i∈1,2,…,m,x′ j represents the jth row in f′, that is, the jth channel in the feature map, j∈1,2,…,n. It can be seen from formula (2) that each element in the matrix G obtained by the outer product is the inner product between channels, so that the linear relationship between channels in the feature map can be captured.

[0072] In a possible implementation, in order to make the feature matrix have a better distribution, the features are normalized, as shown in formula (3), formula (4) and formula (5):

[0073] g=vec(G) (3)

[0074]

[0075] c main =s / ||s|| (5)

[0076] Among them, g is the outer product matrix expansion vector, s is the signed square root regularization calculation of g, C main is the overall fine-grained feature data obtained after normalizing s.

[0077] In step A2, the main network feature map is sorted and significant features are selected and put into the attention network through weakly supervised learning to generate detailed fine-grained feature data.

[0078] Specifically, as the number of training times increases, the perception of features obtained by convolution of sample data in the input training set also changes with the depth of the network, and the network can gradually locate the position of the focus area. Assume that the image is passed through a series of convolutional layers and pooling layers to obtain a feature map of size C×H×W, and then each C×1×1 cross-channel vector on the feature map can represent the corresponding position in the original image at a fixed spatial position as the number of training times increases. Based on the characteristic that classification neural networks can automatically focus on the objects of interest in the image, the attention mechanism network can feed back the focus position information to the original image. The location information of the salient features is learned through category information, thereby realizing weakly supervised learning of the network.

[0079] By calculating the masks of different channels of the feature map and further filtering the image channels, the spatial position of the fine-grained object can be obtained. Specifically, as shown in formula (6) to formula (11):

[0080] p1, p2,…,pn=BaseNet(I) (6)

[0081] Among them, n feature maps p1, p2, ..., pn (i.e., n channels) can be obtained through the main network branch.

[0082] q i =mean(p i ), i∈(0,n) (7)

[0083]

[0084] Among them, the n channels are averaged respectively, and the relative weight of each channel is obtained in turn, which is recorded as W i , i∈(0,n).

[0085] mask=w1+w2+…+w m (9)

[0086] Among them, Wi The larger the value, the more significant the feature of the channel is. By extracting the first m weights and obtaining the attention mask, we can obtain more fine-grained significant features in the significant objects.

[0087] I att =I·mask (10)

[0088] After extracting the location information of multiple most significant features from the feature map, the foreground object is extracted from the original image through mask operation, which is denoted as I att .

[0089] c att =conv(I att ) (11)

[0090] Among them, the extracted more significant features are put into the attention network training to generate detailed fine-grained feature data C att As shown in Figure (3), the attention network includes a feature fusion network, which is a dual-channel feature fusion network, including two feature extractors, respectively composed of Conv1 and Conv2. Its data processing method is the same as that of the feature extractor in the main network, which will not be repeated here.

[0091] In step A3, the overall fine-grained feature data and the detailed fine-grained feature data are sent to softmax for joint training to generate a defect detection model.

[0092] Specifically, the total loss function of the network model includes the main network loss, the attention network loss and the pixel loss. The specific calculation is shown in formula (12):

[0093] Loss(x, I) = λ main Loss main (x,c main )+λ att Loss att (x,c att )+λ pixel Loss pixel (12)

[0094] Among them, x represents the image category information, C main Represents the overall fine-grained feature data of the main network, C att Represents the detailed fine-grained feature data of the attention network, λ main , att and λ pixelis a hyperparameter used to control the weight of the corresponding loss term. Since the training set contains a large amount of unlabeled data, the introduction of a pixel-level comparison detection mechanism can count the feature area of ​​interest and the data of the known feature area to improve the effective utilization of the training set.

[0095] The pixel loss is shown in formula (13):

[0096]

[0097] Among them, Losspixe l is the pixel loss, A is the feature area obtained by segmentation, and i, j are the image positions of the feature pixels.

[0098] The total loss function combines the loss function of the main network and the loss function of the attention network, and jointly trains the parameters to ensure that the network has a good expression ability for the overall image category and discriminative features.

[0099] The classification loss uses the Softmax function. The Softmax loss calculation is shown in formula (14) and formula (15):

[0100] Loss(p,q)=-Σp(x i )log2(q(x i )) (14)

[0101]

[0102] Among them, p(x i ) is x i target value.

[0103] Step 160, the defect detection model is adjusted and tested using the validation set and the test set respectively. If the performance meets the preset requirements, the defect detection model is constructed.

[0104] Specifically, when performing chip surface defect detection through the constructed defect detection model, the chip with surface defects can be marked with a marking box. For different defect types, they can be optionally classified and displayed on the chip image using marking boxes of different colors; or they can be displayed by outputting the chip position and the ID corresponding to the chip defect category identification.

[0105] Embodiment 2

[0106] A second embodiment of the present invention provides a chip surface defect detection device based on weak supervision. Figure 6 The schematic diagram of the structure of the chip surface defect detection device based on weak supervision provided in the second embodiment of the present invention is as follows: Figure 6As shown, the chip surface defect detection device 300 based on weak supervision includes: a sample collection module 301, a first data processing module 302, a second data processing module 303, a model construction module 304, a model training module 305 and a model testing and verification module 306.

[0107] The sample acquisition module 301 is used to obtain sample data on the chip surface, where the sample data includes positive sample data and negative sample data.

[0108] The first data processing module 302 is used to perform fine-grained annotation on the positive sample data, where the fine-grained annotation includes the defect category identification of the chip, the defect location annotation points and their attributes, and the annotation box information.

[0109] Among them, defect category identification includes chip package damage, pin missing, pin deformation, surface scratches, surface contamination, pin short circuit and abnormal pin spacing.

[0110] The second data processing module 303 is used to randomly select a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a validation set and a test set.

[0111] The model building module 304 is used to build a network model based on the main network and the attention network; wherein the main network and the attention network both include a ResNet convolutional network and a feature fusion network; and the total loss function of the network model includes the main network loss, the attention network loss and the pixel loss.

[0112] Among them, the feature fusion network is a dual-channel fusion network; the dual-channel fusion network includes a first convolution kernel and a second convolution kernel, the first convolution kernel is a 1×1×n convolution layer, and the second convolution kernel is a 1×3×m and 3×1×m convolution layer.

[0113] The pixel loss formula is shown in formula (16):

[0114]

[0115] Among them, Loss pixel is the pixel loss, A is the feature area obtained by segmentation, and i, j are the image positions of the feature pixels.

[0116] The model training module 305 is used to train the network model through the training set to generate a defect detection model.

[0117] Among them, the network model is trained through the training set to generate a defect detection model, specifically:

[0118] The main network uses the ResNet convolutional network to extract the main network feature map of the image, and passes the main network feature map through the feature fusion network to generate overall fine-grained feature data.

[0119] Through weakly supervised learning, the main network feature map is sorted and significant features are filtered and put into the attention network to generate detailed fine-grained feature data.

[0120] The overall fine-grained feature data and the detailed fine-grained feature data are sent to softmax for joint training to generate a defect detection model.

[0121] The model testing and verification module 306 is used to adjust the parameters and test the defect detection model through the verification set and the test set respectively. If the performance meets the preset requirements, the defect detection model is constructed.

[0122] In a possible solution, the first data processing module 302 is further configured to perform data amplification preprocessing on the sample data to generate sample data including amplified data.

[0123] Among them, a preset number of sample data are randomly selected from the labeled positive sample data and negative sample data to generate a training set, specifically:

[0124] The amplified data and original data in the sample data are randomly combined. The amplified data and original data in the training set generated in each iteration are selected according to a preset ratio. The preset ratio is different for each iteration, and the total number of images is a fixed value.

[0125] Embodiment 3

[0126] Embodiment 3 of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any one of the chip surface defect detection methods based on weak supervision provided in Embodiment 1.

[0127] Embodiment 4

[0128] A fourth embodiment of the present invention provides a controller, Figure 7 This is a schematic diagram of the controller structure provided by the fourth embodiment of the present invention. Figure 7 As shown, it includes a memory 400, a processor 500 and a computer program stored in the memory, and the processor executes the computer program to implement any one of the chip surface defect detection methods based on weak supervision provided in the above-mentioned embodiment 1.

[0129] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0130] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0131] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A chip surface defect detection method based on weak supervision, characterized in that: The method comprises: Acquiring sample data on the surface of the chip, wherein the sample data includes positive sample data and negative sample data; Fine-grained annotation is performed on the positive sample data, wherein the fine-grained annotation includes the defect category identification of the chip, the defect location annotation points and their attributes, and the annotation box information; Randomly select a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a validation set, and a test set; Constructing a network model based on a main network and an attention network; wherein the main network and the attention network both include a ResNet convolutional network and a feature fusion network; the total loss function of the network model includes a main network loss, an attention network loss, and a pixel loss; Training the network model using the training set to generate a defect detection model; The defect detection model is adjusted and tested using the validation set and the test set respectively. If the performance meets the preset requirements, the defect detection model is constructed.

2. The chip surface defect detection method based on weak supervision according to claim 1 is characterized in that: Before fine-grained labeling of the positive sample data, the method further includes: The sample data is preprocessed by data amplification to generate sample data including amplified data.

3. The chip surface defect detection method based on weak supervision according to claim 2 is characterized in that: The method of randomly selecting a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set is specifically as follows: The amplified data and the original data in the sample data are randomly combined, and the amplified data and the original data in the training set generated in each iteration are selected according to a preset ratio, and the preset ratio is different for each iteration, and the total number of images is a fixed value.

4. The chip surface defect detection method based on weak supervision according to claim 1 is characterized in that: The network model is trained by the training set to generate a defect detection model, specifically: The main network uses a ResNet convolutional network to extract the main network feature map of the image, and passes the main network feature map through a feature fusion network to generate overall fine-grained feature data; By means of weakly supervised learning, the main network feature graph is sorted and significant features are selected and put into the attention network to generate detailed fine-grained feature data; The overall fine-grained feature data and the detailed fine-grained feature data are sent to softmax for joint training to generate a defect detection model.

5. The chip surface defect detection method based on weak supervision according to claim 1 is characterized in that: The feature fusion network is a dual-channel fusion network; the dual-channel fusion network includes a first convolution kernel and a second convolution kernel, the first convolution kernel is a 1×1×n convolution layer, and the second convolution kernel is a 1×3×m and 3×1×m convolution layer.

6. The chip surface defect detection method based on weak supervision according to claim 1 is characterized in that: The pixel loss formula is as follows: Among them, Loss pixel is the pixel loss, A is the feature area obtained by segmentation, and i, j are the image positions of the feature pixels.

7. The chip surface defect detection method based on weak supervision according to claim 1 is characterized in that: The defect category identification includes chip package damage, pin missing, pin deformation, surface scratches, surface contamination, pin short circuit and abnormal pin spacing.

8. A chip surface defect detection device based on weak supervision, characterized in that: The device comprises: A sample acquisition module, used to obtain sample data on the chip surface, wherein the sample data includes positive sample data and negative sample data; A first data processing module is used to perform fine-grained annotation on the positive sample data, wherein the fine-grained annotation includes the defect category identification of the chip, the defect location annotation points and their attributes, and the annotation box information; A second data processing module is used to randomly select a preset number of sample data from the labeled positive sample data and negative sample data to generate a training set, a validation set and a test set; A model building module, used to build a network model based on a main network and an attention network; wherein the main network and the attention network both include a ResNet convolutional network and a feature fusion network; the total loss function of the network model includes a main network loss, an attention network loss, and a pixel loss; A model training module, used to train the network model through the training set to generate a defect detection model; The model testing and verification module is used to adjust parameters and test the defect detection model through the verification set and the test set respectively. If the performance meets the preset requirements, the defect detection model is constructed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claims 1 to 8 are implemented.

10. A controller comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claims 1-8.

Citation Information

Cited By

  • Chip advanced packaging defect detection method and system based on mmmember detection framework

    CN121391822A