A deep learning defect detection method for integrated circuit metal packaging

Abstract

This paper provides a deep learning defect detection method for integrated circuit metal packages. The method first designs a multi-scale generative adversarial network (GAN) with a Transformer, named MST-GAN, to capture the intrinsic patterns of samples at multiple scales and generate multi-scale defect-free templates. A multi-scale weighted mask is then designed to suppress reconstruction errors in multi-scale difference images. Finally, potential defects in the thresholded image are detected online and defect locations are determined. This method addresses the problem of GANs introducing a large amount of interfering noise into the generated templates. A multi-scale weighted mask scheme is proposed to suppress reconstruction errors, a multi-scale adaptive threshold is proposed to highlight potential defects in weighted images, and an image block-based multi-scale defect assessment scheme is proposed to fully evaluate potential defects at multiple scales in the thresholded multi-scale image.

Description

Technical Field

[0001] The present invention relates to the field of industrial integrated circuit detection, and more specifically, to a deep learning defect detection method for integrated circuit metal packaging. Background Art

[0002] Integrated circuits (ICs) are core components of the information technology industry and a key force driving the new scientific and industrial revolution. They have become a strategic advantage for countries around the world in high-tech competition. ICs must be packaged for ease of use. IC packaging provides environmental protection, thermal management, mechanical stability, and electrical connectivity. Generally, packages are made of materials such as plastic, glass, ceramic, or metal, with packages made of different materials having different properties and application scenarios. Compared to packages made of other materials, metal packages offer high thermal conductivity, high strength, and high reliability, and are widely used in fields such as communications and aerospace. On actual IC metal packaging production lines, abnormal production parameters and unexpected process instabilities can cause defects such as scratches, bubbles, and stains on the surface of the IC metal packages. These defects can adversely affect the reliability and functionality of the ICs, and can even seriously affect their performance and service life. Therefore, surface defect detection is an indispensable step in the IC metal packaging production process. Currently, this inspection is often performed by trained quality inspectors, which is time-consuming, inefficient, and limits production capacity. Furthermore, prolonged inspections can fatigue workers and result in a high rate of false positives and missed detections. Therefore, the development of an automated optical inspection system is essential for detecting surface defects in IC metal packages.

[0003] Currently, IC metal package surface defect detection faces many challenges. First, the rough texture of IC metal package surfaces often results in varying contrast and interference in captured images. Furthermore, some defects share similar characteristics with the intrinsic texture of the IC metal package surface, significantly interfering with detection, especially for qualified samples. Second, most current supervised deep learning-based methods for electronic package surface defect detection require a large number of balanced samples for network training, and these samples must be manually annotated. However, due to the diverse and random nature of defects, collecting sufficient balanced samples and manually annotating them on a production line is extremely time-consuming and labor-intensive. Furthermore, most current detection methods assess defects at a single scale, while IC metal package surface defects have irregular shapes and areas, making these methods ineffective at capturing these defects at a single scale. Therefore, to address these challenges in existing metal package surface defect detection, we propose an IC metal package surface defect detection method based on a multi-scale generative adversarial network. This method can be used to determine whether an IC metal package has defects and locate the defect. This approach is crucial for improving IC metal package inspection capabilities.

[0004] Traditional computer vision-based surface defect detection methods mainly use two solutions: template matching and feature extraction + classifier. The former has very high requirements for the alignment between the image to be tested and the template image, while the latter often finds it difficult to extract effective features that can characterize tiny defects, which can easily lead to a large number of misjudgments and missed judgments.

[0005] Although some studies have applied deep learning to defect detection, practical engineering still faces problems such as difficulty in capturing defect features and dependence on a large number of defect samples and labels. For example, patent CN202110868507.9 proposes a chip surface defect detection method based on an improved network. This invention proposes a chip surface defect detection method based on an improved network, which is suitable for chip surface defect detection. This method uses a sliding window to mask the defect area image to determine the area to be repaired in the image. According to the matching block search method, a matching block of the same size and most similar to the block to be repaired is searched in the intact area of ​​the image, and the corresponding pixels in the matching block are filled into the pixel missing positions of the block to be repaired. The preprocessed image is subtracted from the repaired background image to obtain a difference image. The difference image is then binarized using the OTSU adaptive threshold method. The binarized image is median filtered to eliminate error points and obtain an accurate defect area. The defect area is segmented and input into the EfficientNet classification network for defect classification. This method involves matching search operations, which is slow; it requires a large amount of data and labels to train the classification network; and it only evaluates defects from a single scale.

[0006] Previous defect detection methods often evaluated defects at a single scale. However, the surface defect representation of IC metal packages is characterized by irregularity and diversity, and evaluation at a single scale is often insufficient. Generative adversarial network-based methods all use deconvolution for upsampling operations in the decoder, which introduces a large amount of noise into the reconstructed template, seriously interfering with defect detection.

[0007] The prior art discloses a patent for a chip surface defect detection method based on an improved network. This patent proposes a chip surface defect detection method based on an improved network, which is suitable for chip surface defect detection. This method uses a sliding window to mask the defect area image to determine the image area to be repaired. According to the matching block search method, a matching block of the same size and most similar to the block to be repaired is searched in the intact area of ​​the image. The corresponding pixels in the matching block are filled into the pixel missing positions of the block to be repaired. The preprocessed image is subtracted from the repaired background image to obtain a difference image. The difference image is then binarized using the OTSU adaptive threshold method. The binarized image is then subjected to median filtering to eliminate error points and obtain an accurate defect area. The defect area is segmented and input into the EfficientNet classification network for defect classification. This method involves a matching search operation, which is slow; it requires a large amount of data and labels to train the classification network; and it only evaluates defects at a single scale. A patent for a chip surface defect detection method based on a convolutional denoising autoencoder is also disclosed in the prior art. This patent proposes a chip surface defect detection method based on a convolutional denoising autoencoder, aiming to effectively detect weak defects and thus improve the accuracy of chip surface defect detection. The method first captures a defect-free image of the chip surface and performs image size normalization and Gaussian smoothing denoising preprocessing before segmenting it. The segmented image is then added with Gaussian noise for network training. The chip surface image to be inspected is then size normalized and Gaussian smoothing denoised preprocessing to obtain the preprocessed image. This image is then overlapped and segmented and input into the trained network for residual image reconstruction to achieve defect detection. This method only adds Gaussian noise, but the IC metal surface texture has very complex features, so this convolutional denoising autoencoder cannot effectively generate a residual image. For qualified samples, the residual image generated by the network will contain a large amount of noise, which cannot be processed by simple threshold segmentation. Defects are only evaluated at a single scale. Summary of the Invention

[0008] The present invention provides a deep learning defect detection method for integrated circuit metal packages, which avoids the use of deconvolution and can effectively suppress interfering pixels in the reconstructed template.

[0009] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0010] A method for detecting defects in integrated circuit metal packages using deep learning, comprising the following steps:

[0011] S1: Design a multi-scale generative adversarial network (GAN) with a Transformer, named MST-GAN, to capture the intrinsic patterns of samples at multiple scales, generate multi-scale defect-free templates, and design a multi-scale loss function to ensure network convergence;

[0012] S2: Using the input sample and the multi-scale defect-free template obtained in step S1, the multi-scale difference image is obtained by performing element-wise subtraction and then taking the absolute value. A multi-scale weight mask is designed to suppress the reconstruction error in the multi-scale difference image.

[0013] S3: Design a multi-scale adaptive threshold, process the multi-scale difference image obtained after the multi-scale weight mask processing in S2 to obtain a threshold image, and design a multi-scale defect evaluation scheme MIPDE based on image blocks to detect potential defects in the threshold image and obtain defect location.

[0014] Furthermore, the specific process of step S1 is:

[0015] S11: Process the collected original IC metal package surface image and use binarization, Canny, and digital morphology to obtain the region of interest (ROI). This ROI occupies the main area of ​​the IC metal package surface. The ROI is segmented and divided into qualified samples and unqualified samples to create training and test sets. The image pixels are normalized so that the image pixel values ​​are between [-1, 1], and the model parameters are initialized.

[0016] S12: Construct a multi-scale generative adversarial network (GAN) with a Transformer. The network consists of a generator (G), a discriminator (D), and a multi-scale encoder. The generator consists of a multi-scale CNN encoder (MSCE), a cross-scale feature fusion module (CSFF), a Swin Transformer decoder (STD), and a multi-scale template generation module (MSTG). Both the discriminator and the multi-scale encoder consist of a multi-scale CNN encoder (MSCE) and a cross-scale feature fusion module (CSFF).

[0017] The multi-level feature maps of each scale extracted by the multi-scale CNN encoder MSCE are fed into the cross-scale feature fusion module CSFF;

[0018] The cross-scale feature fusion module CSFF combines high-level semantic information with low-level spatial information at different scales, that is, it fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE;

[0019] The Swin Transformer decoder STD uses the fused high-level feature maps obtained by the cross-scale feature fusion module CSFF module to reconstruct the template features;

[0020] The multi-scale template generation module MSTG reconstructs multi-scale defect-free templates;

[0021] S13: Construct a multi-scale loss function to ensure network convergence.

[0022] Furthermore, in step S12, the multi-scale CNN encoder MSCE performs multi-scale feature extraction: three multi-scale feature maps with the same structure are extracted from the small-scale, medium-scale and large-scale inputs respectively, and the feature map of each scale represents the low-level, medium-level and high-level features of the corresponding input. Let the large-scale, medium-scale and small-scale inputs be I x , x = l, m, s, the feature map of the corresponding scale is:

[0023]

[0024] where n x The values ​​of (x=l,m,s) are 3, 2 and 1 respectively, Indicates that the input Ix is passed from the first ResNetBottleneck to the nth x After blocks, the nth x The output of the block, m x,y (x=l,m,s; y=hi,mi,lo) represents the feature map of level y at scale x. The multi-level feature maps extracted at each scale are sent to the cross-scale feature fusion module CSFF for feature fusion.

[0025] Furthermore, in step S12, the cross-scale feature fusion module CSFF fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE: first, the multi-scale feature maps are rearranged according to the hierarchy, and the feature maps of different scales at the same hierarchy are stacked together. These connected feature maps are transformed in dimension through a 1×1 convolutional layer, and then a feature pyramid is constructed through a bottom-up path; the feature pyramid consists of high-level, mid-level, and low-level feature maps fused across scales;

[0026] Assume that the fused high-level, mid-level and low-level feature maps are m′ hi ,m′ mi and m′ lo , 1×1 convolution is conv y (y=lo,mi,hi), then the cross-scale feature fusion is:

[0027] m′ lo =conv lo (conc(∑ x m x,lo ))(2)

[0028] m′ mi =conv mi(conc(∑ x m x,mi ))+sub2(m′ lo )(3)

[0029] m′ hi =conv hi (conc(∑ x m x,hi ))+sub2(m′ mi )(4)

[0030] Where x = l, m, s, sub2 means double downsampling, conc is a stacking operation, different conv y The corresponding input channels are different, but the output channels are all 256; the feature pyramid is used for the convergence of MST-GAN during the training phase, and the high-level feature maps fused in the feature pyramid are used as the input of the Swin Transformer decoder STD.

[0031] Furthermore, in step S12, the Swin Transformer decoder STD uses the fused high-level feature map obtained by the cross-scale feature fusion module CSFF to reconstruct the template features; the Swin Transformer decoder STD consists of four sub-modules, of which the first three sub-modules include two consecutive Swin Transformer Blocks and a PatchExpanding, and the last sub-module has only two consecutive Swin Transformer Blocks. Patch Expanding is used for upsampling. In order to suppress the interference pixels in the reconstructed template caused by zero padding in deconvolution, Pixel Shuffle is used instead of deconvolution in PatchExpanding. After layer normalization, a 1×1 convolution layer is connected. When the input dimension of Patch Expanding is H×W×C, its output dimension is 2H×2W×C / 2; after the fused high-level feature map with a dimension of 8×8×256 is input into the Swin Transformer decoder STD, a template feature with a dimension of 64×64×32 is obtained, which is input into the multi-scale template generation module MSTG for reconstructing a multi-scale defect-free template.

[0032] Furthermore, in step S12, the multi-scale template generation module MSTG consists of three paths, which correspond to two Patch Expanding and one 1×1 convolution, one Patch Expanding and one 1×1 convolution, and only one 1×1 convolution, respectively, for reconstructing a multi-scale defect-free template.

[0033] Furthermore, the specific process of step S13 is:

[0034] A multi-scale loss function is constructed to ensure network convergence. The sub-networks composed of the multi-scale CNN encoder MSCE and the cross-scale feature fusion module CSFF in the generator G and discriminator D of the multi-scale generative adversarial network GAN are respectively denoted as G E1 and D -1 , the multi-scale encoder is denoted as E z :

[0035] Let X represent the multi-scale input of the multi-scale generative adversarial network GAN, then the loss function of the generator and discriminator of the multi-scale generative adversarial network GAN can be defined as:

[0036] L g =ω con *L con +ω enc *L enc +ω adv *L adv (5)

[0037]

[0038] Formula (5) is defined as:

[0039] L con =∑ x ‖X x -G(X) x ‖1,x=l,m,s (7)

[0040] L enc =∑ y ||G E1 (X) y -E2(G(X)) y ||1,y=hi,mi,lo (8)

[0041] L adv =∑ y ||D -1 (X) y -D -1 (G(X)) y ||1,y=hi,mi,lo (9)

[0042] Among them, G(x), G E1 (x), E2(G(x)), D(x) and D -1 (x) represents the MST-GAN generator G, sub-network G E1 and E2, the discriminator D and the subnetwork D in the discriminator -1The corresponding output, subscripts x and y represent the scale of the image and the level of the template or feature map, respectively, ‖·‖1 and ‖·‖2 represent the 1-norm and 2-norm, respectively. Since the input is 256×256, the values ​​of l, m, s are 256, 128, 64, respectively, while hi, mi, lo are 32, 16, 8, respectively.

[0043] Furthermore, the specific process of step S2 is:

[0044] First, the multi-scale difference map is averaged by multiple qualified samples in the training set to realize the multi-scale average feature map. The three scale maps obtained after the multi-scale average feature map is inverted and normalized are marked as F l ,F m and F s , then, the multi-scale weight mask is achieved by the following fusion:

[0045] F′ l =F l +up2(F m )+up4(F s ) (10)

[0046] F′ m =F m +sub2(F l )+up2(F s ) (11)

[0047] F′ s =F s +sub4(F l )+sub2(F m ) (12)

[0048] where F′ x Represents the weight mask at scale x, x = l, m, s, and the subscripts up and sub represent the multiples of upsampling and downsampling, respectively.

[0049] Furthermore, the specific process of step S3 is:

[0050] 1) Apply different thresholds to different weighted multi-scale difference images at each scale, defined as:

[0051] T x,n =α x μ x,n +σ x,n ,x=l,m,s (13)

[0052] A multi-scale strategy is adopted, that is, for large-scale images, we use a smaller threshold, and for small images, we use a larger threshold, so that the method can better combine the characteristics of various scales for defect detection. The thresholded multi-scale image will be used for further defect evaluation in step 2);

[0053] 2) Based on the sliding window strategy, MIPDE can evaluate the potential defects highlighted in the thresholded multi-scale difference map. For an inspection sample of size H×W, the sliding window strategy is used to obtain Row×Col image blocks, the sliding window size is w×w, and the sliding step size is ss, where

[0054]

[0055] For the multi-scale difference maps after thresholding at different scales, the window size and sliding step size are set differently;

[0056] set up is the k-th channel value of the (i, j)-th pixel in the (r, c)-th image block of the n-th detection sample on the x-scale. The defect probability of the image block is Can be defined as

[0057]

[0058] Where K represents the number of channels of the image block, which is equal to 3 here; then, the binary defect probability map of the image block is defined as:

[0059]

[0060] in Where ε is the sensitivity factor set to 0.0001, if is not less than τ, the corresponding image block is evaluated as possibly defective; otherwise, it is probably non-defective;

[0061] The defect assessment score is defined to evaluate the nth sample in (18), which is calculated by the binary defect probability maps of all image patches at three scales:

[0062]

[0063] in Indicates rounding down, and the evaluation method is:

[0064]

[0065] Where th is set to the maximum defect evaluation score of all qualified samples in the training set; if the defect evaluation score of the inspected sample is not less than th, the sample is judged to be unqualified, and vice versa.

[0066] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0067] 1. This patent builds a framework for IC metal package surface defect detection that combines semi-supervised deep learning and statistical modeling. This includes a multi-scale generative adversarial network (MST-GAN) with a Transformer for multi-scale template reconstruction, a multi-scale weighted masking scheme and a multi-scale adaptive thresholding scheme for highlighting potential defects, and an image patch-based multi-scale defect assessment scheme for further evaluating potential defects.

[0068] 2. Unlike GANs in previous detection frameworks, a fully multi-scale GAN with Transformers is designed to excellently capture the intrinsic patterns of qualified samples at multiple scales. The designed multi-scale GAN involves several new modules, including a multi-scale CNN encoder (MSCE), cross-scale feature fusion (CSFF), a Swin Transformer decoder (STD), and multi-scale template generation (MSTG). In particular, compared with traditional decoders, STD avoids the use of deconvolution and can effectively suppress interfering pixels in the reconstructed template. A multi-scale loss function is also designed to ensure network convergence.

[0069] 3. Previous semi-supervised deep learning methods evaluate defects at a single scale and are not good at capturing defects of widely varying sizes simultaneously. In addition to the multi-scale templates established by the proposed multi-scale GAN, several multi-scale schemes have been proposed to highlight potential defects and further evaluate them for inspection at multiple scales; they involve multi-scale weight masks, multi-scale adaptive thresholds, and image patch-based multi-scale defect assessment schemes. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flowchart of the processing process of the method of the present invention;

[0071] Figure 2 It is the structural diagram of MSCE;

[0072] Figure 3 This is the structural diagram of the CSFF module;

[0073] Figure 4 This is the structural diagram of the Swin Transformer decoder;

[0074] Figure 5 This is the structural diagram of Patch Expanding;

[0075] Figure 6 This is the structural diagram of the MSTG module;

[0076] Figure 7 Schematic diagram of multi-scale weight mask;

[0077] Figure 8 This is a diagram of the online detection process. DETAILED DESCRIPTION

[0078] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0079] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0080] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0081] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0082] Example 1

[0083] like Figure 1 As shown, a deep learning defect detection method for integrated circuit metal packaging includes the following steps:

[0084] S1: Design a multi-scale generative adversarial network (GAN) with a Transformer, named MST-GAN, to capture the intrinsic patterns of samples at multiple scales, generate multi-scale defect-free templates, and design a multi-scale loss function to ensure network convergence;

[0085] S2: Using the input sample and the multi-scale defect-free template obtained in step S1, the multi-scale difference image is obtained by performing element-wise subtraction and then taking the absolute value. A multi-scale weight mask is designed to suppress the reconstruction error in the multi-scale difference image.

[0086] S3: Design a multi-scale adaptive threshold, process the multi-scale difference image obtained after the multi-scale weight mask processing in S2 to obtain a threshold image, and design a multi-scale defect evaluation scheme MIPDE based on image blocks to detect potential defects in the threshold image and obtain defect location.

[0087] Example 2

[0088] like Figure 1 As shown, a deep learning defect detection method for integrated circuit metal packaging includes the following steps:

[0089] S1: Design a multi-scale generative adversarial network (GAN) with a Transformer, named MST-GAN, to capture the intrinsic patterns of samples at multiple scales, generate multi-scale defect-free templates, and design a multi-scale loss function to ensure network convergence;

[0090] S2: Using the input sample and the multi-scale defect-free template obtained in step S1, the multi-scale difference image is obtained by performing element-wise subtraction and then taking the absolute value. A multi-scale weight mask is designed to suppress the reconstruction error in the multi-scale difference image.

[0091] S3: Design a multi-scale adaptive threshold, process the multi-scale difference image obtained after the multi-scale weight mask processing in S2 to obtain a threshold image, and design a multi-scale defect evaluation scheme MIPDE based on image blocks to detect potential defects in the threshold image and obtain defect location.

[0092] The specific process of step S1 is:

[0093] S11: Process the collected original IC metal package surface image and use binarization, Canny, and digital morphology to obtain the region of interest (ROI). This ROI occupies the main area of ​​the IC metal package surface. The ROI is segmented and divided into qualified samples and unqualified samples to create training and test sets. The image pixels are normalized so that the image pixel values ​​are between [-1, 1], and the model parameters are initialized.

[0094] S12: Construct a multi-scale generative adversarial network (GAN) with a Transformer. The network consists of a generator (G), a discriminator (D), and a multi-scale encoder. The generator consists of a multi-scale CNN encoder (MSCE), a cross-scale feature fusion module (CSFF), a Swin Transformer decoder (STD), and a multi-scale template generation module (MSTG). Both the discriminator and the multi-scale encoder consist of a multi-scale CNN encoder (MSCE) and a cross-scale feature fusion module (CSFF).

[0095] The multi-level feature maps of each scale extracted by the multi-scale CNN encoder MSCE are fed into the cross-scale feature fusion module CSFF;

[0096] The cross-scale feature fusion module CSFF combines high-level semantic information with low-level spatial information at different scales, that is, it fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE;

[0097] The Swin Transformer decoder STD uses the fused high-level feature maps obtained by the cross-scale feature fusion module CSFF module to reconstruct the template features;

[0098] The multi-scale template generation module MSTG reconstructs multi-scale defect-free templates;

[0099] S13: Construct a multi-scale loss function to ensure network convergence.

[0100] Example 3

[0101] like Figure 1 As shown, a deep learning defect detection method for integrated circuit metal packaging includes the following steps:

[0102] S1: Design a multi-scale generative adversarial network (GAN) with a Transformer, named MST-GAN, to capture the intrinsic patterns of samples at multiple scales, generate multi-scale defect-free templates, and design a multi-scale loss function to ensure network convergence;

[0103] S2: Using the input sample and the multi-scale defect-free template obtained in step S1, the multi-scale difference image is obtained by performing element-wise subtraction and then taking the absolute value. A multi-scale weight mask is designed to suppress the reconstruction error in the multi-scale difference image.

[0104] S3: Design a multi-scale adaptive threshold, process the multi-scale difference image obtained after the multi-scale weight mask processing in S2 to obtain a threshold image, and design a multi-scale defect evaluation scheme MIPDE based on image blocks to detect potential defects in the threshold image and obtain defect location.

[0105] The specific process of step S1 is:

[0106] S11: Process the collected original IC metal package surface image and use binarization, Canny, and digital morphology to obtain the region of interest (ROI). This ROI occupies the main area of ​​the IC metal package surface. The ROI is segmented and divided into qualified samples and unqualified samples to create training and test sets. The image pixels are normalized so that the image pixel values ​​are between [-1, 1], and the model parameters are initialized.

[0107] S12: Construct a multi-scale generative adversarial network (GAN) with a Transformer. The network consists of a generator (G), a discriminator (D), and a multi-scale encoder. The generator consists of a multi-scale CNN encoder (MSCE), a cross-scale feature fusion module (CSFF), a Swin Transformer decoder (STD), and a multi-scale template generation module (MSTG). Both the discriminator and the multi-scale encoder consist of a multi-scale CNN encoder (MSCE) and a cross-scale feature fusion module (CSFF).

[0108] The multi-level feature maps of each scale extracted by the multi-scale CNN encoder MSCE are fed into the cross-scale feature fusion module CSFF;

[0109] The cross-scale feature fusion module CSFF combines high-level semantic information with low-level spatial information at different scales, that is, it fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE;

[0110] The Swin Transformer decoder STD uses the fused high-level feature maps obtained by the cross-scale feature fusion module CSFF module to reconstruct the template features;

[0111] The multi-scale template generation module MSTG reconstructs multi-scale defect-free templates;

[0112] S13: Construct a multi-scale loss function to ensure network convergence.

[0113] like Figure 2 As shown, in step S12, the multi-scale CNN encoder MSCE performs multi-scale feature extraction: three multi-scale feature maps with the same structure are extracted from the small-scale, medium-scale and large-scale inputs respectively. The feature map of each scale represents the low-level, medium-level and high-level features of the corresponding input. Let the large-scale, medium-scale and small-scale inputs be I x , x = l, m, s, the feature map of the corresponding scale is:

[0114] {m x,lo , m x,mi , m x,hi}={f 1～nx (I x ), f 1～nx+1 (I x ), f 1～nx+2 (I x )},x=l,m,s (1)

[0115] where n x The values ​​of (x=l,m,s) are 3, 2 and 1 respectively, f 1～nx (I x ) indicates that the input Ix is passed from the first ResNetBottleneck to the nth x After blocks, the nth x The output of the block, m x,y (x=l,m,s; y=hi,mi,lo) represents the feature map of level y at scale x. The multi-level feature maps extracted at each scale are sent to the cross-scale feature fusion module CSFF for feature fusion.

[0116] like Figure 3 As shown, in step S12, the cross-scale feature fusion module CSFF fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE: first, the multi-scale feature maps are rearranged according to the hierarchy, and the feature maps of different scales at the same hierarchy are stacked together. These connected feature maps are transformed in dimension through a 1×1 convolutional layer, and then a feature pyramid is constructed through a bottom-up path; the feature pyramid consists of high-level, mid-level, and low-level feature maps fused across scales;

[0117] Assume that the fused high-level, mid-level and low-level feature maps are m′ hi ,m′ mi and m′ lo , 1×1 convolution is conv y (y=lo,mi,hi), then the cross-scale feature fusion is:

[0118] m′ lo =conv lo (conc(∑ x m x,lo ))(2)

[0119] m′ mi =conv mi (conc(∑ x m x,mi ))+sub2(m′ lo )(3)

[0120] m′ hi =conv hi (conc(∑ x m x,hi ))+sub2(m′ mi )(4)

[0121] Where x = l, m, s, sub2 means double downsampling, conc is a stacking operation, different conv y The corresponding input channels are different, but the output channels are all 256; the feature pyramid is used for the convergence of MST-GAN during the training phase, and the high-level feature maps fused in the feature pyramid are used as the input of the Swin Transformer decoder STD.

[0122] like Figure 4-5As shown in FIG, in step S12, the Swin Transformer decoder STD uses the fused high-level feature map obtained by the cross-scale feature fusion module CSFF to reconstruct the template features; the Swin Transformer decoder STD consists of four sub-modules, of which the first three sub-modules include two consecutive Swin Transformer Blocks and a PatchExpanding, and the last sub-module has only two consecutive Swin Transformer Blocks. Patch Expanding is used for upsampling. In order to suppress the interference pixels in the reconstructed template caused by zero padding in deconvolution, PixelShuffle is used instead of deconvolution in PatchExpanding. After layer normalization, a 1×1 convolution layer is connected. When the input dimension of Patch Expanding is H×W×C, its output dimension is 2H×2W×C / 2; after the fused high-level feature map with a dimension of 8×8×256 is input into the Swin Transformer decoder STD, a template feature with a dimension of 64×64×32 is obtained, which is input into the multi-scale template generation module MSTG for reconstructing a multi-scale defect-free template.

[0123] like Figure 6 As shown, in step S12, the multi-scale template generation module MSTG consists of three paths, which correspond to two Patch Expanding and one 1×1 convolution, one Patch Expanding and one 1×1 convolution, and only one 1×1 convolution, respectively, for reconstructing a multi-scale defect-free template.

[0124] The specific process of step S13 is:

[0125] A multi-scale loss function is constructed to ensure network convergence. The sub-networks composed of the multi-scale CNN encoder MSCE and the cross-scale feature fusion module CSFF in the generator G and discriminator D of the multi-scale generative adversarial network GAN are respectively denoted as G E1 and D -1 , the multi-scale encoder is denoted as E2:

[0126] Let X represent the multi-scale input of the multi-scale generative adversarial network GAN, then the loss function of the generator and discriminator of the multi-scale generative adversarial network GAN can be defined as:

[0127] L g =ω con *L con +ω enc *L enc +ω adv *L adv (5)

[0128]

[0129] Formula (5) is defined as:

[0130] L con =∑ x ‖X x -G(X) x ‖1,x=l,m,s (7)

[0131] L enc =∑ y ||G E1 (X) y -E2(G(X)) y ||1,y=hi,mi,lo (8)

[0132] L adv =∑ y ||D -1 (X) y -D -1 (G(X)) y ||1,y=hi,mi,lo (9)

[0133] Among them, G(x), G E1 (x), E2(G(x)), D(x) and D -1 (x) represents the MST-GAN generator G, sub-network G E1 and E2, the discriminator D and the subnetwork D in the discriminator -1 The corresponding output, subscripts x and y represent the scale of the image and the level of the template or feature map, respectively, ‖·‖1 and ‖·‖2 represent the 1-norm and 2-norm, respectively. Since the input is 256×256, the values ​​of l, m, s are 256, 128, 64, respectively, while hi, mi, lo are 32, 16, 8, respectively.

[0134] like Figure 7 As shown, the specific process of step S2 is:

[0135] First, the multi-scale difference map is averaged by multiple qualified samples in the training set to realize the multi-scale average feature map. The three scale maps obtained after the multi-scale average feature map is inverted and normalized are marked as F l ,F m and F s , then, the multi-scale weight mask is achieved by the following fusion:

[0136] F′ l =F l +up2(F m )+up4(Fs ) (10)

[0137] F′ m =F m +sub2(F l )+up2(F s ) (11)

[0138] F′ s =F s +sub4(F l )+sub2(F m ) (12)

[0139] where F′ x Represents the weight mask at scale x, x = l, m, s, and the subscripts up and sub represent the multiples of upsampling and downsampling, respectively.

[0140] like Figure 8 As shown, the specific process of step S3 is:

[0141] 1) Apply different thresholds to different weighted multi-scale difference images at each scale, defined as:

[0142] T x,n =α x μ x,n +σ x,n ,x=l,m,s (13)

[0143] A multi-scale strategy is adopted, that is, for large-scale images, we use a smaller threshold, and for small images, we use a larger threshold, so that the method can better combine the characteristics of various scales for defect detection. The thresholded multi-scale image will be used for further defect evaluation in step 2);

[0144] 2) Based on the sliding window strategy, MIPDE can evaluate the potential defects highlighted in the thresholded multi-scale difference map. For an inspection sample of size H×W, the sliding window strategy is used to obtain Row×Col image blocks, the sliding window size is w×w, and the sliding step size is ss, where

[0145]

[0146] For the multi-scale difference maps after thresholding at different scales, the window size and sliding step size are set differently, as shown in Table 1:

[0147] Table 1 Sliding window of multi-scale difference map after thresholding

[0148] sizes of images w ss 256×256 16 8 128×128 8 4 64×64 4 2

[0149] set up is the k-th channel value of the (i, j)-th pixel in the (r, c)-th image block of the n-th detection sample on the x-scale. The defect probability of the image block is Can be defined as

[0150]

[0151] Where K represents the number of channels of the image block, which is equal to 3 here; then, the binary defect probability map of the image block is defined as:

[0152]

[0153] in Where ε is the sensitivity factor set to 0.0001, if is not less than τ, the corresponding image block is evaluated as possibly defective; otherwise, it is probably non-defective. The defect evaluation score is defined to evaluate the n-th sample in (18), which is calculated by the binary defect probability maps of all image blocks at three scales:

[0154]

[0155] in Indicates rounding down, and the evaluation method is:

[0156]

[0157] Where th is set to the maximum defect evaluation score of all qualified samples in the training set; if the defect evaluation score of the inspected sample is not less than th, the sample is judged to be unqualified, and vice versa.

[0158] The same or similar reference numerals correspond to the same or similar components;

[0159] The positional relationships described in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for detecting defects in integrated circuit metal packages using deep learning, characterized in that: The following steps are involved: S1: Design a multi-scale generative adversarial network (GAN) with a Transformer, named MST-GAN, to capture the intrinsic patterns of samples at multiple scales, generate multi-scale defect-free templates, and design a multi-scale loss function to ensure network convergence; S2: Using the input sample and the multi-scale defect-free template obtained in step S1, the multi-scale difference image is obtained by performing element-wise subtraction and then taking the absolute value. A multi-scale weight mask is designed to suppress the reconstruction error in the multi-scale difference image. S3: Design a multi-scale adaptive threshold, process the multi-scale difference image obtained after the multi-scale weight mask processing in S2 to obtain a threshold image, and design a multi-scale defect evaluation scheme based on image blocks (MIPDE) to detect potential defects in the threshold image and obtain defect location; The specific process of step S3 is: 1) Apply different thresholds to different weighted multi-scale difference images at each scale, defined as: T x,n =a x ·m x,n +s x,n ,x=l,m,s The subscript x represents the scale of the input image, and l, m, and s correspond to large, medium, and small scales, respectively. A multi-scale strategy is adopted, where a smaller threshold is used for large-scale images and a larger threshold is used for small-scale images. This allows the method to better combine the characteristics of various scales for defect detection. The thresholded multi-scale image is used for further defect assessment in step 2. 2) Based on the sliding window strategy, MIPDE can evaluate the potential defects highlighted in the thresholded multi-scale difference map. For an inspection sample of size H×W, the sliding window strategy is used to obtain Row×Col image blocks, the sliding window size is w×w, and the sliding step size is ss, where For the multi-scale difference maps after thresholding at different scales, the window size and sliding step size are set differently; set up is the k-th channel value of the (i, j)-th pixel in the (r, c)-th image block of the n-th detection sample on the x-scale. The defect probability of the image block is Can be defined as Where K represents the number of channels of the image block, which is equal to 3 here; then, the binary defect probability map of the image block is defined as: in Where ε is the sensitivity factor, if is not less than τ, the corresponding image block is evaluated as possibly defective; otherwise, it may be non-defective. A defect assessment score is defined to evaluate the n-th sample, which is calculated through the binary defect probability maps of all image blocks at three scales: in Indicates rounding down, and the evaluation method is: Where th is set to the maximum defect evaluation score of all qualified samples in the training set; if the defect evaluation score of the inspected sample is not less than th, the sample is judged to be unqualified, and vice versa.

2. The integrated circuit metal package deep learning defect detection method according to claim 1, characterized in that: The specific process of step S1 is: S11: Process the collected original IC metal package surface image and use binarization, Canny, and digital morphology to obtain the region of interest (ROI). This ROI occupies the main area of ​​the IC metal package surface. The ROI is segmented and divided into qualified samples and unqualified samples to create training and test sets. The image pixels are normalized so that the image pixel values ​​are between [-1, 1], and the model parameters are initialized. S12: Construct a multi-scale generative adversarial network (GAN) with a Transformer. The network consists of a generator (G), a discriminator (D), and a multi-scale encoder. The generator consists of a multi-scale CNN encoder (MSCE), a cross-scale feature fusion module (CSFF), a SwinTransformer decoder (STD), and a multi-scale template generation module (MSTG). Both the discriminator and the multi-scale encoder consist of a multi-scale CNN encoder (MSCE) and a cross-scale feature fusion module (CSFF). The multi-level feature maps of each scale extracted by the multi-scale CNN encoder MSCE are fed into the cross-scale feature fusion module CSFF; The cross-scale feature fusion module CSFF combines high-level semantic information with low-level spatial information at different scales, that is, it fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE; The Swin Transformer decoder STD uses the fused high-level feature maps obtained by the cross-scale feature fusion module CSFF module to reconstruct the template features; The multi-scale template generation module MSTG reconstructs multi-scale defect-free templates; S13: Construct a multi-scale loss function to ensure network convergence.

3. The integrated circuit metal package deep learning defect detection method according to claim 2, characterized in that: In step S12, the multi-scale CNN encoder MSCE performs multi-scale feature extraction: three multi-scale feature maps with the same structure are extracted from the small-scale, medium-scale and large-scale inputs respectively, and the feature map of each scale represents the low-level, medium-level and high-level features of the corresponding input. Let the large-scale, medium-scale and small-scale inputs be I x , x = l, m, s, the feature map of the corresponding scale is: where n x , the values ​​of x=l,m,s are 3, 2 and 1 respectively, Indicates that the input Ix is passed from the first ResNetBottleneck to the nth x After blocks, the nth x The output of the block, m x,y , x=l,m,s; y=hi,mi,lo, represents the feature map of level y at scale x. The multi-level feature maps of each scale extracted are sent to the cross-scale feature fusion module CSFF for feature fusion.

4. The integrated circuit metal package deep learning defect detection method according to claim 3, characterized in that: In step S12, the cross-scale feature fusion module CSFF fuses the multi-level feature maps of multiple scales extracted by the multi-scale CNN encoder MSCE: first, the multi-scale feature maps are rearranged according to the hierarchy, and the feature maps of different scales at the same hierarchy are stacked together. These connected feature maps are transformed in dimension through a 1×1 convolutional layer, and then a feature pyramid is constructed through a bottom-up path; The feature pyramid consists of high-level, mid-level, and low-level feature maps fused across scales; Assume that the fused high-level, mid-level and low-level feature maps are m′ respectively hi ,m′ mi and m′ lo , 1×1 convolution is conv y , y = lo, mi, hi, then the cross-scale feature fusion is: m′ lo =conv lo (conc(∑ x m x,lo )) m′ mi =conv mi (conc(∑ x m x,mi ))+sub2(m′ lo ) m′ hi =conv hi (conc(∑ x m x,hi ))+sub2(m′ mi ) Where x = l, m, s, sub2 means double downsampling, conc is a stacking operation, different conv y The corresponding input channels are different, but the output channels are all 256; the feature pyramid is used for the convergence of MST-GAN during the training phase, and the high-level feature maps fused in the feature pyramid are used as the input of the Swin Transformer decoder STD.

5. The integrated circuit metal package deep learning defect detection method according to claim 4, characterized in that: In step S12, the Swin Transformer decoder STD uses the fused high-level feature map obtained by the cross-scale feature fusion module CSFF to reconstruct the template features; the Swin Transformer decoder STD consists of four sub-modules, of which the first three sub-modules include two consecutive Swin Transformer Blocks and a Patch Expanding, and the last sub-module has only two consecutive Swin Transformer Blocks. Patch Expanding is used for upsampling. In order to suppress the interference pixels in the reconstructed template caused by zero padding in deconvolution, Pixel Shuffle is used instead of deconvolution in Patch Expanding. After layer normalization, a 1×1 convolution layer is connected. When the input dimension of Patch Expanding is H×W×C, its output dimension is 2H×2W×C / 2; after the fused high-level feature map with a dimension of 8×8×256 is input into the Swin Transformer decoder STD, a template feature with a dimension of 64×64×32 is obtained, which is input into the multi-scale template generation module MSTG for reconstructing a multi-scale defect-free template.

6. The integrated circuit metal package deep learning defect detection method according to claim 5, characterized in that: In step S12, the multi-scale template generation module MSTG consists of three paths, which respectively correspond to two PatchExpanding and one 1×1 convolution, one Patch Expanding and one 1×1 convolution, and only one 1×1 convolution, for reconstructing a multi-scale defect-free template.

7. The integrated circuit metal package deep learning defect detection method according to claim 6, characterized in that: The specific process of step S13 is: A multi-scale loss function is constructed to ensure network convergence. The sub-networks composed of the multi-scale CNN encoder MSCE and the cross-scale feature fusion module CSFF in the generator G and discriminator D of the multi-scale generative adversarial network GAN are respectively denoted as G E1 and D -1 , the multi-scale encoder is denoted as E2: Let X represent the multi-scale input of the multi-scale generative adversarial network GAN, then the loss function of the generator and discriminator of the multi-scale generative adversarial network GAN can be defined as: L g =ω con *L con +oh enc *L enc +oh adv *L adv Among them L con , L enc , L adv is defined as: L con =∑ x ‖X x -G(X) x ‖1,x=l,m,s L enc =∑ y ||G E1 (X) y -E2(G(X)) y ||1,and=hi,mi,lo L adv =∑ y ||D -1 (X) y -D -1 (G(X)) y ||1,y=hi,mi,lo Among them, G(X), G E1 (X), E2(G(X)), D(X) and D -1 (X) represents the MST-GAN generator G, sub-network G E1 and E2, the discriminator D and the subnetwork D in the discriminator -1 The corresponding output, subscripts x and y represent the scale of the image and the level of the template or feature map, respectively, ‖·‖1 and ‖·‖2 represent the 1-norm and 2-norm, respectively. Since the input is 256×256, the values ​​of l, m, s are 256, 128, 64, respectively, while hi, mi, lo are 32, 16, 8, respectively.

8. The integrated circuit metal package deep learning defect detection method according to claim 7, characterized in that: The specific process of step S2 is: First, the multi-scale difference map is averaged by multiple qualified samples in the training set to realize the multi-scale average feature map. The three scale maps obtained after the multi-scale average feature map is inverted and normalized are marked as F l ,F m and F s , then, the multi-scale weight mask is achieved by the following fusion: F′ l =F l +up2(F m )+up4(F s ) F′ m =F m +sub2(F l )+up2(F s ) In s =F s +sub4(F l )+sub2(F m ) where F′ x Represents the weight mask at scale x, x = l, m, s, and the subscripts up and sub represent the multiples of upsampling and downsampling, respectively.

9. The integrated circuit metal package deep learning defect detection method according to claim 1, characterized in that: The ε is 0.0001.

Images