A parallel detection method for chip surface defects based on improved convolutional variational autoencoder

Abstract

The present invention discloses a parallel detection method for chip surface defects based on an improved convolutional variational autoencoder, which improves the problem in the prior art that chip surface defect detection still requires intelligence. The invention comprises the following steps: step 1, chip image acquisition and preprocessing; step 2, constructing and training an improved convolutional variational autoencoder; step 3, performing real-time defect detection on the chip to be tested. The present invention partially improves the traditional variational autoencoder, and only uses normal samples for unsupervised training. Without the need for labeled defect sample data, it can detect all defects that are different from normal samples, reducing manpower and material resources. At the same time, the method decomposes the original image into multiple sub-images and processes them in parallel, which can efficiently, quickly and cheaply perform various defect detection on the chip surface.

Description

Technical Field

[0001] The present invention relates to the technical field of applying computer vision to chip screening, and in particular to a parallel detection method for chip surface defects based on an improved convolutional variational autoencoder. Background Art

[0002] Chip surface defect detection aims to eliminate defective chips with surface defects such as character defects, pin defects, and surface scratches on the production line, making it crucial for controlling product quality. Traditional defect detection methods based on explicit feature extraction require manual design and extraction, which is laborious and time-consuming. Most current deep learning-based surface defect detection methods rely on supervised learning from a large number of defect samples, requiring extensive end-to-end learning from a large number of labeled training samples. However, collecting such a large number of defect samples is extremely difficult. The random and incidental nature of defects makes it difficult to fully collect all defect types, making it difficult for trained models to detect defects they have not learned. Furthermore, labeling defect samples is labor-intensive and time-consuming. Furthermore, previous defect detection methods are computationally intensive, requiring long runtimes and slow detection speeds, making them unable to meet the timeliness requirements of industrial product defect detection. Even some targeted detection methods cannot fully detect all types of surface defects. Therefore, the aforementioned issues in the field of chip surface defect detection, such as the limited number of labeled datasets, long detection times, and incomplete detection, urgently need to be addressed. Summary of the Invention

[0003] The present invention improves the problem in the prior art that chip surface defect detection still requires intelligence, and provides a parallel chip surface defect detection method based on an improved convolutional variational autoencoder with short detection time and comprehensive detection.

[0004] The technical solution of the present invention is to provide a parallel chip surface defect detection method based on an improved convolutional variational autoencoder with the following steps: step 1, chip image acquisition and preprocessing; step 2, constructing and training an improved convolutional variational autoencoder; step 3, performing real-time defect detection on the chip to be tested.

[0005] Preferably, in step 1, a high-definition camera module, a host image processing module and a lighting module are used to collect and store images, collect surface images of normal chips, decompose the collected normal chip images into multiple images of the same size, create multiple image detection tasks, create a thread for each image detection task, and multiple threads process image defect detection tasks in parallel. In addition, a defect sound and light alarm module is equipped to prompt the detection of defective chips.

[0006] Preferably, the step 2 comprises the following steps: Step 2.1, constructing an improved convolutional variational autoencoder network, which consists of an encoder and a decoder:

[0007] The encoder is set to contain five convolutional submodules. Each convolutional submodule consists of two full convolutional layers and a maximum pooling layer, which is used to curl image features and reduce the number of parameters. The decoder structure has a structure that is completely symmetrical with the encoder. The convolutional layer is converted into a transposed convolutional layer, and the maximum pooling layer is converted into a maximum unpooling layer. Between each convolutional submodule of the encoder and decoder, a self-attention module CBAM and a deep connection attention network module DCAnet are added for connection. The CBAM module also contains a channel attention submodule CAM and a spatial attention submodule SAM.

[0008] Step 2.2, train the convolutional variational autoencoder:

[0009] During the training phase, the collected normal chip surface image is input into the improved convolutional variational autoencoder network, and the cross entropy loss is used as the reconstruction loss and included as one of the loss functions. The calculation method is as follows: Loss1 = ||x'-x|| 2 Where x' represents the reconstructed image data, x represents the input image data; at the same time, in order to regularize the encoder, KL divergence is used as another term of the loss function, Loss2=KL(N(μ,σ 2 )|N(0,1)), where N(μ,σ 2 ) is the normal distribution of the latent variables in the low-dimensional latent space representation of the input image data, obviously the mean μ, variance σ 2 is the characteristic parameter of the latent variable, N(0,1) is the standard normal distribution, and the total loss function of the process is Loss=Loss1+Loss2, while the network structure of the other encoder E2 is exactly the same as that of encoder E1.

[0010] Preferably, step 3 includes the following steps: real-time acquisition of the surface image of the chip to be tested on the industrial production line, decomposing the acquired image of the chip to be tested into images of the same size of NxN pixels and SxT, creating multiple image detection tasks, creating a thread for each image detection task, and multiple threads processing the image defect detection task in parallel, respectively inputting the real-time acquired chip surface image into the encoder E1 and encoder E2 in the detection network, when there is a defect in the input chip surface image Q1, the encoder E2 in the network model will extract the defect image feature and obtain the low-dimensional latent space feature representation C2 of the image, denoting ||C1-C2|| as one of the chip surface defect anomaly score values, and the convolutional variational autoencoder network model with the encoder E1. The decoder D1 connected to the two phases can automatically remove defect features while retaining normal features in the process of generating a reconstructed image. Therefore, the reconstructed image Q2 of the defective chip is subtracted from the input defective chip surface image Q1 to obtain a residual image. The information contained in the residual image can not only determine whether the chip under test has a defect but also calibrate the defect location. The residual image is digitized to obtain its specific numerical representation ||Q1-Q2||, which is recorded as another defect score value. The above two defect score values ​​are weighted and summed to obtain the final abnormal score value calculation formula M=λ1||C1-C2||+λ2||Q1-Q2||. The abnormal score value is compared with the set threshold. If it exceeds the threshold, it is determined that there is a defect, and the defect is identified using the residual image.

[0011] Compared with the existing technology, the chip surface defect parallel detection method based on the improved convolutional variational autoencoder of the present invention has the following advantages: the traditional variational autoencoder is partially improved, only normal samples are used for unsupervised training, and all defects that are different from normal samples can be detected without the need for labeled defect sample data, thus reducing manpower and material resources. At the same time, this method decomposes the original image into multiple sub-images and processes them in parallel, which can efficiently, quickly and low-costly detect various defects on the chip surface.

[0012] This method effectively detects comprehensive defects on chip surfaces, including defects in characters, pins, and packaging, eliminating defective chips with surface defects. Training the model does not require manual image annotation, making it an unsupervised defect detection model, reducing both human and time costs. It also decomposes the real-time inspection image into multiple sub-images and computes multiple defect detection subtasks in parallel, significantly reducing defect detection time and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a specific flow diagram of the present invention;

[0014] Figure 2This is a schematic diagram of the convolutional variational autoencoder network principle constructed by the present invention. DETAILED DESCRIPTION

[0015] The following is a further description of the chip surface defect parallel detection method based on the improved convolutional variational autoencoder of the present invention in conjunction with the accompanying drawings and specific embodiments: This embodiment includes the following steps:

[0016] S1, chip image acquisition and preprocessing, S2, construction and training of improved convolutional variational autoencoder, S3, real-time defect detection of the chip under test.

[0017] S1 includes the following steps:

[0018] S1.1. Use a high-definition camera module, a host image processing module, and a lighting module to capture and store images for subsequent image processing. A defect sound and light alarm module is also included to indicate the presence of defective chips. A CCD camera and two LED array light sources are used for cross-sector dark field illumination.

[0019] S1.2. Capture a normal chip surface image and decompose it into NxN pixels of the same size (SxT). Create multiple image inspection tasks, create a thread for each sub-image inspection task, and process the image defect inspection tasks in parallel across multiple threads. Note that the inspection process for each sub-image is identical. The specific size and number of sub-images are determined based on the actual host computing power to minimize the time required to inspect defects in the entire image.

[0020] S2 includes the following steps:

[0021] S2.1. Construct an improved convolutional variational autoencoder network, which consists of two encoders and one decoder:

[0022] During the training phase, one encoder, E1, uses normal images to obtain parameters C1 (i.e., normal image features) describing the distribution of latent variables in a normal image. These parameters (which remain unchanged after training) are input into a decoder D1 connected to the corresponding encoder E1. Decoder D1 decodes these parameters to obtain a reconstructed image of the input image. During the detection phase, the other encoder, E2, extracts parameters C2 (which describe the characteristics of the image under test) describing the distribution of latent variables. These parameters are then compared with parameters C1 to determine whether the image has defects.

[0023] To better enable the network model to learn to extract features from the input image, we improved upon the traditional variational autoencoder architecture. First, we configured the encoder to include five convolutional submodules, each consisting of two fully convolutional layers and a maximum pooling layer to convolve image features and reduce the number of parameters. In convolutional module one, the convolution kernel size is 9x9 with a stride of 1x1, and the maximum pooling layer size is 2x2; in convolutional module two, the convolution kernel size is 7x7 with a stride of 1x1, and the maximum pooling layer size is 2x2; in convolutional module three, the convolution kernel size is 5x5 with a stride of 1x1, and the maximum pooling layer size is 2x2; in convolutional module four, the convolution kernel size is 3x3 with a stride of 1x1, and the maximum pooling layer size is 2x2; and in convolutional module five, the convolution kernel size is 1x1 with a stride of 1x1, and the maximum pooling layer size is 2x2. The decoding network structure is completely symmetrical with the encoding network, except that the convolutional layers are replaced with transposed convolutional layers and the maximum pooling layers are replaced with maximum unpooling layers. Deconvolution and unpooling restore the image reduced by convolution and pooling to its original size, enabling reconstruction of the original image. The activation function of all convolutional layers, except for the final decoder layer, uses the tanh function. A fully connected layer and a sampling layer connect the encoder E1 and decoder D1.

[0024] In addition, we added a lightweight self-attention module (CBAM) and a deeply connected attention network (DCAnet) between the convolutional submodules of the encoder and decoder to further effectively extract salient image features. The CBAM module consists of a channel attention submodule (CAM) and a spatial attention submodule (SAM). This combination is a serial CAM-SAM approach, enhancing the model's focus on valid image information. The CAM module processes the input feature map through a width-based max pooling layer and a height-based average pooling layer, respectively, to produce two reduced sub-feature maps. These sub-feature maps are then fed into a two-layer neural network (MLP) with a shared Reinforced Unit (ReLU) activation function. The features output by the MLP are then summed and activated with a sigmoid activation function to generate the final channel self-attention features, which are then fed into the SAM module. Similarly, the SAM architecture consists of a max pooling layer, an average pooling layer, a convolutional layer, and a sigmoid activation function (combined in a serial manner) to generate the spatial self-attention features, which are then multiplied with the previous channel self-attention features to produce the final features. DCAnet connects adjacent self-attention modules to each other, allowing information to flow between self-attention modules, thereby improving the feature extraction performance of the self-attention module CBAM.

[0025] The above improvements are designed to enable the encoder in the network model to extract the most significant and effective features of the input image and obtain the best low-dimensional latent space representation of the image. At the same time, the decoder can reconstruct an image similar to the input image based on the low-dimensional latent space representation.

[0026] S2.2. Train the convolutional variational autoencoder.

[0027] During the training phase, the captured normal chip surface image is fed into the improved convolutional variational autoencoder network. Due to the excellent feature extraction capabilities of encoder E1 and decoder D1 in the convolutional variational autoencoder network, the salient features of the normal chip surface image can be effectively extracted. These features can be represented in a low-dimensional latent space C1, and a reconstructed image is generated using decoder D1. To ensure that the reconstructed image is as similar as possible to the input image, we use cross-entropy loss as the reconstruction loss and include it in the loss function. The calculation method is as follows: Loss1 = ||x'-x|| 2 Where x' represents the reconstructed image data, x represents the input image data; at the same time, in order to regularize the encoder, KL divergence is used as another term of the loss function, Loss2=KL(N(μ,σ 2 )|N(0,1)), where N(μ,σ 2 ) is the normal distribution of the latent variables in the low-dimensional latent space C1, where the mean μ and variance σ are 2 is the characteristic parameter of the latent variable, and N(0,1) is a standard normal distribution. The overall loss function for this process is Loss = Loss1 + Loss2. The network structure of the other encoder, E2, is identical to that of encoder E1, except that its parameters use customized optimization parameters, selected based on the training process of encoder E1. This allows it to effectively extract the salient features of the chip under test during the detection phase and obtain its low-dimensional latent space representation (i.e., the latent variable distribution parameters).

[0028] S3 involves the following steps:

[0029] S3.1. Real-time surface images of chips under test on an industrial production line are captured and decomposed into NxN pixels of equal size SxT. Multiple image detection tasks are created, with one thread created for each. Multiple threads process image defect detection tasks in parallel. These are fed into a trained convolutional variational autoencoder network model, and an anomaly score is calculated. A threshold is set based on experience. If the threshold is exceeded, a defect is identified and the defect is mapped using a residual image.

[0030] Real-time chip surface images are fed into encoders E1 and E2 of the detection network. If a defect is detected in the input chip surface image Q1, encoder E2 in the network model extracts the defect image features and obtains a low-dimensional latent space feature representation C2 for the image. This feature representation differs from the low-dimensional latent space feature representation C1 of a normal image. We denote ||C1-C2|| as one of the chip surface defect anomaly scores.

[0031] At the same time, the decoder D1 connected to the encoder E2 in the convolutional variational autoencoder network model automatically removes defect features while retaining normal features during the reconstructed image generation process. The reconstructed image Q2 of the defective chip is then subtracted from the input defective chip surface image Q1 to produce a residual image. The information contained in the residual image not only determines whether the chip under test has a defect but also locates the defect. The residual image is digitized to obtain its specific numerical representation ||Q1-Q2||, which is recorded as another defect score. The two defect scores are weighted and summed to obtain the final anomaly score calculated using the formula M = λ1||C1-C2||+λ2||Q1-Q2||. This anomaly score is compared with a set threshold. If it exceeds the threshold, it is considered a defect. The residual image is used to identify the defect, thus achieving defect detection on the chip surface.

Claims

1. A parallel chip surface defect detection method based on an improved convolutional variational autoencoder, characterized by: Contains the following steps, Step 1: Chip image acquisition and preprocessing; Step 2: Build and train the improved convolutional variational autoencoder; The following steps are involved: Step 2.

1. Build an improved convolutional variational autoencoder network, which consists of an encoder and a decoder: The encoder is set to contain five convolutional submodules. Each convolutional submodule consists of two full convolutional layers and a maximum pooling layer, which is used to curl image features and reduce the number of parameters. The decoder structure has a structure that is completely symmetrical with the encoder. The convolutional layer is converted into a transposed convolutional layer, and the maximum pooling layer is converted into a maximum unpooling layer. Between each convolutional submodule of the encoder and decoder, a self-attention module CBAM and a deep connection attention network module DCAnet are added for connection. The CBAM module also contains a channel attention submodule CAM and a spatial attention submodule SAM. Step 2.2, train the convolutional variational autoencoder: During the training phase, the collected normal chip surface image is input into the improved convolutional variational autoencoder network, and the cross entropy loss is used as the reconstruction loss and included as one of the loss functions. The calculation method is as follows: Loss1 = ||x'-x|| 2 Where x' represents the reconstructed image data, x represents the input image data; at the same time, in order to regularize the encoder, KL divergence is used as another term of the loss function, Loss2=KL(N(μ,σ 2 )|N(0,1)), where N(μ,σ 2 ) is the normal distribution of latent variables in the low-dimensional latent space representation of the input image data, obviously the mean μ, variance σ 2 is the characteristic parameter of the latent variable, N(0,1) is the standard normal distribution, and the total loss function of the process is Loss=Loss1+Loss2, while the network structure of the other encoder E2 is exactly the same as that of encoder E1; Step 3: Perform real-time defect detection on the chip to be tested; The method comprises the following steps: collecting surface images of chips to be tested on an industrial production line in real time, decomposing the collected images of the chips to be tested into images of the same size SxT with NxN pixels, creating multiple image detection tasks, creating a thread for each image detection task, and having multiple threads process image defect detection tasks in parallel; inputting the real-time collected chip surface images into encoder E1 and encoder E2 in the detection network, respectively, so that encoder E1 extracts salient features of normal chip surface images, and the features are represented by a low-dimensional latent space C1, and a reconstructed image is generated by decoder D1; When the input chip surface image Q1 contains defects, the encoder E2 in the network model extracts the defect image features and obtains the low-dimensional latent space feature representation C2 of the image. Note ||C1-C2|| as one of the chip surface defect anomaly score values. The decoder D1 connected to the encoder E2 in the convolutional variational autoencoder network model can automatically remove the defect features while retaining the normal features in the process of generating the reconstructed image. Therefore, the reconstructed image Q2 of the defective chip is subtracted from the input defective chip surface image Q1 to obtain a residual map. The information contained in the residual map can not only determine whether the chip to be tested has defects but also calibrate the defect location. The residual map is digitized to obtain its specific numerical representation ||Q1-Q2||, which is recorded as another defect score value. The above two defect score values ​​are weighted and summed to obtain the final abnormality score calculation formula M=λ1||C1-C2||+λ2||Q1-Q2||. The abnormality score value is compared with the set threshold. If it exceeds the threshold, it is determined that there is a defect, and the defect is identified using the residual map.

2. The chip surface defect parallel detection method based on the improved convolutional variational autoencoder according to claim 1 is characterized in that: In the step 1, a high-definition camera module, a host image processing module, and a lighting module are used to collect and store images, collect surface images of normal chips, decompose the collected normal chip images into multiple images of the same size, create multiple image detection tasks, create a thread for each image detection task, and multiple threads process image defect detection tasks in parallel. In addition, a defect sound and light alarm module is equipped to prompt the detection of defective chips.

Images