An unsupervised chip defect detection method and device
By employing an unsupervised chip defect detection method, a defect mask is generated for image enhancement by combining reconstruction and detection networks, and model parameters are optimized. This solves the problems of training sample dependence and poor detection performance in deep learning chip defect detection, and achieves efficient and accurate chip defect detection.
Patent Information
- Application Number
- CN202311247912.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2043-09-26
AI Technical Summary
In existing technologies, deep learning chip defect detection requires a large number of labeled images for training, and the defect patterns generated by unsupervised noise differ greatly from naturally generated images, resulting in poor detection performance.
An unsupervised chip defect detection method is adopted. The chip image is reconstructed and features are extracted by a reconstruction network model and fused with a detection network model. A defect mask is generated using a noise function for image enhancement. Multiple loss functions are used to optimize model parameters, achieving efficient detection without label training.
It improves the efficiency and accuracy of chip defect detection, reduces reliance on labeled images, enhances the effectiveness and accuracy of model training, and reduces labor costs.
Smart Images
Figure CN117237309B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of defect detection, and more particularly to an unsupervised chip defect detection method and apparatus. Background Technology
[0002] Image recognition detection technology is widely used in the field of defect detection. Traditional optical inspection equipment requires complex parameters to be set for effective detection. With the development of deep learning, data-driven technology has begun to be applied to defect detection.
[0003] However, deep learning-based methods require a large number of training samples, which are labeled images. This necessitates the manual collection of numerous images and the significant manpower required for defect labeling, hindering the widespread adoption of deep learning technology in the field of defect detection.
[0004] Meanwhile, existing technologies that use unsupervised noise to generate defect patterns, which are then added to images, differ significantly from naturally generated defect images, exhibiting obvious edges and appearing jarring in the image. During defect detection, these anomalous regions are located based on these sharp edges, resulting in poor detection performance when facing real defect images. Summary of the Invention
[0005] This invention provides an unsupervised chip defect detection method and apparatus to solve the problem of low accuracy in chip defect detection using current unsupervised methods.
[0006] Firstly, this application provides an unsupervised chip defect detection method, including:
[0007] The process involves acquiring an image of the chip to be detected, inputting the image into a preset reconstruction network model, and then having the reconstruction network model perform several consecutive downsampling processes on the image to be detected to obtain several feature images. After the last downsampling process, the feature images obtained are then subjected to several upsampling processes to obtain a reconstructed chip image. Finally, some feature images are fused together to obtain a feature information map of the reconstructed chip image.
[0008] The chip image to be detected, the reconstructed chip image, and the corresponding feature information map are input into a preset detection network model, so that the detection network model fuses the chip image to be detected with the feature information map and the reconstructed chip image to obtain a defect numerical map, and obtains a chip image marked with the defect location based on the defect numerical map and the chip image to be detected.
[0009] By first reconstructing the chip to be detected using both the reconstruction network model and the detection network model, this method eliminates the need for auxiliary comparison with other chips, thereby improving the efficiency of chip defect detection. Furthermore, the feature images obtained during the chip reconstruction process are used as feature information.Figure 1 The data is then input into the detection network model, which effectively compares the reconstructed image and the image to be detected, and accurately determines the location of defects in the image to be detected. This eliminates the reliance on reconstructed images alone to obtain defect locations, and also prevents situations where low-quality generated defect images lead to low accuracy in actual defect detection during model training.
[0010] Furthermore, the acquisition of the image of the chip to be detected specifically involves:
[0011] An irregular defect mask is attached to a normal chip image to generate a defective chip image; wherein the irregular defect mask is generated by a noise function.
[0012] The image processing method in the image processing database is selected to perform image enhancement on the defective chip image to obtain the chip image to be detected.
[0013] By converting normal chip images into defective chip images based on defect masks and then enhancing these defective images, more realistic defective chip images can be obtained. This makes it easier to use these images for model training, further reducing edge artifacts of the defect mask in the image and improving the accuracy and effectiveness of model training.
[0014] Furthermore, after obtaining the feature information map of the reconstructed chip image, the process also includes:
[0015] The reconstruction loss function of the reconstruction network model is calculated based on the image of the chip to be detected and the image of the chip to be reconstructed, and the parameters in the reconstruction network model are adjusted based on the value of the reconstruction loss function.
[0016] The reconstruction loss function includes a first loss function and a second loss function;
[0017] The first loss function is used to calculate the pixel difference between the chip image to be detected and the reconstructed chip image, and the second loss function is used to calculate the image structure similarity between the chip image to be detected and the reconstructed chip image.
[0018] By using different parameter tuning functions for the reconstruction network model based on two different loss functions, it is possible to ensure that the pixel differences after reconstruction are not significant and that the circuit structure in the chip is not altered.
[0019] Furthermore, the preset reconstruction network model is specifically as follows:
[0020] A downsampling layer and an upsampling layer are set up respectively, wherein a number of convolutional layers are set in both the downsampling layer and the upsampling layer;
[0021] The image of the chip to be detected is downsampled sequentially through each convolutional layer in the downsampling layer to obtain the feature image after each downsampling, and the final feature image is output.
[0022] The feature image output by the downsampling layer is continuously upsampled through each convolutional layer in the upsampling layer to obtain a reconstructed image of the same size as the image of the chip to be detected.
[0023] Furthermore, the process of fusing partial feature images to obtain a feature information map of the reconstructed chip image specifically involves:
[0024] Acquire several feature images and set the size of all feature images to a preset size;
[0025] After convolution and normalization of all feature images, the corresponding feature information maps of several reconstructed chip images are obtained by inputting them into the activation function.
[0026] By incorporating the feature information of the reconstruction process into the detection network model, the model gains more information sources and avoids the problem of directly identifying abnormal regions through obvious edges.
[0027] Furthermore, after the detection network model fuses the image of the chip to be detected with the feature information map and the reconstructed chip image to obtain a defect numerical map, the process includes:
[0028] Based on the defect numerical map and the defect mask data, and combined with a preset detection loss function, the parameters in the detection network model are updated.
[0029] Furthermore, the step of overlaying the defect numerical map with the image of the chip to be inspected and outputting a chip image marked with the defect location also includes:
[0030] Determine whether the maximum value in the defect numerical graph is greater than a preset value;
[0031] If the maximum value in the defect value graph is greater than a preset value, then the chip to be tested is determined to be a damaged chip.
[0032] If the maximum value in the defect value graph is less than or equal to a preset value, then the chip to be tested is determined to be a good chip.
[0033] Furthermore, the step of obtaining a chip image marked with defect locations based on the defect numerical map and the chip image to be inspected specifically involves:
[0034] After normalizing the defect numerical map, multiply it by a preset value to obtain an anomaly score numerical heatmap.
[0035] The anomaly score heatmap is superimposed on the chip image to be inspected to obtain a chip image with the defect location marked.
[0036] Furthermore, the preset value is specifically as follows:
[0037] Several normal chip images are input into the detection network model to obtain the defect numerical map corresponding to the normal chip images;
[0038] A preset value is set based on the maximum value in the defect value graph corresponding to several normal chip images.
[0039] By training with normal chip images to obtain preset values, it is possible not only to accurately identify the location of chip defects, but also to confirm whether the defects are sufficient to damage the chip.
[0040] Secondly, this application provides an unsupervised chip defect detection device, including: an image reconstruction module and an image detection module;
[0041] The image reconstruction module is used to acquire the image of the chip to be detected, input the image of the chip to be detected into a preset reconstruction network model, so that the reconstruction network model performs several consecutive downsampling processes on the image of the chip to be detected to acquire several feature images, performs several upsampling processes on the feature images obtained after the last downsampling process to obtain the reconstructed chip image, and fuses some feature images to obtain the feature information map of the reconstructed chip image.
[0042] The image detection module is used to input the chip image to be detected, the reconstructed chip image, and the corresponding feature information map into a preset detection network model, so that the detection network model fuses the chip image to be detected with the feature information map and the reconstructed chip image to obtain a defect numerical map, and obtains a chip image marked with the defect location based on the defect numerical map and the chip image to be detected.
[0043] Furthermore, the image reconstruction module includes: an image generation unit and an image acquisition unit;
[0044] The image generation unit is used to attach an irregular defect mask to a normal chip image to generate a defective chip image; wherein the irregular defect mask is generated by a noise function;
[0045] The image acquisition unit is used to select an image processing method from the image processing database to perform image enhancement on the defective chip image and acquire the chip image to be inspected.
[0046] Furthermore, the image reconstruction module also includes a reconstruction loss function module:
[0047] The reconstruction loss function module is used to calculate the value of the reconstruction loss function of the reconstruction network model based on the chip image to be detected and the reconstructed chip image, and to adjust the parameters in the reconstruction network model based on the value of the reconstruction loss function.
[0048] The reconstruction loss function includes a first loss function and a second loss function;
[0049] The first loss function is used to calculate the pixel difference between the chip image to be detected and the reconstructed chip image, and the second loss function is used to calculate the image structure similarity between the chip image to be detected and the reconstructed chip image.
[0050] Furthermore, the image reconstruction module includes: a convolutional layer setting unit, a downsampling unit, and an upsampling unit;
[0051] The convolutional layer setting unit is used to set up a downsampling layer and an upsampling layer respectively, wherein both the downsampling layer and the upsampling layer contain a plurality of convolutional layers;
[0052] The downsampling unit is used to sequentially downsample the image of the chip to be detected through each convolutional layer in the downsampling layer, obtain the feature image after each downsampling, and output the final feature image.
[0053] The upsampling unit is used to continuously upsample the feature image output by the downsampling layer through each convolutional layer in the upsampling layer to obtain a reconstructed image of the same size as the image of the chip to be detected.
[0054] Furthermore, the image reconstruction module includes: a feature image selection unit and an information map acquisition unit;
[0055] The feature image selection unit is used to acquire several feature images and set the size of all feature images to a preset size;
[0056] The information map acquisition unit is used to perform convolution on all feature images, normalize them, and input them into an activation function to obtain feature information maps of several corresponding reconstructed chip images.
[0057] Furthermore, the image detection module includes: a detection loss function unit;
[0058] The detection loss function unit is used to update the parameters in the detection network model based on the defect numerical map and the defect mask data, combined with a preset detection loss function.
[0059] Furthermore, the image detection module includes: a first judgment unit and a chip determination unit;
[0060] The first judgment unit is used to determine whether the maximum value in the defect numerical graph is greater than a preset value;
[0061] The chip determination unit is used to determine that the chip to be tested is a damaged chip when the first judgment unit determines that the maximum value in the defect value graph is greater than a preset value;
[0062] The chip determination unit is further configured to determine the chip to be tested as a good chip when the first judgment unit determines that the maximum value in the defect value graph is less than or equal to a preset value.
[0063] Furthermore, the image detection module includes: a heatmap generation unit and a defect image acquisition unit;
[0064] The heat map generation unit is used to normalize the defect numerical map and multiply it by a preset value to obtain an anomaly score numerical heat map.
[0065] The defect image acquisition unit is used to overlap the abnormal score heatmap with the chip image to be inspected to obtain a chip image marked with the defect location.
[0066] Furthermore, the first judgment unit includes: a normal image training unit and a preset value setting unit;
[0067] The normal image training unit is used to input several normal chip images into the detection network model to obtain the defect numerical map corresponding to the normal chip image.
[0068] The preset value setting unit is used to set a preset value based on the maximum value in the defect value map corresponding to several normal chip images.
[0069] By first reconstructing the chip to be detected using both the reconstruction network model and the detection network model, this method eliminates the need for auxiliary comparison with other chips, thereby improving the efficiency of chip defect detection. Furthermore, the feature images obtained during the chip reconstruction process are used as feature information. Figure 1 The data is then input into the detection network model, which effectively compares the reconstructed image and the image to be detected, and accurately determines the location of defects in the image to be detected. This eliminates the reliance on reconstructed images alone to obtain defect locations, and also prevents situations where low-quality generated defect images lead to low accuracy in actual defect detection during model training. Attached Figure Description
[0070] Figure 1 : A schematic flowchart of an embodiment of an unsupervised chip defect detection method provided by the present invention;
[0071] Figure 2: A schematic diagram of chip image detection for an unsupervised chip defect detection method provided by the present invention;
[0072] Figure 3 : A module structure diagram of an embodiment of an unsupervised chip defect detection device provided by the present invention;
[0073] Figure 4 : A partial structural diagram of an image reconstruction module provided by the present invention;
[0074] Figure 5 : A partial structural diagram of another image reconstruction module provided by the present invention;
[0075] Figure 6 : A partial structural schematic diagram of another image reconstruction module provided by the present invention;
[0076] Figure 7 : A partial structural diagram of an image detection module provided by the present invention;
[0077] Figure 8 : This is a structural diagram of another image detection module provided by the present invention. Detailed Implementation
[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0079] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.
[0080] It should be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0081] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.
[0082] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.
[0083] Because deep learning methods require a large number of training samples, which are labeled images, a large number of images need to be collected manually, and a lot of manpower is needed to label defects, which hinders the popularization of deep learning technology in the field of defect detection. Therefore, this invention generates a training set to train a neural network model using only normal images as the training set, thereby achieving unsupervised defect detection of chips.
[0084] Example 1
[0085] Please refer to Figure 1 The present invention provides an unsupervised chip defect detection method, comprising steps S1 to S2, the specific steps of which are as follows:
[0086] Step S1: Obtain the image of the chip to be detected, input the image of the chip to be detected into a preset reconstruction network model, so that the reconstruction network model performs several consecutive downsampling processes on the image of the chip to be detected to obtain several feature images, performs several upsampling processes on the feature images obtained after the last downsampling process to obtain the reconstructed chip image, and fuses some feature images to obtain the feature information map of the reconstructed chip image.
[0087] Step S2: Input the chip image to be detected, the reconstructed chip image, and the corresponding feature information map into a preset detection network model, so that the detection network model fuses the chip image to be detected with the feature information map and the reconstructed chip image to obtain a defect numerical map, and obtains a chip image marked with the defect location based on the defect numerical map and the chip image to be detected.
[0088] By first reconstructing the chip to be detected using both the reconstruction network model and the detection network model, this method eliminates the need for auxiliary comparison with other chips, thereby improving the efficiency of chip defect detection. Furthermore, the feature images obtained during the chip reconstruction process are used as feature information. Figure 1 The data is then input into the detection network model, which effectively compares the reconstructed image and the image to be detected, and accurately determines the location of defects in the image to be detected. This eliminates the reliance on reconstructed images alone to obtain defect locations, and also prevents situations where low-quality generated defect images lead to low accuracy in actual defect detection during model training.
[0089] In one embodiment, the acquisition of the image of the chip to be detected when setting the training set is specifically as follows:
[0090] An irregular defect mask is attached to a normal chip image to generate a defective chip image; wherein the irregular defect mask is generated by a noise function.
[0091] The image processing method in the image processing database is selected to perform image enhancement on the defective chip image to obtain the chip image to be detected.
[0092] In an alternative embodiment, an irregular mask is generated using Perlin noise and imported into a DTD dataset image as a defect mask attached to a normal chip image; wherein the defect mask is rotated.
[0093] In an optional embodiment, an image processing method from the image processing database is selected to perform image enhancement on the defective chip image to obtain an image of the chip to be inspected, specifically:
[0094] The defective chip image is augmented using a data augmentation method that randomly selects four processing methods from the following ten imgaug libraries and executes them simultaneously: adjusting image brightness, multiplying each pixel by a number, adjusting image sharpness, increasing or decreasing hue and saturation by random values, inverting all pixel values above a threshold, reducing the number of bits in each color channel (resulting in color shrinkage with each bit change), inverting all values in the image, adjusting contrast by cutting the "p%" of the lowest / highest histogram values, equalizing the image histogram, and rotating the image by -45 to 45 degrees.
[0095] By converting normal chip images into defective chip images based on defect masks and then enhancing these defective images, more realistic defective chip images can be obtained. This makes it easier to use these images for model training, further reducing edge artifacts of the defect mask in the image and improving the accuracy and effectiveness of model training.
[0096] In an optional embodiment, when using the present invention, the chip image to be detected is obtained by directly obtaining a real chip image as the chip image to be detected.
[0097] In one embodiment, obtaining the image of the chip to be detected specifically involves: obtaining the chip strip image and performing grayscale processing on it; dividing the grayscale processed strip image into units, collecting normal images, and establishing a training set.
[0098] Furthermore, after obtaining the feature information map of the reconstructed chip image, the process also includes:
[0099] The reconstruction loss function of the reconstruction network model is calculated based on the image of the chip to be detected and the image of the chip to be reconstructed, and the parameters in the reconstruction network model are adjusted based on the value of the reconstruction loss function.
[0100] The reconstruction loss function includes a first loss function and a second loss function;
[0101] The first loss function is used to calculate the pixel difference between the chip image to be detected and the reconstructed chip image, and the second loss function is used to calculate the image structure similarity between the chip image to be detected and the reconstructed chip image.
[0102] In an optional embodiment, the first loss function MSE is specifically:
[0103]
[0104] Among them, L MSE Let I be the loss function, and let I be the image of the chip to be detected. r To reconstruct the chip image, m is the number of pixels in the chip image to be detected;
[0105] The number of pixels in the chip image to be detected is the same as that in the reconstructed chip image.
[0106] In an optional embodiment, the second loss function SSIM is specifically:
[0107]
[0108] Among them, L ssim Let I be the loss function, and let I be the image of the chip to be detected. r To reconstruct the chip image, ssim is the structural similarity function, and H and W are the height and width of the chip image to be detected, respectively.
[0109] The height and width of the chip image to be detected are the same as those of the reconstructed chip image.
[0110] By using different parameter tuning functions for the reconstruction network model based on two different loss functions, it is possible to ensure that the pixel differences after reconstruction are not significant and that the circuit structure in the chip is not altered.
[0111] MSE only calculates the differences between individual pixels, failing to connect the pixels together, resulting in a lack of structural integrity in the reconstructed image. SSIM, on the other hand, calculates image differences from three aspects: brightness, contrast, and structure. Combining MSE (which focuses on local features) and SSIM (which focuses on global features) as loss functions to train the reconstruction network model can improve the quality of reconstructed chip images.
[0112] Furthermore, the preset reconstruction network model is specifically as follows:
[0113] A downsampling layer and an upsampling layer are set up respectively, wherein a number of convolutional layers are set in both the downsampling layer and the upsampling layer;
[0114] The image of the chip to be detected is downsampled sequentially through each convolutional layer in the downsampling layer to obtain the feature image after each downsampling, and the final feature image is output.
[0115] The feature image output by the downsampling layer is continuously upsampled through each convolutional layer in the upsampling layer to obtain a reconstructed image of the same size as the image of the chip to be detected.
[0116] In an optional embodiment, the reconstruction network model is a Unet network based on pure convolution, with the following structure: it consists of 9 modules, each module is composed of two 3*3 convolutions, each convolution is processed by normalization and ReLU activation function, the first 5 modules perform downsampling processing on the chip image to be detected, the last 4 modules perform upsampling processing on the chip image to be detected, and finally the 3*3 convolution restores the number of channels of the chip image to be detected to be consistent with the input, and outputs the reconstructed chip image.
[0117] Furthermore, the process of fusing partial feature images to obtain a feature information map of the reconstructed chip image specifically involves:
[0118] Acquire several feature images and set the size of all feature images to a preset size;
[0119] In an optional embodiment, several non-adjacent feature images in the upsampling layer are selected. In one embodiment, the 3rd, 5th, and 8th feature images in the upsampling of the reconstruction network model are selected, and their sizes are changed to the size of the chip image to be detected.
[0120] After convolution and normalization of all feature images, the corresponding feature information maps of several reconstructed chip images are obtained by inputting them into the activation function.
[0121] In an optional embodiment, a 1*1 convolutional layer is used to convolve all feature images to integrate the information between the channels of each feature image into three channels. Then, each feature image is normalized and processed by the ReLU activation function to output the feature information map of several corresponding reconstructed chip images.
[0122] By incorporating the feature information of the reconstruction process into the detection network model, the model gains more information sources and avoids the problem of directly identifying abnormal regions through obvious edges.
[0123] Furthermore, after the detection network model fuses the image of the chip to be detected with the feature information map and the reconstructed chip image to obtain a defect numerical map, the process includes:
[0124] Based on the defect numerical map and the defect mask data, and combined with a preset detection loss function, the parameters in the detection network model are updated.
[0125] In an optional embodiment, the detection loss function is specifically:
[0126] FL(p t )=-α t (1-p t ) γ log(p t );
[0127] In the formula, FL is the Focal loss function with weighted cross-entropy loss, and α t The weights of the normal region can be controlled, (1-p) t ) γ It can be used to improve the loss in hard-to-classify regions.
[0128] During training, the Focal loss function can control the weight of negative sample loss, guiding the model gradient to update in the correct direction, enabling the model to learn more effective information and detect chip defects more accurately.
[0129] Furthermore, the step of overlaying the defect numerical map with the image of the chip to be inspected and outputting a chip image marked with the defect location also includes:
[0130] Determine whether the maximum value in the defect numerical graph is greater than a preset value;
[0131] If the maximum value in the defect value graph is greater than a preset value, then the chip to be tested is determined to be a damaged chip.
[0132] If the maximum value in the defect value graph is less than or equal to a preset value, then the chip to be tested is determined to be a good chip.
[0133] Furthermore, the step of obtaining a chip image marked with defect locations based on the defect numerical map and the chip image to be inspected specifically involves:
[0134] After normalizing the defect numerical map, multiply it by a preset value to obtain an anomaly score numerical heatmap.
[0135] In an optional embodiment, the defect numerical map is softmax processed in the channel dimension and multiplied by 255, and the value in the second channel is taken to obtain the anomaly score heatmap.
[0136] like Figure 2 As shown, the abnormal score heatmap is superimposed on the chip image to be inspected to obtain a chip image with the defect location marked.
[0137] Furthermore, the preset value is specifically as follows:
[0138] Several normal chip images are input into the detection network model to obtain the defect numerical map corresponding to the normal chip images;
[0139] A preset value is set based on the maximum value in the defect value graph corresponding to several normal chip images.
[0140] By training with normal chip images to obtain preset values, it is possible not only to accurately identify the location of chip defects, but also to confirm whether the defects are sufficient to damage the chip.
[0141] Example 2
[0142] Please refer to Figure 3 This is a module structure diagram of an embodiment of an unsupervised chip defect detection device provided by the present invention.
[0143] An unsupervised chip defect detection device includes: an image reconstruction module 310 and an image detection module 320;
[0144] The image reconstruction module 310 is used to acquire the image of the chip to be detected, input the image of the chip to be detected into a preset reconstruction network model, so that the reconstruction network model performs several consecutive downsampling processes on the image of the chip to be detected to obtain several feature images, performs several upsampling processes on the feature images obtained after the last downsampling process to obtain the reconstructed chip image, and fuses some feature images to obtain the feature information map of the reconstructed chip image.
[0145] The image detection module 320 is used to input the chip image to be detected, the reconstructed chip image and the corresponding feature information map into a preset detection network model, so that the detection network model fuses the chip image to be detected with the feature information map and the reconstructed chip image to obtain a defect numerical map, and obtains a chip image marked with the defect location based on the defect numerical map and the chip image to be detected.
[0146] Furthermore, the image reconstruction module 310 includes: an image generation unit 311 and an image acquisition unit 312; as shown Figure 4 .
[0147] The image generation unit 311 is used to attach an irregular defect mask to a normal chip image to generate a defective chip image; wherein the irregular defect mask is generated by a noise function;
[0148] The image acquisition unit 312 is used to select an image processing method from the image processing database to perform image enhancement on the defective chip image and acquire the chip image to be detected.
[0149] Furthermore, the image reconstruction module 310 also includes a reconstruction loss function module 313:
[0150] The reconstruction loss function module 313 is used to calculate the value of the reconstruction loss function of the reconstruction network model based on the chip image to be detected and the reconstructed chip image, and to adjust the parameters in the reconstruction network model based on the value of the reconstruction loss function.
[0151] The reconstruction loss function includes a first loss function and a second loss function;
[0152] The first loss function is used to calculate the pixel difference between the chip image to be detected and the reconstructed chip image, and the second loss function is used to calculate the image structure similarity between the chip image to be detected and the reconstructed chip image.
[0153] Furthermore, the image reconstruction module 310 includes: a convolutional layer setting unit 314, a downsampling unit 315, and an upsampling unit 316; as shown below. Figure 5 .
[0154] The convolutional layer setting unit 314 is used to set up a downsampling layer and an upsampling layer respectively, wherein both the downsampling layer and the upsampling layer contain a plurality of convolutional layers;
[0155] The downsampling unit 315 is used to sequentially downsample the image of the chip to be detected through each convolutional layer in the downsampling layer, obtain the feature image after each downsampling, and output the final feature image.
[0156] The upsampling unit 316 is used to continuously upsample the feature image output by the downsampling layer through each convolutional layer in the upsampling layer to obtain a reconstructed image of the same size as the image of the chip to be detected.
[0157] Furthermore, the image reconstruction module 310 includes: a feature image selection unit 317 and an information map acquisition unit 318; as shown below. Figure 6 .
[0158] The feature image selection unit 317 is used to acquire a number of feature images and set the size of all feature images to a preset size;
[0159] The information map acquisition unit 318 is used to perform convolution on all feature images and then normalize them, and input them into an activation function to obtain feature information maps of several corresponding reconstructed chip images.
[0160] Furthermore, the image detection module 320 includes: a detection loss function unit 321;
[0161] The detection loss function unit 321 is used to update the parameters in the detection network model based on the defect numerical map and the data of the defect mask, combined with a preset detection loss function.
[0162] Furthermore, the image detection module 320 includes: a first judgment unit 322 and a chip determination unit 323; as shown Figure 7 .
[0163] The first judgment unit 322 is used to determine whether the maximum value in the defect value graph is greater than a preset value;
[0164] The chip determination unit 323 is used to determine that the chip to be tested is a damaged chip when the first judgment unit 322 determines that the maximum value in the defect value graph is greater than a preset value.
[0165] The chip determination unit 323 is further configured to determine the chip to be tested as a good chip when the first judgment unit 322 determines that the maximum value in the defect value graph is less than or equal to a preset value.
[0166] Furthermore, the image detection module 320 includes: a heatmap generation unit 324 and a defect image acquisition unit 325; such as Figure 8 .
[0167] The heat map generation unit 324 is used to normalize the defect numerical map and multiply it by a preset value to obtain an anomaly score numerical heat map.
[0168] The defect image acquisition unit 325 is used to overlap the abnormal score heat map with the chip image to be detected to obtain a chip image marked with the defect location.
[0169] Furthermore, the first judgment unit 322 includes: a normal image training unit 3221 and a preset value setting unit 3222;
[0170] The normal image training unit 3221 is used to input several normal chip images into the detection network model to obtain the defect numerical map corresponding to the normal chip image.
[0171] The preset value setting unit 3222 is used to set a preset value based on the maximum value in the defect value map corresponding to several normal chip images.
[0172] By first reconstructing the chip to be detected using both the reconstruction network model and the detection network model, the method eliminates the need for auxiliary comparison with other chips, thereby improving the efficiency of chip defect detection. Furthermore, the feature images obtained during the chip reconstruction process are used as feature information. Figure 1The data is then input into the detection network model, which effectively compares the reconstructed image and the image to be detected, and accurately determines the location of defects in the image to be detected. This eliminates the reliance on reconstructed images alone to obtain defect locations, and also prevents situations where low-quality generated defect images lead to low accuracy in actual defect detection during model training.
[0173] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. An unsupervised chip defect detection method, characterized in that, The method comprises the following steps: acquiring a chip image to be detected, inputting the chip image to be detected into a preset reconstruction network model, so that the reconstruction network model performs continuous times of down-sampling processing on the chip image to be detected to acquire a plurality of feature images, performing a plurality of times of up-sampling processing on the feature image obtained after the last time of down-sampling processing to obtain a reconstructed chip image, and fusing part of the feature images to obtain feature information maps of the reconstructed chip image; inputting the chip image to be detected, the reconstructed chip image and the corresponding feature information maps into a preset detection network model, so that the detection network model fuses the chip image to be detected, the feature information maps and the reconstructed chip image to acquire a defect value map, and acquires a chip image marked with a defect position according to the defect value map and the chip image to be detected; wherein, the preset reconstruction network model is specifically: a down-sampling layer and an up-sampling layer are respectively arranged, wherein a plurality of convolution layers are arranged in the down-sampling layer and the up-sampling layer; the chip image to be detected is sequentially down-sampled through the convolution layers in the down-sampling layer to obtain a feature image after each time of down-sampling, and the feature image finally obtained is output; the feature image output by the down-sampling layer is sequentially up-sampled through the convolution layers in the up-sampling layer to acquire a reconstructed image with the same size as the chip image to be detected; the feature information maps of the reconstructed chip image are obtained by fusing part of the feature images, specifically: a plurality of feature images are acquired, and the size of all the feature images is set to a preset size; all the feature images are respectively subjected to convolution, normalization and input into an activation function to obtain a plurality of feature information maps corresponding to the reconstructed chip image; the chip image marked with the defect position is obtained according to the defect value map and the chip image to be detected, specifically: the defect value map is multiplied by a preset value after normalization to obtain an abnormal score value heat map; the abnormal score value heat map is overlapped with the chip image to be detected to obtain the chip image marked with the defect position.
2. The unsupervised chip defect detection method of claim 1, wherein, the chip image to be detected is acquired, specifically: an irregular defect mask is attached to a normal chip image to generate a defect chip image; wherein the irregular defect mask is generated by a noise function; an image processing method in the image processing database is selected to perform image enhancement on the defect chip image to acquire the chip image to be detected.
3. The unsupervised chip defect detection method of claim 2, wherein, after the feature information maps of the reconstructed chip image are obtained, the following steps are further included: calculating a value of a reconstruction loss function of the reconstruction network model according to the chip image to be detected and the reconstructed chip image, and adjusting parameters in the reconstruction network model according to the value of the reconstruction loss function; wherein, the reconstruction loss function comprises a first loss function and a second loss function; the first loss function is used for calculating pixel differences between the chip image to be detected and the reconstructed chip image, and the second loss function is used for calculating image structural similarities between the chip image to be detected and the reconstructed chip image.
4. The unsupervised chip defect detection method of claim 2, wherein, after the detection network model fuses the chip image to be detected, the feature information maps and the reconstructed chip image to acquire the defect value map, the following steps are included: According to data of the defect value map and the defect mask, and in combination with a preset detection loss function, parameters in the detection network model are updated.
5. The unsupervised chip defect detection method of claim 1, wherein, The defect value map is overlapped with the chip image to be detected, and a chip image marked with a defect position is output. It is judged whether a maximum value in the defect value map is greater than a preset value. If the maximum value in the defect value map is greater than the preset value, it is determined that the chip to be detected is a damaged chip. If the maximum value in the defect value map is less than or equal to the preset value, it is determined that the chip to be detected is a good chip.
6. The unsupervised chip defect detection method of claim 5, wherein, The preset value is specifically: A plurality of normal chip images are input into a detection network model, and a defect value map corresponding to the normal chip images is obtained. According to a maximum value in the defect value map corresponding to the plurality of normal chip images, a preset value is set.
7. An unsupervised chip defect detection apparatus, characterized by, It includes: An image reconstruction module and an image detection module. The image reconstruction module is configured to obtain a chip image to be detected, input the chip image to be detected into a preset reconstruction network model, so that the reconstruction network model performs continuous times of down-sampling processing on the chip image to be detected to obtain a plurality of feature images, performs a plurality of times of up-sampling processing on a feature image obtained after the last time of down-sampling processing to obtain a reconstructed chip image, fuses part of the feature images to obtain feature information of the reconstructed chip image, and acquires a feature information map corresponding to the reconstructed chip image. The image detection module is configured to input the chip image to be detected, the reconstructed chip image, and the corresponding feature information map into a preset detection network model, so that the detection network model fuses the chip image to be detected with the feature information map and the reconstructed chip image to obtain a defect value map, and acquires a chip image marked with a defect position according to the defect value map and the chip image to be detected. The image reconstruction module includes a convolution layer setting unit, a down-sampling unit, and an up-sampling unit. The convolution layer setting unit is configured to set a down-sampling layer and an up-sampling layer respectively, wherein a plurality of convolution layers are arranged in the down-sampling layer and the up-sampling layer. The down-sampling unit is configured to sequentially perform down-sampling on the chip image to be detected through the convolution layers in the down-sampling layer to obtain a feature image after each time of down-sampling, and output a feature image obtained after the last time of down-sampling. The up-sampling unit is configured to sequentially perform up-sampling on the feature image output by the down-sampling layer through the convolution layers in the up-sampling layer to obtain a reconstructed image with a same size as the chip image to be detected. The image reconstruction module includes a feature image selection unit and an information map acquisition unit. The feature image selection unit is configured to obtain a plurality of feature images, and set a size of all the feature images to a preset size. The information map acquisition unit is configured to perform convolution and normalization on all the feature images respectively, and input an activation function to obtain a plurality of feature information maps corresponding to the reconstructed chip image. The image detection module comprises a heat map generation unit and a defect image acquisition unit; the heat map generation unit is configured to multiply a defect value map by a preset value after normalization to obtain an abnormal score value heat map; and the defect image acquisition unit is configured to superimpose the abnormal score value heat map on a chip image to be detected to obtain a chip image marked with a defect position.
8. The unsupervised chip defect detection apparatus of claim 7, wherein, The image reconstruction module comprises an image generation unit and an image acquisition unit. The image generation unit is configured to attach an irregular defect mask to a normal chip image to generate a defect chip image; wherein the irregular defect mask is generated by a noise function. The image acquisition unit is configured to select an image processing method in the image processing database to perform image enhancement on the defect chip image to obtain a chip image to be detected.
9. The unsupervised chip defect detection apparatus of claim 8, wherein, Further comprising a reconstruction loss function module: The reconstruction loss function module is configured to calculate a value of a reconstruction loss function of a reconstruction network model according to the chip image to be detected and a reconstructed chip image, and adjust parameters in the reconstruction network model according to the value of the reconstruction loss function. The reconstruction loss function comprises a first loss function and a second loss function. The first loss function is configured to calculate pixel difference between the chip image to be detected and the reconstructed chip image, and the second loss function is configured to calculate image structural similarity between the chip image to be detected and the reconstructed chip image.
10. The unsupervised chip defect detection apparatus of claim 8, wherein, The image detection module comprises a detection loss function unit. The detection loss function unit is configured to update parameters in the detection network model according to data of the defect value map and the defect mask, and in combination with a preset detection loss function.
11. The unsupervised chip defect detection apparatus of claim 7, wherein, The image detection module comprises a first judgment unit and a chip determination unit. The first judgment unit is configured to determine whether a maximum value in the defect value map is greater than a preset value. The chip determination unit is configured to determine that the chip to be detected is a damaged chip when the first judgment unit determines that the maximum value in the defect value map is greater than the preset value. The chip determination unit is further configured to determine that the chip to be detected is a good chip when the first judgment unit determines that the maximum value in the defect value map is less than or equal to the preset value.
12. The unsupervised chip defect detection apparatus of claim 11, wherein, The first judgment unit comprises a normal image training unit and a preset value setting unit. The normal image training unit is configured to input a plurality of normal chip images into a detection network model to obtain defect value maps corresponding to the normal chip images. The preset value setting unit is configured to set a preset value according to a maximum value in the defect value maps corresponding to the plurality of normal chip images.
Citation Information
Patent Citations
Unsupervised defect detection model training method and defect detection method
CN114693685A