Surface defect sample enhancement method based on improved top-hat operation and generative adversarial network
By improving the top hat algorithm and combining with the GAN network, high-quality defect samples are generated, the problem of insufficient sample number in chip defect detection is solved, the detection accuracy and efficiency are improved, and the robustness of the model and the reliability of the detection system are enhanced.
Patent Information
- Application Number
- CN202411296167.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2025-01-10
AI Technical Summary
In the existing chip defect detection methods, due to insufficient sample size, the detection accuracy and low detection efficiency are difficult to effectively identify complex chip defect types.
Using the surface defect sample enhancement method based on improved top hat operation and generation adversarial network, high-quality defect samples are generated by improving the top hat algorithm and combining the GAN network, increasing the number and diversity of defect samples, and optimizing the top hat transformation weight coefficient to Improve detection accuracy.
By generating high-quality defect samples, the robustness and generalization capabilities of the model are improved, the accuracy and reliability of the detection system are enhanced, and the diversity and quality of the training data are ensured.
Smart Images
Figure CN119295853A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a surface defect sample enhancement method based on improved top-hat operation and generative adversarial network. Background Art
[0002] In the semiconductor manufacturing industry, chip quality control is a key link to ensure product reliability and performance. With the rapid development of integrated circuit technology, the integration of chips is getting higher and higher, which also increases the possibility of defects in the manufacturing process. Chip defects come in many forms, such as metal layer fractures, oxide layer defects, contact hole blockage, etc. These defects not only affect the performance of the chip, but may even cause the entire chip to fail. Therefore, the development of efficient and accurate chip defect detection methods is crucial to improving yield and reducing costs. Traditional detection methods rely on manual visual inspection or rule-based automatic detection systems, but these methods are often inefficient and prone to missed detection. With the continuous improvement of detection accuracy requirements, seeking more advanced solutions has become an inevitable trend.
[0003] In recent years, deep learning techniques, especially convolutional neural networks, have achieved remarkable success in image recognition and classification tasks, and these techniques have also been introduced to the field of chip defect detection. Deep learning models can automatically learn features from large amounts of image data without manually designing complex feature extraction algorithms, which enables them to more accurately identify and classify complex chip defect types. However, training high-performance deep learning models requires a large amount of labeled data. In chip defect detection, since certain specific types of defects are rare, the number of samples of these categories in the training data set is very limited. The insufficient number of samples will seriously affect the learning and generalization capabilities of the model, thereby limiting the detection accuracy.
[0004] To address this problem, it is necessary to explore sample data augmentation techniques to increase the diversity and quality of training data. Summary of the invention
[0005] The purpose of the present invention is to provide a surface defect sample enhancement method based on improved top-hat operation and generative adversarial network, aiming to improve the robustness and generalization ability of the model by enhancing the quantity and quality of defect samples, thereby solving the technical problems of insufficient sample quantity, low accuracy and low detection efficiency in existing chip surface defect detection methods. The adopted technical solution is as follows:
[0006] A surface defect sample enhancement method based on improved top-hat operation and generative adversarial network includes the following steps:
[0007] Step 1, collect product images img1 with surface defects and establish a defect image library;
[0008] Step 2: Apply the improved top-hat transform to the image img1 to obtain the enhanced product image img2;
[0009] Step 3: Construct the generator model and discriminator model of the generative adversarial network, and incorporate the weight coefficient of the top-hat transformation into the generator model as a trainable parameter;
[0010] Step 4: Input product image img1 to the generator model to generate product image img3; the discriminator model receives product image img2, and combines product image img2 and generated image img3 to evaluate the difference between the two and the quality of img3;
[0011] Step 5: By comparing the defects in the product image img3 with the defects in the product image img2, the hyperparameters of the generative adversarial network, including the weight coefficient of the top-hat transformation, are adjusted, and the training is repeated;
[0012] Through the above network optimization method, the optimal top-hat transformation weight coefficient for image enhancement can be obtained.
[0013] Preferably, in step 2, the improved top hat transform linearly combines the original image with the results of the two transformations by combining the ring structure improved top hat transform CWTH and the dual structure element improved top hat transform DWTH to enhance the target features and suppress background noise.
[0014] Preferably, in step 2, before combining, the image grayscale values need to be standardized and converted to the (0, 1) interval based on the current maximum grayscale value of the image, so as to enhance the overall image.
[0015] Preferably, in step 2, the calculation method of the improved top hat transformation is: if the grayscale value of the image processed by the CWTH algorithm or the DWTH algorithm is greater than the grayscale value of the original image, select the method corresponding to the maximum grayscale value to perform image enhancement on the point;
[0016] If the grayscale value of this point obtained by both methods is smaller than that of the original image, the local signal-to-noise ratio of the target background is kept in an appropriate range through a linear combination method to process background suppression or detect potential targets.
[0017] Preferably, in step 1, the defect image library includes the following defects: scratches, stains, and missing corners.
[0018] Preferably, the step of image preprocessing is also included before step 2.
[0019] Preferably, the image preprocessing includes image size standardization, histogram equalization, and grayscale adjustment.
[0020] Preferably, the product is a chip.
[0021] Compared with the prior art, the advantages of the present invention are:
[0022] 1. Improved the top hat algorithm and increased the number of defect samples.
[0023] The present invention proposes an improved top-hat algorithm, which effectively enhances the contrast between small-sized defects and the background by standardizing the grayscale value of the image and linearly combining the CWTH algorithm and the DWTH algorithm according to different situations. This improvement makes the small-sized defects that were originally difficult to identify in the original image img1 due to insufficient contrast become more prominent in the processed image img2. Therefore, this method not only helps to identify more defect samples, but also greatly enriches the number and diversity of defect samples in the training data set in this way.
[0024] 2. Combine GAN to optimize the top hat algorithm and improve the quality of defect samples.
[0025] The present invention not only improves the top-hat algorithm itself, but also introduces GAN in the process of generating defect samples. By combining the GAN network, the present invention optimizes the weight coefficient in the top-hat algorithm, so that the generator can generate a more realistic defect image img3.
[0026] At the same time, the optimized discriminator can more accurately distinguish the real image img2 from the synthesized defect image img3, and retain the features of the original image img1, preventing feature distortion during the enhancement process, which helps to improve the authenticity and quality of the generated image. Based on this network structure, the present invention can not only generate high-quality defect samples, but also ensure that these samples are visually very close to the actual defects, thereby improving the accuracy and reliability of the detection system.
[0027] In summary, the present invention not only increases the number of defect samples but also improves the quality of these samples by improving the top-hat algorithm and combining it with the GAN network, thereby providing a more reliable data basis for defect detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is an overall flow chart of the method of the present invention.
[0029] Figure 2 It is a flowchart illustrating the improved top-hat transformation algorithm of the present invention.
[0030] Figure 3 It is a schematic diagram of weight coefficients in the generative adversarial network optimized improved top-hat transformation algorithm of the present invention. DETAILED DESCRIPTION
[0031] The surface defect sample enhancement method based on improved top hat operation and generative adversarial network of the present invention will be described in more detail below in conjunction with the schematic diagram, wherein the preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein, while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.
[0032] like Figure 1 As shown in the figure, a surface defect sample enhancement method based on improved top-hat operation and generative adversarial network is shown in the figure. The main steps are as follows:
[0033] Step 1: Collect product images img1 with surface defects and establish a defect image library.
[0034] First, an image dataset containing surface defects is collected. The dataset uses an industrial camera or other suitable image acquisition equipment to photograph the sample surface to ensure that the collected images have sufficient resolution to clearly display potential surface defects, and the collected images have a unified resolution and color space standard to ensure data consistency.
[0035] These images should contain samples of surface defects of different types and sizes, including but not limited to scratches, stains, and missing corners. The dataset should cover products of different materials, such as metal, plastic, and ceramic. At the same time, the data should be diverse and contain images under different lighting conditions, angles, and backgrounds to improve the generalization ability of the model. Then, before performing the improved top-hat transformation, the images are preprocessed including image size normalization, histogram equalization, and grayscale adjustment.
[0036] Step 2: Perform an improved top-hat transform on image img1 to obtain product image img2.
[0037] The improved top-hat transformation technique is used to enhance the contrast of small-sized defects in the image and make the defect features more prominent.
[0038] Specifically, the CWTH algorithm and the DWTH algorithm are used to process the original captured image img1, and then the grayscale value of the image is standardized to enhance the overall image. Then, the CWTH algorithm and the DWTH algorithm are linearly combined according to different situations, so as to effectively enhance the contrast between small-sized defects and the background. Furthermore, the results after processing by two common CWTH algorithms and DWTH algorithms are judged: if the grayscale value of the image processed by the CWTH algorithm or the DWTH algorithm is greater than the grayscale value of the original image, the method corresponding to the maximum grayscale value is selected for image enhancement; if the grayscale value of this point obtained by the two methods is smaller than the original image, the local signal-to-noise ratio of the target background is kept in an appropriate range through the linear combination method to handle background suppression or detect potential targets.
[0039] Step 3: Use the generative adversarial network (GAN) to optimize and improve the weight coefficients in the top-hat transformation algorithm.
[0040] By optimizing the weight coefficients in the top-hat transformation algorithm through GAN optimization, the optimal parameter settings can be automatically found to generate more realistic defect images. This method not only improves the effect of image enhancement, but also reduces the workload of manual parameter adjustment, ensuring the consistency and reliability of image quality.
[0041] Specifically, a generator model and a discriminator model of a generative adversarial network are constructed, and the weight coefficients of the top-hat transformation are incorporated into the generator model as trainable parameters.
[0042] Input the original product image img1 to the generator model to generate the enhanced defect image img3;
[0043] The discriminator model receives the real enhanced image img2, and combines the real image img2 and the generated image img3 to evaluate the difference between the two and the quality of img3.
[0044] By comparing the defects in the generated image img3 with the defects in the real enhanced image img2, the hyperparameters of the generative adversarial network (including the weight coefficients of the top-hat transformation) are adjusted, and training is repeated until the generated image img3 is as close as possible to the real image img2 and retains features consistent with the original image.
[0045] After training the network to obtain the optimal top-hat transformation weight coefficient, the improved top-hat transformation algorithm and GAN can be used directly for image enhancement, which greatly expands the number of defect samples and ensures image quality.
[0046] Figure 2 The process of the improved top-hat transform algorithm of the present invention is shown. As shown in the figure, in order to fully integrate the advantages of the CWTH algorithm and the DWTH algorithm, we proposed an improved top-hat transform algorithm, which aims to effectively suppress background interference while enhancing target features. The core idea of the algorithm is to achieve the purpose of optimizing the image enhancement effect by linearly combining the original image with the image processed by CWTH and DWTH.
[0047] Specifically, for the preprocessed images, the top-hat transform was first performed using the CWTH and DWTH methods. In order to compare the processing results of the two methods and enhance the overall image, the grayscale values of the results obtained by the two algorithms were normalized, that is, converted to the (0,1) interval. The improved top-hat transform is a linear combination of the processing results of the CWTH and DWTH algorithms and the original image.
[0048] In this process, the results after processing by two common CWTH algorithms and DWTH algorithms are determined:
[0049] If the grayscale value of the image processed by the CWTH algorithm or the DWTH algorithm is greater than the grayscale value of the original image, it is considered that a target element that needs to be enhanced appears, and the maximum grayscale value max(C,D) of the two methods is selected as the enhancement result of the point;
[0050] If the grayscale value of this point obtained by both methods is smaller than that of the original image, it means that neither algorithm has played an enhancement role, and the corresponding point is a background point that needs to be suppressed or the corresponding point is a chip area where faint defects may exist. At this time, the local signal-to-noise ratio of the target area and the background is kept within an appropriate range through the linear combination method to effectively suppress the background or detect faint defects on the chip.
[0051] The linear combination considers the effects of the original image, the image processed by the CWTH algorithm, and the image processed by the DWTH algorithm at the same time:
[0052] α×O+β×C+γ×D(1)
[0053] Among them, α, β, and γ are the weight coefficients in the combination, and O, C, and D correspond to the original image, the image processed by the CWTH algorithm, and the image processed by the DWTH algorithm, respectively.
[0054] To prevent the image from being over-enhanced, the weight coefficient satisfies:
[0055] α+β+γ=1 (2)
[0056] Finally, the output is the image img2 processed by the improved top-hat transformation algorithm.
[0057] In order to obtain the optimal combination of weight coefficients, the generative adversarial network is used to further enhance the image img2 to generate the image img3. Figure 3 This process is demonstrated in .
[0058] In the process of network optimization weight coefficients, we first start from the initial training, randomly generate a set of α, β, and γ, and perform an improved top-hat transformation operation on the preprocessed image img1 to obtain the enhanced image img2. Subsequently, a generative adversarial network is constructed, which consists of a generator G and a discriminator D.
[0059] The task of the generator is to generate an image img3 that looks real, while the discriminator tries to distinguish the generated image img3 from the real image img2 and ensure that the features are consistent with the original image.
[0060] To this end, three loss functions are constructed including the adversarial loss L adv , reconstruction loss Lrec , and the top-hat transformation consistency loss L oth To guide the network's learning process.
[0061] The adversarial loss is used to ensure that the image G(I) generated by the generator can deceive the discriminator D;
[0062] The reconstruction loss ensures that the generated image G(I) remains as similar as possible to the image T(I) after the top-hat transformation;
[0063] The top-hat transformation consistency loss ensures that the features of the generated image G(I) after top-hat transformation are consistent with the original image O(I).
[0064] Preferably, the adversarial loss L adv , reconstruction loss L rec and top-hat transformation consistency loss L oth Binary cross entropy loss, structural similarity index, and root mean square error are used for calculation, respectively.
[0065] The total loss function L is the sum of three loss functions:
[0066] L=L adv +L rec +L oth (3)
[0067] During the training process of the generative adversarial network, the generator G and the discriminator D are iteratively optimized using an alternating training strategy until the loss function converges to the set threshold to learn the optimal set of top-hat transformation weight coefficients. After training, the generative adversarial network model and the weight coefficients in the optimal improved top-hat transformation algorithm learned are saved. These coefficients are then applied to new images to perform an improved top-hat transformation operation to enhance the characteristics of the target area and suppress background noise.
[0068] The obtained enhanced image img2 can be used as the input of the generator G to generate a further enhanced new image img3. This enhancement process can be repeated to generate a series of enhanced image sets, which are used to expand the training data set, thereby improving the generalization ability and robustness of the model.
[0069] Finally, through this iterative optimization mechanism, the present invention not only optimizes the parameter setting of the top-hat transformation, but also generates a large number of high-quality training samples, thereby improving the overall performance of the model.
[0070] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.
Claims
1. A surface defect sample enhancement method based on improved top-hat operation and generative adversarial network, characterized in that: The following steps are involved: Step 1, collect product images img1 with surface defects and establish a defect image library; Step 2: Apply the improved top-hat transform to the image img1 to obtain the enhanced product image img2; Step 3: Construct the generator model and discriminator model of the generative adversarial network, and incorporate the weight coefficient of the top-hat transformation into the generator model as a trainable parameter; Step 4: Input product image img1 to the generator model to generate product image img3; the discriminator model receives product image img2, and combines product image img2 and generated image img3 to evaluate the difference between the two and the quality of img3; Step 5: By comparing the defects in the product image img3 with the defects in the product image img2, adjust the hyperparameters of the generative adversarial network, including the weight coefficient of the top-hat transformation, and repeatedly train to obtain the optimal top-hat transformation weight coefficient for image enhancement.
2. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 1 is characterized in that: In step 2, the improved top-hat transform linearly combines the original image with the results of the two transformations by combining the ring structure improved top-hat transform CWTH and the double structure element improved top-hat transform DWTH to enhance the target features and suppress background noise.
3. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 2 is characterized in that: In step 2, before combining, the image grayscale value needs to be standardized and converted to the (0, 1) interval based on the current image grayscale maximum value, so as to enhance the overall image.
4. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 2 is characterized in that: In step 2, the calculation method of the improved top hat transformation is: if the gray value of the image processed by the CWTH algorithm or the DWTH algorithm is greater than the gray value of the original image, the method corresponding to the maximum gray value is selected to perform image enhancement on the point; If the grayscale value of this point obtained by both methods is smaller than that of the original image, the local signal-to-noise ratio of the target background is kept in an appropriate range through a linear combination method to process background suppression or detect potential targets.
5. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 1, characterized in that: In step 1, the defect image library includes the following defects: Scratches, stains, chips.
6. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 1, characterized in that: Before step 2, an image preprocessing step is also included.
7. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 6 is characterized in that: Image preprocessing includes image size normalization, histogram equalization, and grayscale adjustment.
8. The surface defect sample enhancement method based on improved top-hat operation and generative adversarial network according to claim 1, characterized in that: The product is a chip.