Defect image generation method and device based on generative adversarial network and medium

Abstract

The invention discloses a defect image generation method and device based on a generative adversarial network and a medium. The method comprises the following steps: generating a defect contour image of a chip by improving an SAGAN network; inputting a to-be-generated defect-free image and a defect contour image, determining the position of the defect contour image in the defect-free image, cutting the defect-free image to obtain a cut defect-free image, and extracting the cut defect-free image to obtain a to-be-generated image of the defect contour; inputting a to-be-generated image of the defect contour into the image generation network to obtain a defect difference image, and carrying out difference on the defect difference image and the cut defect-free image to obtain a local defect image; and evaluating the local defect image, and mapping the local defect image conforming to a preset score back to the original defect-free image to obtain a final generated defect image. The problems that the defect sample of the storage chip is difficult to obtain and the pixel-level labeling cost is high can be solved.

Description

Technical Field

[0001] The present invention relates to the field of visual inspection in the storage chip packaging process, and particularly to a defect image generation method, device and medium based on a generative adversarial network. Background Art

[0002] In the packaging process of storage chips, defects have a great impact on production costs. Defect detection can effectively reduce the production costs of chips and improve the quality of chips. In the defect detection of chips, the detection of cracks and scratches is the most important, which directly determines the quality of the final product. Traditional defect detection is often carried out by human eyes, and this detection method has many disadvantages such as low efficiency, high cost and instability. With the development of machine vision and image processing technologies, the manual visual inspection method has gradually been replaced by visual inspection.

[0003] Defect detection based on machine vision can be divided into two methods: based on traditional image processing and based on deep learning. On the surface of products with complex textures such as chips, the method based on traditional image processing is often affected by environmental factors such as illumination, and the detection effect is poor. The defect detection method based on deep learning can accurately locate defects by learning the semantic features of images, but this type of method requires a large number of defect samples to be collected for training before detection. However, in the packaging process of storage chips, on the one hand, the probability of defects occurring is low, and it is difficult to collect a large number of defect samples before actual production. On the other hand, the pixel-level annotation of defect areas is very costly in terms of manpower and material resources. Therefore, it is necessary to develop a defect sample generation algorithm for storage chips to generate a defect data set based on defect-free images, so as to obtain sufficient training samples.

[0004] A defect image generation method disclosed in Chinese invention patent CN119251332A generates a defect mask based on a defect-free image and Gaussian noise, and then generates a defect image based on the defect-free image and the defect mask. However, this scheme requires a defect-free image to obtain the defect mask, and its input is a complete defect-free image. The trained model is only effective for this product. If a new product needs to be generated, the model in the second stage needs to be retrained. Summary of the Invention

[0005] In order to solve at least one of the problems existing in the prior art, the present invention provides a storage chip defect image generation method, system, device and medium based on a two-stage generative adversarial network, which has high robustness and adaptability, can be applied to chips with different backgrounds, and can generate a defect data set based on an unseen defect-free sample.

[0006] In order to achieve the object of the present invention, a defect image generation method based on a generative adversarial network provided by the invention includes the following steps:

[0007] Input random Gaussian noise and generate a defective contour image of the chip through an improved SAGAN network;

[0008] Input the defect-free image to be generated and the defective contour image, determine the position of the defective contour image in the defect-free image through a defect generation position restriction algorithm, crop the defect-free image based on the position to obtain the cropped defect-free image, and extract the cropped defect-free image to obtain the image to be generated of the defective contour;

[0009] Input the image to be generated of the defective contour into an image generation network to obtain a defect difference map, and perform image difference processing on the defect-free image after defect cropping and the difference map to obtain a local defect image;

[0010] Evaluate the local defect image through a defect classification network to obtain a scoring result, and map the local defect image that meets the preset score back to the original defect-free image to obtain the final generated defect image.

[0011] The improved SAGAN network includes a generator and a discriminator. Adaptive discriminator augmentation and r1 regularization are used to optimize the training process. It is pre-trained through a concrete crack dataset, and then fine-tuned using a chip defect dataset to train the generator and discriminator to obtain the trained improved SAGAN network; based on the weights of the generator of the improved SAGAN network with the optimal training effect, input random Gaussian noise to generate a defective contour image.

[0012] Furthermore, perform Blob analysis on the defective contour image generated by the improved SAGAN network to screen and filter out small isolated defective contours.

[0013] Furthermore, the generator includes a multi-layer transposed convolution structure. The last transposed convolution structure includes a transposed convolution layer and an activation function layer, and the other transposed convolution structures include a transposed convolution layer, a batch normalization layer, and an activation function layer; the discriminator includes a multi-layer convolution structure. The last convolution structure includes a convolution layer and an activation function layer, and the other convolution structures include a convolution layer, a batch normalization layer, an activation function layer, and a Dropout layer.

[0014] Furthermore, the improved SAGAN network includes a generator and a discriminator. Adaptive discriminator augmentation and r1 regularization are introduced during the training of the improved SAGAN network, that is, during the training process, the input image of the discriminator undergoes adaptive data augmentation according to the training progress of the generator and the quality of the generated image, and an r1 regularization term is added to the loss function of the discriminator.

[0015] Furthermore, the methods of adaptive discriminator augmentation include reversible augmentation types such as 90-degree rotation, integer displacement, zooming in and out, rotation, and brightness change.

[0016] Further, the improved SAGAN network includes a generator and a discriminator. The CBAM attention mechanism and the Self-Attention mechanism are introduced into the generator. The CBAM attention mechanism is used to generate channel and spatial attention maps through global average pooling and global maximum pooling to enhance the network's attention to key features. The Self-Attention mechanism is used to capture long-range dependence relationships in the image by calculating the similarity relationships between positions in the input feature map.

[0017] Further, the defect generation position limitation algorithm identifies all defect contours in the defect contour image, calculates the minimum bounding rectangle of each defect contour and its relative distance to the image edge, selects the two smallest relative distance directions, and determines the specific position of the defect contour in the defect-free image according to the single-sided or double-sided direction.

[0018] Further, the defect-free image is cropped based on the position to obtain the cropped defect-free image, and the defect-free image after cropping is extracted to obtain the image to be generated for the defect contour, including:

[0019] According to the generation position of a single defect contour image, a local image is cropped from the input defect-free image as the cropped defect-free image; ROI extraction is performed on the cropped defect-free image based on the single defect contour image, and the gray level of the non-contour area is set to 0 to obtain the image to be generated for the defect contour.

[0020] Further, both the improved SAGAN network and the image generation network are trained based on the chip defect dataset. The acquisition method of the chip defect dataset is as follows: Defect images on the chip are acquired and annotated. The defect images, the binary mask images obtained by annotation, and the corresponding defect-free images together constitute the chip defect dataset.

[0021] Further, when training the improved SAGAN network, it is first trained based on the concrete road crack dataset to obtain a pre-trained model, and then, based on the binary annotation mask images in the chip defect dataset, the improved SAGAN network is fine-tuned on the basis of the pre-trained model. The generator and discriminator of the network are trained, and the weights of the generator are saved to obtain the improved SAGAN network for generating the defect contour images of the chip. The stability of the training process is improved by pre-training with the concrete crack dataset and fine-tuning with the chip defect dataset;

[0022] When training the image generation network, first process the chip defect dataset to obtain the defect image where a single defect contour is located, the defect-free image of the corresponding area, and the labeled mask image of the single defect contour; according to the labeled mask image of the single defect contour after cropping, perform ROI extraction on the cropped defect-free image to obtain the cropped defect-free image after extraction; perform image difference processing on the cropped defect-free image and the defect image of the single defect contour, and perform ROI extraction based on the labeled mask image of the defect contour to obtain the difference image after extraction; use the cropped defect-free image after extraction as the input of the image generation network, and the output of the image generation network is the difference image, and train the image generation network.

[0023] In the generation stage, based on the weights of the improved SAGAN network generator with the best training effect, input random Gaussian noise, and the generated defect contour image can be obtained.

[0024] Furthermore, collect the local defect images generated by the image generation network under different backgrounds, label them according to the generation effect, and train the defect classification network.

[0025] Furthermore, based on the optimal defect classification network obtained by training, score the local defect images, and map the local defect images that meet the preset score back to the original defect-free image to obtain the final generated defect image. For example, discard the images with a score lower than 0.6, and map the local defect images with a score higher than or equal to 0.6 back to the original defect-free image to obtain the final generated defect image.

[0026] Furthermore, the defect classification network adopts the EfficientNet-v2 architecture, and the SoftMax function is used in the last layer to score the generation effect of the generated image, and a score threshold is set to screen qualified generated defect images.

[0027] Furthermore, continuously generate defect images to obtain the storage chip defect dataset.

[0028] The defects in the generated defect images include two types: cracks and scratches on the surface of the storage chip, and the generation process has high robustness, is applicable to different chip backgrounds, and can generate high-quality defect images based on unseen defect-free samples.

[0029] The present invention also provides a defect image generation system based on a generative adversarial network, including the following units:

[0030] The defect contour generation unit is used to input random Gaussian noise and generate the defect contour image of the chip through the improved SAGAN network;

[0031] The defective contour to-be-generated and defect-free image acquisition unit is used to input the defect-free image to be generated and the defective contour image, determine the position of the defective contour image in the defect-free image through the defect generation position limitation algorithm, crop the defect-free image based on the position to obtain the cropped defect-free image, and extract the cropped defect-free image to obtain the to-be-generated image of the defective contour;

[0032] The local defective image acquisition unit is used to input the to-be-generated image of the defective contour into the image generation network to obtain the defective difference map, and perform image difference processing on the cropped defect-free image and the defective difference map to obtain the local defective image;

[0033] The defective image generation unit is used to evaluate the local defective image through the defect classification network to obtain a scoring result, and map the local defective image meeting the preset score back to the original defect-free image to obtain the final generated defective image.

[0034] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a defective image generation method based on a generative adversarial network is implemented.

[0035] The present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a defective image generation method based on a generative adversarial network is implemented.

[0036] The present invention provides a defective image generation method for a storage chip based on a two-stage generative adversarial network. The generation process of the defect is decoupled. In the first stage, the generation of the defective contour is realized based on the improved SAGAN network, which improves the quantity and diversity of the generated defective contours. In the second stage, the defective contour obtained in the first stage is converted into a local defective image through the image generation network.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] (1) The improved SAGAN has a better generation effect compared with other generative adversarial networks. Indicators such as FID and KID are superior to networks such as DCGAN, WGAN, and SAGAN. And it is more stable during training when the dataset scale is small, and is not prone to common problems in GAN network training such as overfitting and model collapse.

[0039] (2) The generation quality of the defective image of the storage chip is high. Through scoring by the defect classification network, the results with poor generation effects can be screened out, improving the robustness of the generation process and the quality of the final generation result.

[0040] (3) The generation process is less affected by the chip background, and can generate defect images on unseen chip backgrounds, which is close to the industrial production environment and meets the real needs. Description of the Drawings

[0041] Figure 1 is the overall flowchart of a method for generating storage chip defects based on a two-stage generative adversarial network provided by the present invention

[0042] Figure 2 is a schematic diagram of the network structure of the generator in the improved SAGAN network proposed by the present invention.

[0043] Figure 3 is a schematic diagram of the network structure of the discriminator in the improved SAGAN network in an embodiment of the present invention.

[0044] Figure 4 is a schematic diagram of the network structure of the CBAM attention mechanism in the improved SAGAN network in an embodiment of the present invention.

[0045] Figure 5 is a schematic diagram of the network structure of the Self-Attention self-attention mechanism in the improved SAGAN network in an embodiment of the present invention.

[0046] Figure 6 is a schematic diagram of the training and generation process of the improved SAGAN network in an embodiment of the present invention.

[0047] Figure 7 is the flowchart of the defect generation position limitation algorithm in an embodiment of the present invention. Detailed Embodiments

[0048] To more clearly illustrate the purpose, technical solutions and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and embodiments. The embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] See Figure 1 , a method for generating storage chip defects based on a two-stage generative adversarial network provided by the present invention includes the following steps:

[0050] Step 1: Input random Gaussian noise, and generate a defect contour image of the chip through the improved SAGAN network.

[0051] This step can improve the diversity of defect contours.

[0052] The generative adversarial network (GAN) adopted in the embodiments of the present invention includes two parts: a generator and a discriminator. The generator is used to receive random Gaussian noise as input and generate a defective contour image; the discriminator is used to receive the defective contour image and the real image generated by the generator, and determine whether the source of the input image is a defective contour image or a real image. The optimization goal of the generator is to make the discriminator unable to distinguish the source of the input image, while the goal of the discriminator is to identify the true source of the input image as accurately as possible. Through this adversarial training mechanism, the quality of the generated image by the generator is continuously improved.

[0053] In the embodiments of the present invention, the adopted generative adversarial network is an improved SAGAN network. A CBAM attention mechanism and a Self-Attention self-attention mechanism are added to the generator, and adaptive discriminator augmentation and r1 regularization are added during the training process. This improvement aims to improve the quality of the image generated by the generator and prevent the discriminator from overfitting under the condition of a limited data set, thereby avoiding the phenomenon of mode collapse.

[0054] In some embodiments of the present invention, the generator adopts a 6-layer transposed convolution structure, and the input is Gaussian random noise with a dimension of 100. Each layer of the transposed convolution structure performs upsampling on the input, and finally generates a defective contour image of 128×128. The first five layers of the transposed convolution structure of the generator all include a transposed convolution layer, a batch normalization layer, and a ReLU activation function layer. The 6th layer of the transposed convolution structure includes a transposed convolution layer and an activation function layer. No batch normalization layer is added after the transposed convolution layer of the 6th layer of the transposed convolution structure, and the activation function layer adopts a Tanh function. By adopting the ReLU activation function, the problem of gradient attenuation can be avoided. By using the Tanh function, the output can be normalized, that is, restricted within the range of [-1,1]. During the training process, the real image is also normalized to [-1,1] to align with it.

[0055] Spectral Normalization is applied to all transposed convolution layers in the generator. The role of spectral normalization is to limit the singular value of the convolution kernel in the generator, thereby effectively avoiding the problem of mode collapse and improving the quality of the generated image. The convolution kernel size of the transposed convolution layer is 4×4, and the stride of each layer except the first layer is 1, and the stride of the first layer is 2. After the transposed convolution layers of the 2nd layer and the 4th layer, a CBAM (Convolutional Block Attention Module) attention mechanism is added respectively to enhance the key information in the generated image; after the ReLU activation function layers of the 4th layer and the 5th layer, a Self-Attention self-attention mechanism is added to further improve the details and consistency of the generated image.

[0056] As Figure 4 shown, the CBAM attention mechanism includes two modules: a channel attention module (Channel Attention Module, CAM) and a spatial attention module (Spatial Attention Module, SAM). The channel attention module generates a channel attention map through global average pooling and global max pooling, and then improves the network's attention to important channels through weighted summation; the spatial attention module generates a spatial attention map through global average pooling and global max pooling of the input feature map in the spatial dimension to strengthen the network's focus on key spatial regions. Combining these two modules, the CBAM attention mechanism can effectively improve the generator's ability to express image features.

[0057] As Figure 5 shown, the Self-Attention mechanism captures long-range dependencies in the image by calculating the similarity relationships between the positions of the input feature map. Specifically, the Self-Attention mechanism calculates the similarity between each pixel and other pixels, and passes its context information to each pixel through weighted summation, thereby enhancing the details and local consistency of image generation. In this way, the generator can obtain the context information of the image in a larger range and improve the quality of the generated image.

[0058] As Figure 3 shown, in some embodiments of the present invention, the discriminator adopts a 6-layer convolutional structure. The first layer structure includes a convolutional layer plus a LeakyReLU activation function, and the last layer structure includes a convolutional layer plus a Sigmoid activation function layer. The remaining layers all include a convolutional layer, a batch normalization layer, a LeakyReLU activation function layer, and a Dropout layer. All convolutional layers apply spectral normalization to prevent the discriminator from overfitting. During training, the Dropout layer randomly discards a part of the neurons to further avoid the overfitting problem. The input size of the discriminator is 1x128x128, and the convolutional kernel size is 4×4.

[0059] By using the LeakyReLU activation function, the gradient can be effectively transmitted to avoid gradient disappearance; the Sigmoid activation function is used to normalize the output to the range of 0-1.

[0060] In some embodiments of the present invention, the training steps of the improved SAGAN network include:

[0061] (1) Using the three publicly available concrete road crack datasets, Crack500, CFD, and DeepCrack, as the pre-training datasets, the improved SAGAN network is trained on these datasets, and the generated model is used as the pre-trained model. The reason for choosing the concrete crack dataset for pre-training is that this type of dataset has sufficient samples and certain similarities with chip defects.

[0062] (2) Based on the binary annotation mask images in the chip defect dataset, the improved SAGAN network is fine-tuned on the basis of the pre-trained model.

[0063] This training step aims to enable the generator and discriminator to learn more general feature representations by pre-training on a large and diverse concrete crack dataset, thereby improving the training stability and generalization ability on the limited chip defect dataset. The goal of the pre-training stage is to make the improved SAGAN network converge as much as possible in a complex environment, while the fine-tuning stage helps the improved SAGAN network better adapt to the specific patterns and features of chip defects, avoiding the training instability problem of the GAN network with limited datasets.

[0064] Among them, the acquisition steps of the chip defect dataset include: collecting defective products during the chip production process, capturing images through a camera, then cropping out the chip area as the real defect image, and manually annotating the defect image. The defect images, the binary mask images obtained by annotation, and the corresponding defect-free images together constitute the chip defect dataset. In some embodiments of the present invention, the chip defect dataset contains 281 defect images, the binary mask images obtained by annotation, and the corresponding defect-free images.

[0065] The application object of the embodiments of the present invention is the storage chip image, and the generated defects are two types: cracks and scratches on the chip surface.

[0066] During the process of training the generator and discriminator, in order to further improve the stability and training efficiency of the improved SAGAN network, Adaptive Discriminator Augmentation (ADA) and r1 regularization are introduced.

[0067] Among them, adaptive discriminator enhancement is a method for dynamically adjusting the discriminator's input data augmentation strategy. Specifically, during the training process, the input images of the discriminator will be adaptively augmented according to the training progress of the generator and the quality of the generated images. The core idea is to dynamically adjust the noise and transformation degree of the input data, so that the discriminator can better counter the optimization of the generator when the quality of the generated images is poor, thereby improving the stability of training. This method effectively avoids the problem that the discriminator overfits and causes training to collapse when the quality of the generated images is poor in the initial stage.

[0068] In some embodiments of the present invention, the adaptive data augmentation includes any one or more reversible augmentation types such as 90-degree rotation, integer displacement, zooming in and out, rotation, brightness change, etc.

[0069] As Figure 6 shown, the adaptive discriminator enhancement strategy is only adopted during the training process, and this enhancement strategy is not used during the generation process.

[0070] Among them, r1 regularization is a method for preventing the discriminator from overfitting by controlling the discriminator's gradient behavior, thereby improving the stability of the generator training. Adding an r1 regularization term to the discriminator's loss function can limit the norm of the discriminator's gradient and reduce the risk of overfitting. Specifically, the loss term of r1 regularization can be calculated by the following formula:

[0071]

[0072] Among them, L r1 is the regularized loss term; λ is a hyperparameter used to control the weight of the regularization term. By appropriately selecting λ, the training between the generator and the discriminator can be balanced, so that the model can fully train the discriminator and avoid overfitting; E is the expected value calculation of the samples in the true data distribution p data ; D(x) represents the discriminator's discrimination value for the input image x, is the gradient of the discriminator output with respect to the input image x, is the L2 norm of the gradient.

[0073] The role of this regularization term is to avoid overfitting by restricting the magnitude of the discriminator's gradient, making the update of the discriminator more stable, preventing gradient explosion and training instability. The r1 regularization term is added to the discriminator's loss function, and the final loss function can be expressed as:

[0074] L D = L real + L fake + L r1

[0075] Among them, LD Denotes the total loss of the discriminator, \(L\). real Is the loss of the discriminator on real images, \(L_{r}\). fake Is the loss of the discriminator on generated images, \(L_{f}\). r1 Is the added R1 regularization term. In this way, R1 regularization helps to stabilize the training process, improve the effect of the mutual game between the generator and the discriminator, and thus improve the quality and diversity of the generated images.

[0076] Step 2: Input the defective contour image and the defect-free image to be generated. Determine the position of the defective contour image in the defect-free image through the defect generation position limitation algorithm. Crop the defect-free image based on the position to obtain the cropped defect-free image. Extract the cropped defect-free image to obtain the image to be generated of the defective contour.

[0077] In some embodiments of the present invention, the defective contour image is the defective contour image generated in Step 2, or can also be the defective contour image obtained after filtering and screening the defective contour image generated in Step 2.

[0078] Among them, when performing filtering and screening, perform Blob analysis on the defective contour image obtained in Step 2. The specific operation is as follows: perform threshold segmentation on the defective contour image to obtain a binary image; use the connected component analysis method to identify and extract isolated defective contour regions, and calculate the area of each defective contour region for screening to filter out isolated defective contours with smaller sizes.

[0079] In some embodiments of the present invention, the screening threshold is set to 40 to ensure that only defective contour regions meeting the area requirements are retained.

[0080] The main function of the defect generation position limiting algorithm is to determine the position of the generated defect contour in the image to be generated. In some embodiments of the present invention, considering that the size of the defect contour is 128×128, while the size of the image to be generated is usually much larger than this size. If the image to be generated is scaled to the size of the defect contour, information loss may occur during the scaling process, affecting the image quality; conversely, if the defect contour is scaled to the size of the image to be generated, the original proportion of the defect contour will be changed, thereby affecting the positioning and performance of the defect in the image. In addition, there is no direct correspondence between the defect contour and the size of the image to be generated, so size matching cannot be simply performed. The defect generation position limiting algorithm generates the defect contour based on a suitable area cropped from the image to be generated, rather than based on the complete image. This method can ensure that the generated defect contour maintains a consistent proportional relationship in images of different sizes, avoiding distortion or change in the positional relationship caused by scaling operations. Through this defect generation position limiting algorithm, the position of a single defect contour in the image to be generated can be accurately determined, ensuring the correct proportion and position of the generated defect in the image.

[0081] As Figure 7 shown, the process of the defect generation position limiting algorithm includes:

[0082] (1) Extract all single defect contours in the defect contour image: Find the connected regions of each defect contour in the defect contour image through the findContours operator of OpenCV, then calculate the minimum bounding rectangle of the connected region, and crop the image of the single defect contour according to the minimum bounding rectangle;

[0083] (2) For a single defect contour, subtract the width and height of the defect-free image from the width and height of the single defect contour image respectively. The result is the limiting range of the defect contour in the x and y directions. Beyond this range, the defect contour may exceed the image boundary during generation;

[0084] (3) Since defects mainly occur at the edges, attach the cropped single defect contour image with a smaller size to a certain edge of the defect-free image with a larger size from the inside, calculate the relative distances from the minimum bounding rectangle to the four sides of the defect contour image, sort these relative distances, and obtain the two smallest relative distances and the directions where they are located. Among them, the position of the defect contour can be divided into one of up, down, left, right, upper left, upper right, lower left, and lower right by these two directions.

[0085] If the direction is one of up, down, left, or right, align the single defect contour image with the defect-free image from the inside in this direction; if the direction is up or down, calculate a random value according to the defect contour limit range in the x direction as the position of the single defect contour in the x direction; if the direction is left or right, calculate a random value according to the defect contour limit range in the y direction as the position of the single defect contour image in the y direction;

[0086] If the direction is upper left, upper right, lower left, or lower right, directly generate the single defect contour image at the corner of the defect-free image in this direction;

[0087] (4) Traverse all the defect contours to obtain the generation position of each defect contour.

[0088] Furthermore, according to the generation position of the single defect contour image, crop a local image from the input defect-free image as the cropped defect-free image; perform ROI (Region of Interest) extraction on the cropped defect-free image based on the single defect contour image, and set the gray level of the non-contour area to 0 to obtain the image to be generated of the defect contour.

[0089] Step 3: Use the image to be generated obtained in Step 2 as the input of the image generation network, and perform image difference processing on the cropped defect-free image and the defect difference map output by the image generation network to obtain the generated local defect image.

[0090] In some embodiments of the present invention, the image generation network adopted is the Pix2Pix network. In other embodiments, other image generation networks can also be adopted, such as the Dynamic Pix2Pix network. The Pix2Pix network has relatively low requirements for hardware resources, and it is preferably to use the Pix2Pix network as the image generation network.

[0091] Train the Pix2Pix network through the chip defect data set. The training steps include:

[0092] (1) Process the chip defect data set to obtain the defect image where the single defect contour is located, the defect-free image of the corresponding area, and the annotation mask image of the single defect contour.

[0093] In some embodiments of the present invention, a single defect image in the chip defect data set may contain multiple defect contours, and during the generation process, only a single defect contour is generated each time. For this reason, according to the binary mask image of the defect annotation, uniformly fix the size of the defect image to 256×256, and crop the defect image where the single defect contour is located, the defect-free image of the corresponding area, and the annotation mask image of the single defect contour.

[0094] (2) According to the labeled mask image of a single defect contour after cropping, perform ROI extraction on the cropped defect-free image, that is, set the gray value of the area with 0 in the labeled mask image to 0, and obtain the cropped defect-free image after extraction.

[0095] (3) Perform image difference processing on the cropped defect-free image and the defect image of a single defect contour, and perform ROI extraction based on the labeled mask image of the defect contour to obtain the difference image after extraction;

[0096] (4) Use the cropped defect-free image after extraction as the input of the Pix2Pix network. The output of the Pix2Pix network is the difference image. Train the Pix2Pix network and save the generator weights at different epochs during the training process of the Pix2Pix network.

[0097] The Pix2Pix network is an image conversion network based on a conditional generative adversarial network, which can learn the mapping relationship from the input image to the target image. In the present invention, the Pix2Pix network is used to generate local defect images.

[0098] Using the difference image as the generation target can avoid abnormal transitions at the edges of the generated area, improve the naturalness and accuracy of the generated image, and avoid being affected by the background during the generation process. Therefore, different chip products can be processed without retraining the model.

[0099] In the network generation stage, based on the optimal Pix2Pix network generator weights obtained through training, input the image to be generated obtained in step 3, and then perform image difference on the cropped defect-free image obtained in step 3 and the output of the Pix2Pix network to obtain the generated local defect image.

[0100] Step 4: Loop through each local defect image and evaluate it through the defect classification network. If the classification results of all local defect images are NG (unqualified), return to step 2 to regenerate; if there is a classification result of OK (qualified), map the local defect image back to the original defect-free image to obtain the final generated defect image.

[0101] In some embodiments of the present invention, the generation results are divided into two categories, namely the OK category with good generation effect and the NG category with poor generation effect, where NG is labeled as 0 and OK is labeled as 1. The evaluation criteria for the generation effect include the gray distribution of the defect area, whether the defect morphology is similar to the real defect, etc.

[0102] In some embodiments of the present invention, based on the models at different epochs during the training process of the Pix2Pix network, local defect images are generated on various chip products, and then these local defect images are classified into two categories, OK and NG, through manual annotation, with 3000 images in each category, serving as the dataset for training the defect classification network.

[0103] In some embodiments of the present invention, the defect classification network uses EfficientNet-v2. In other embodiments, other classification networks can also be adopted, such as any one of ResNet, YOLOV5-cls, YOLOV11-cls, and the EfficientNet-v2 network.

[0104] In some embodiments of the present invention, SoftMax is used in the last layer of the defect classification network, and the result after SoftMax is regarded as the generation score of the image. The closer the score value is to 1, the better the generation effect; the closer the score value is to 0, the worse the generation effect.

[0105] In some embodiments of the present invention, 0.6 is set as the evaluation threshold for the generation effect. A blank image with the same size as the defect-free image is generated as the corresponding defect mask image for the subsequent generated defect images; during image generation, each local defect image generated in step 4 is classified. If the result output by the defect classification network is lower than 0.6, the generation result of this time is discarded; if the classification result is greater than or equal to 0.6, according to the coordinates obtained by the defect generation position limitation algorithm in step 2, the local defect image is mapped into the original defect-free image, and at the same time, the corresponding single defect contour image is mapped back into the blank defect mask image. The above-mentioned multiple local defect images are all mapped back into the same defect-free image, and similarly, the single defect contour images are also mapped back into the same mask image.

[0106] The above steps 1 to 3 are executed cyclically, continuously generating defect images and corresponding mask images, and a chip defect dataset can be generated.

[0107] In some embodiments of the present invention, a defect image generation system based on a generative adversarial network is provided, including the following units:

[0108] A defect contour generation unit, configured to input random Gaussian noise and generate a defect contour image of a chip through an improved SAGAN network;

[0109] The defective contour to-be-generated and defect-free image acquisition unit is used to input the defect-free image to be generated and the defective contour image, determine the position of the defective contour image in the defect-free image through the defect generation position limitation algorithm, crop the defect-free image based on the position to obtain the cropped defect-free image, and extract the cropped defect-free image to obtain the to-be-generated image of the defective contour;

[0110] The local defective image acquisition unit is used to input the to-be-generated image of the defective contour into the Pix2Pix network to obtain the defective difference map, and perform image difference processing on the cropped defect-free image and the defective difference map to obtain the local defective image;

[0111] The defective image generation unit is used to evaluate the local defective image through the defect classification network to obtain a scoring result, and map the local defective image that meets the preset score back to the original defect-free image to obtain the final generated defective image.

[0112] In some embodiments of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the foregoing method for generating a defective image based on a generative adversarial network is implemented.

[0113] In some embodiments of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the foregoing method for generating a defective image based on a generative adversarial network is implemented.

[0114] For the method, system, device, and medium for generating a defective image based on a generative adversarial network provided by the embodiments of the present invention, when generating a defective contour image, a defective contour image with different shapes can be directly generated by inputting random Gaussian noise into the improved SAGAN network without inputting a defect-free image. Compared with the existing method, the diversity of the generated defective contours in the embodiments of the present invention is greater; the to-be-generated image of the defective contour in the embodiments of the present invention is the result extracted from the ROI of the defective contour image in the defect-free image. The gray levels of all pixels except the to-be-generated area are 0, and only the defective area is generated. Therefore, it is applicable to the backgrounds of different products and can be applied to different products without retraining the model when applying to different products; in the embodiments of the present invention, the defect classification network is used to score the defect generation result, eliminating the need for manual screening of the generated pictures, and the defect generation process has stronger robustness.

[0115] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating defective images based on a generative adversarial network, characterized in that It includes the following steps: Input random Gaussian noise and generate a defect contour image of the chip through an improved SAGAN network; Input the defect-free image to be generated and the defect contour image, determine the position of the defect contour image in the defect-free image through a defect generation position limit algorithm, crop the defect-free image based on the position to obtain the cropped defect-free image, and extract the cropped defect-free image to obtain the image to be generated of the defect contour; Input the image to be generated of the defect contour into an image generation network to obtain a defect difference map, and perform image difference processing on the defect difference map and the cropped defect-free image to obtain a local defect image; Evaluate the local defect image through a defect classification network to obtain a scoring result, and map the local defect image that meets the preset score back to the original defect-free image to obtain the final generated defect image.

2. The method for generating defective images based on a generative adversarial network according to claim 1, wherein The improved SAGAN network includes a generator and a discriminator. Adaptive discriminator augmentation and r1 regularization are introduced during the training of the improved SAGAN network, that is, during the training process, the input image of the discriminator is adaptively data-augmented according to the training progress of the generator and the quality of the generated image, and an r1 regularization term is added to the loss function of the discriminator.

3. A method for generating defective images based on a generative adversarial network according to claim 1, characterized in that, The improved SAGAN network includes a generator and a discriminator. The CBAM attention mechanism and the Self-Attention self-attention mechanism are introduced into the generator. The CBAM attention mechanism is used to generate channel and spatial attention maps through global average pooling and global max pooling to enhance the network's attention to key features. The Self-Attention self-attention mechanism is used to capture long-range dependencies in the image by calculating the similarity relationship between the positions of the input feature map.

4. A defect image generation method based on a generative adversarial network according to claim 1, characterized in that, The defect generation position limit algorithm identifies all defect contours in the defect contour image, calculates the minimum bounding rectangle of each defect contour and its relative distance to the image edge, selects the two smallest relative distance directions, and determines the specific position of the defect contour in the defect-free image according to the single-sided or double-sided direction.

5. A method for generating defective images based on a generative adversarial network according to claim 1, characterized in that Cropping the defect-free image based on the position to obtain the cropped defect-free image, and extracting the cropped defect-free image to obtain the image to be generated of the defect contour, includes: Crop a local image from the input defect-free image according to the generation position of a single defect contour image as the cropped defect-free image; perform ROI extraction on the cropped defect-free image based on the single defect contour image, and set the gray level of the non-contour area to 0 to obtain the image to be generated of the defect contour.

6. A method for generating defective images based on a generative adversarial network according to any one of claims 1-5, characterized in that, Both the improved SAGAN network and the image generation network are trained based on a chip defect dataset. The acquisition method of the chip defect dataset is: obtain defect images on the chip, annotate the defect images, and the defect images, the binary mask images obtained by annotation, and the corresponding defect-free images together constitute the chip defect dataset.

7. A method for generating defective images based on a generative adversarial network according to claim 6, characterized in that, When training the improved SAGAN network, first train it based on the concrete road crack dataset to obtain a pre-trained model, and then fine-tune the improved SAGAN network based on the binary annotation mask images in the chip defect dataset on the basis of the pre-trained model to obtain an improved SAGAN network for generating defect contour images of the chip; When training the image generation network, first process the chip defect dataset to obtain defect images where individual defect contours are located, defect-free images of corresponding regions, and annotation mask images of individual defect contours; according to the annotation mask images of the individual defect contours after cropping, perform ROI extraction on the cropped defect-free images to obtain the cropped defect-free images after extraction; perform image difference processing on the cropped defect-free images and the defect images of individual defect contours, and perform ROI extraction based on the annotation mask images of the defect contours to obtain the difference images after extraction; use the cropped defect-free images after extraction as the input of the image generation network, and the output of the image generation network is the difference image, and train the image generation network.

8. A defective image generation system based on a generative adversarial network, characterized in that, It includes the following units: A defect contour generation unit, which is used to input random Gaussian noise and generate a defect contour image of the chip through the improved SAGAN network; A unit for obtaining the defect contour to be generated and the cropped defect-free image, which is used to input the defect-free image to be generated and the defect contour image, determine the position of the defect contour image in the defect-free image through the defect generation position limiting algorithm, crop the defect-free image based on the position, obtain the cropped defect-free image, and perform extraction on the cropped defect-free image to obtain the image to be generated of the defect contour; A local defect image acquisition unit, which is used to input the image to be generated of the defect contour into the image generation network to obtain a defect difference map, and perform image difference processing on the cropped defect-free image and the defect difference map to obtain a local defect image; A defect image generation unit, which is used to evaluate the local defect image through a defect classification network to obtain a scoring result, and map the local defect images that meet the preset score back to the original defect-free image to obtain the final generated defect image.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating defect images based on a generative adversarial network according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for generating defect images based on a generative adversarial network according to any one of claims 1 to 7.

Images