Chip defect sample generation method and device

Abstract

The invention relates to the technical field of semiconductor industrial quality inspection, and provides a chip defect sample generation method and equipment, which are used for improving the reasonability and fineness of defect generation. According to the method, each area segmented from a normal chip image is marked, each area identifier is associated with first spatial information of the corresponding area to provide spatial constraint when defects are generated in the rear area, and in the defect generation process, the defects are generated based on a mask image corresponding to the area where the defects need to be generated. Defect generation is forced to be carried out in the corresponding area, so that the defect conforms to the chip structure, the reasonability of the defect is improved, meanwhile, quantized fine defects are generated in the corresponding area through the defect quantization parameters and the defect surface description, the defect size is controlled, and the accuracy of follow-up defect detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of quality inspection in the semiconductor industry, and provides a chip defect sample generation method and device. BACKGROUND

[0002] In the production and manufacturing process of LED chips, defect detection is a key link to ensure product quality, and accurate identification and evaluation of chip defects are of great significance to improve product reliability and reduce production cost. With the upgrading and iteration of production line technology, yield is gradually improved, and defect samples become more difficult to collect. The sample of some defect types is only in the single digit, far from the optimal number required for model learning, and fewer samples will cause data imbalance in downstream detection tasks, affecting the accuracy of detection.

[0003] At present, the existing defect sample generation methods mainly fall into the following three categories:

[0004] 1. Traditional image processing generation method, which generates new defect samples by editing and expanding LED chip images through regularized operations, including but not limited to geometric transformation based on pixel space, image perturbation, image pasting, morphological operation, and feature fusion based on feature space (i.e. generating by probability through image high-level features and statistical distribution), such as copy-paste algorithm.

[0005] However, the defect samples generated by the above method lack diversity and flexibility, are difficult to simulate complex real scenes, and lack authenticity, which may introduce artificial artifacts. Therefore, when facing new and complex detection tasks, the model may have poor generalization ability.

[0006] 2. Generation method based on graphics rendering, which constructs a three-dimensional scene through a rendering engine (such as UE, Unity, Blender, etc.) or combines Nerf+3D Gaussian method, and renders 3D->2D labeled defect images or videos.

[0007] However, the above method generates good results in natural scenes, but has poor universality in medical or industrial subfields.

[0008] 3. Generation method based on deep learning model, which uses deep learning algorithms (such as Generative Adversarial Network (GAN), Diffusion Models, Variational Autoencoder (VAE), etc.) to automatically learn data distribution and generate high-fidelity and diversified defect samples. Among them, GAN and diffusion model dominate in the field of image generation.

[0009] However, in the field of LED chip quality inspection, the specific area (such as the electrode area) and the defect size will affect the quality inspection result. For example, when the area of the dirt in the electrode area is less than 1 / 4 of the area of the region, it is determined to be a good product, and when the area of the dirt is greater than 1 / 4 of the area of the region, it is determined to be a defective product. The above method usually generates defects for the entire image, and it is difficult to accurately control the specific area of the defect, and it is impossible to accurately control the defects with different determination results. This results in that the generated defect samples are difficult to simulate various possible chip defects in the real scene, thereby affecting the detection result.

[0010] Therefore, generating fine quantifiable defect samples has important research significance for improving the quality of LED chip quality inspection. SUMMARY

[0011] Embodiments of the present application provide a chip defect sample generation method and device for improving the rationality and fineness of chip defects.

[0012] In a first aspect, embodiments of the present application provide a method for generating a chip defect sample, comprising:

[0013] performing region segmentation on a normal chip image to obtain an initial mask image;

[0014] According to a pre-set chip structure image, each region contained in the initial mask image is marked with a region identifier, and each region identifier is associated with first spatial information of the corresponding region; wherein the first spatial information is used as a spatial constraint of the defect;

[0015] According to each region contained in the initial mask image, a target mask image corresponding to at least one specified region identifier is generated;

[0016] Obtain the defect quantization parameter and the defect surface information corresponding to each of the at least one region identifier, wherein the defect quantization parameter is used to describe the second spatial information of the defect, and the defect surface information is used to describe the detail information of the defect;

[0017] Input the target mask image and the normal chip image into a trained sample generation model, use the target mask image to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameter, the defect surface information and the first spatial information corresponding to the corresponding region identifier respectively, and output the abnormal chip image with defects in each of the at least one region;

[0018] The defect quantization parameter is used as the label of the abnormal chip image to generate a defect sample.

[0019] The beneficial effects of the above technical solutions are: by regionally segmenting the normal chip image, combining the chip structure image drawn in advance, marking each region with a region identifier, and associating the region identifier with the first spatial information of the corresponding region, for each region segmented, at least one region identifier corresponding mask image can be generated to control the generation of defects in the corresponding region, and in the defect generation process, the first spatial information as the spatial constraint of the region where the defect is located is used to ensure that the generation of defects in the corresponding region according to the region identifier does not exceed the range of the region, and the rationality of the defects is improved. In the defect generation, the specified at least one region identifier is respectively associated with the defect quantization parameter describing the second spatial information of the defect, and the defect surface information describing the details of the defect, so that the mask image and the normal chip image are used as model input, and the mask image is used to guide the generation of defects in at least one region of the normal chip image, so that the generated defects conform to the structure of the chip, and the rationality of the defects is further improved. In the defect generation, defects can be added in the corresponding region according to the defect quantization parameter, the defect surface information and the first spatial information corresponding to the at least one region identifier respectively, and an abnormal chip image with defects in at least one region is output, realizing the quantization and fine control of regional defects while ensuring the defect generation efficiency, so as to better guide the learning of defect features by the detection task and improve the accuracy of defect detection.

[0020] Optionally, the method further comprises:

[0021] encoding the normal chip image into a latent vector, and performing noise processing on the latent vector to obtain an initial vector;

[0022] multiplying the initial vector and the target mask image pixel by pixel to obtain a target vector, and determining at least one region in the normal chip image according to the target vector;

[0023] for the at least one region, respectively performing: determining the range of the defect according to the first spatial information corresponding to the region, and adding the defect in the range according to the defect quantization parameter and the defect surface information corresponding to the region;

[0024] taking the image of the at least one region after adding the defect as an abnormal chip image.

[0025] The beneficial effects of the above technical solution are as follows: by encoding the normal chip image into a low-dimensional latent vector, the image data is compressed, which improves the defect generation speed. The target mask image is usually a binary image used to specify the region of interest that needs to be focused on. After multiplying with the latent vector, the non-region of interest can be suppressed, thereby guiding the model to generate defects only in the specified region, realizing the control of the defect region, and thus ensuring the rationality of defect generation.

[0026] Optionally, the defect quantization parameters include at least defect location, defect size, defect shape, and defect grayscale value. Adding defects within the specified range based on the defect quantization parameters and defect surface information corresponding to the region includes:

[0027] When the defect shape is circular, within the area, with the defect location as the center and the defect size as the radius, the defect area within the area is determined, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area;

[0028] When the defect shape is rectangular, within the area, the defect area is determined with the defect position as the top left vertex and the defect size as the length and width, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area;

[0029] When the defect shape is irregular, a set of irregular points is randomly generated within the area, the area enclosed by the set of points is taken as the defect area, and the pixel value associated with the defect surface information is set as the defect gray value within the defect area.

[0030] The beneficial effects of the above technical solution are as follows: by setting defect quantification parameters, the location, size, shape and other attributes of defects can be described, thereby quantifying the defects in the chip and ensuring the accuracy of subsequent defect detection. At the same time, the defect quantification results are semantically associated with the defect details through the defect grayscale value, thereby accurately controlling the precision of defect generation.

[0031] Optionally, after outputting the abnormal chip image, the method further includes:

[0032] Extract the attention weight matrix of the abnormal chip image and normalize the attention weight matrix.

[0033] For at least one defect in the abnormal chip image, perform the following respectively:

[0034] Filter out at least one pixel whose attention weight is greater than a preset weight threshold within the range of the defects;

[0035] The overlap rate between the defect and the corresponding region is determined based on at least one selected pixel and the pixels contained in the region where the defect is located in the target mask image.

[0036] When the overlap rate is lower than a preset overlap threshold, the abnormal chip image is discarded.

[0037] The beneficial effects of the above technical solution are as follows: Since the attention weight reflects the area of ​​concern for defects during the model generation process, by filtering at least one pixel with an attention weight greater than the preset weight threshold, the defect area understood by the model is obtained, and the corresponding area in the mask image accurately records the range where the defect should be. By calculating the overlap rate between the defect and the corresponding area by at least one pixel and the pixels contained in the corresponding area, the positional shift of the defect is avoided, and the accuracy of the generated defect is guaranteed.

[0038] Optionally, the training configuration of the sample generation model includes:

[0039] Construct a training sample pair set, each training sample pair containing a normal sample image, a defective sample image, a region mask image, and descriptive information of the defect in the defective sample image;

[0040] Multiple training samples are input into the initial generation model in batches for iterative training to obtain the sample generation model. The following operations are performed in each iteration:

[0041] After noise processing, the latent vector of the normal sample image is multiplied with the region mask image to obtain the defect diffusion vector;

[0042] The region of interest in the normal sample image is determined based on the defect diffusion vector, and a defect is added to the region of interest based on the description information to obtain the target defect image;

[0043] Based on the defect difference between the target defect image and the corresponding defect sample image, and at least one constraint term of chip process knowledge, the target defect generation loss is obtained, and the model parameters are adjusted using the target defect generation loss.

[0044] The beneficial effects of the above technical solution are as follows: by using a region mask image, the model is forced to perform defect mutations only in a specified region, thereby ensuring the rationality of defect generation. At the same time, the defect standard is used as a constraint term of the loss function, thereby avoiding the phenomenon of defects exceeding the range, further ensuring the rationality of defect generation.

[0045] Secondly, embodiments of this application provide an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0046] The communication interface is used to send and receive images;

[0047] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0048] Perform region segmentation on the normal chip image to obtain the initial mask image;

[0049] Based on the preset chip structure image, each region contained in the initial mask image is marked with a region identifier, and each region identifier is associated with the first spatial information of the corresponding region; wherein, the first spatial information serves as the spatial constraint of the defect.

[0050] Based on the regions contained in the initial mask image, generate a target mask image corresponding to at least one specified region identifier;

[0051] Obtain the defect quantization parameters and defect surface information corresponding to each of the at least one region identifier, wherein the defect quantization parameters are used to describe the second spatial information of the defect, and the defect surface information is used to describe the detailed information of the defect;

[0052] The target mask image and the normal chip image are input into a trained sample generation model. The target mask image is used to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information and the first spatial information corresponding to the corresponding region identifier. The model outputs abnormal chip images with defects in each of the at least one region.

[0053] The defect quantification parameters are used as labels for the abnormal chip images to generate defect samples.

[0054] Optionally, the processor uses the target mask image to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information, and the first spatial information corresponding to the corresponding region identifier, and outputs abnormal chip images with defects in each of the at least one region. Specifically, the operation is as follows:

[0055] The normal chip image is encoded into a latent vector, and the latent vector is subjected to noise processing to obtain an initial vector;

[0056] The initial vector is multiplied pixel by pixel with the target mask image to obtain the target vector, and at least one region in the normal chip image is determined based on the target vector;

[0057] For the at least one region, the following steps are performed: determining the range of the defect based on the first spatial information corresponding to the region, and adding a defect within the range based on the defect quantization parameters and defect surface information corresponding to the region;

[0058] The image with defects added to at least one region is used as the abnormal chip image.

[0059] Optionally, the defect quantization parameters include at least defect location, defect size, defect shape, and defect grayscale value. The processor adds defects within the range based on the defect quantization parameters and defect surface information corresponding to the region. Specifically, the operation is as follows:

[0060] When the defect shape is circular, within the area, with the defect location as the center and the defect size as the radius, the defect area within the area is determined, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area;

[0061] When the defect shape is rectangular, within the area, the defect area is determined with the defect position as the top left vertex and the defect size as the length and width, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area;

[0062] When the defect shape is irregular, a set of irregular points is randomly generated within the area, the area enclosed by the set of points is taken as the defect area, and the pixel value associated with the defect surface information is set as the defect gray value within the defect area.

[0063] Optionally, after outputting the abnormal chip image, the processor also performs:

[0064] Extract the attention weight matrix of the abnormal chip image and normalize the attention weight matrix.

[0065] For at least one defect in the abnormal chip image, perform the following respectively:

[0066] Filter out at least one pixel whose attention weight is greater than a preset weight threshold within the range of the defects;

[0067] The overlap rate between the defect and the corresponding region is determined based on at least one selected pixel and the pixels contained in the region where the defect is located in the target mask image.

[0068] When the overlap rate is lower than a preset overlap threshold, the abnormal chip image is discarded.

[0069] Optionally, the processor trains the sample generation model in the following manner:

[0070] Construct a training sample pair set, each training sample pair containing a normal sample image, a defective sample image, a region mask image, and descriptive information of the defect in the defective sample image;

[0071] Multiple training samples are input into the initial generation model in batches for iterative training to obtain the sample generation model. The following operations are performed in each iteration:

[0072] After noise processing, the latent vector of the normal sample image is multiplied with the region mask image to obtain the defect diffusion vector;

[0073] The region of interest in the normal sample image is determined based on the defect diffusion vector, and a defect is added to the region of interest based on the description information to obtain the target defect image;

[0074] Based on the defect difference between the target defect image and the corresponding defect sample image, and at least one constraint term of chip process knowledge, the target defect generation loss is obtained, and the model parameters are adjusted using the target defect generation loss.

[0075] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions that, when executed, can implement the steps of the chip defect sample generation method described above.

[0076] The technical effects of any of the implementation methods in the second to third aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 A schematic flowchart illustrating a chip defect sample generation method provided in an embodiment of this application;

[0079] Figure 2 These are schematic diagrams of the structures of different chip models provided in the embodiments of this application;

[0080] Figure 3 This is a schematic diagram of a chip structure image template provided in an embodiment of this application;

[0081] Figure 4A schematic diagram illustrating the process of generating defects by region as provided in an embodiment of this application;

[0082] Figure 5 This is a schematic diagram of the defect location and verification process provided in an embodiment of this application;

[0083] Figure 6A A schematic diagram of a defective product image provided for an embodiment of this application;

[0084] Figure 6B A schematic diagram of a defective but good product provided for an embodiment of this application;

[0085] Figure 7 A schematic flowchart of the sample generation model provided in the embodiments of this application;

[0086] Figure 8 A system structure diagram generated from chip defect samples provided in the embodiments of this application;

[0087] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0089] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0090] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0091] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0092] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0093] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0094] With the development of Artificial Intelligence Generated Content (AIGC) technology, high-quality defect samples can be generated using deep learning-based generation algorithms to supplement defect samples that are difficult to collect on the production line. This solves the problems of insufficient samples and sample imbalance, improves detection performance, and accelerates the optimization and iteration cycle of downstream detection tasks.

[0095] In the field of LED chip industrial quality inspection, deep learning-based generation algorithms face two major challenges when generating defect samples: insufficient generation of region structures and a lack of defect quantification capabilities. These challenges prevent precise control over quantitative parameters such as the location, size, and shape of defects. Insufficient region structure generation stems from the fact that LED chip structures typically comprise multiple regions (e.g., electrode areas, light-emitting areas, MESA areas, cleavage channels), and general generation methods do not consider the chip's structural features and information. This leads to blurry or unconventional defect samples in some regions, deviating from actual requirements. The lack of defect quantification capabilities arises because existing generation models (such as Anomaly Diffusion and RealNet) lack region identification mechanisms. Defects cannot be pinpointed to specific regions, and it is difficult to generate quantified defects that differ across different regions within the chip structure (e.g., electrode areas, light-emitting areas). This makes it difficult to simulate various possible defect scenarios in real-world LED chip defect detection, affecting the training and optimization of detection algorithms and reducing the accuracy and efficiency of defect detection.

[0096] In view of this, embodiments of this application provide a chip defect sample generation method. By segmenting a normal chip image into regions and combining it with a pre-drawn chip structure image, each segmented region is marked with a region identifier, and the region identifier is associated with the first spatial information of the corresponding region. In this way, for each segmented region, a mask image corresponding to at least one specified region identifier can be generated to control the generation of defects in the corresponding region. During the defect generation process, the first spatial information serves as a spatial constraint on the region where the defect is located, ensuring that the defect generated in the corresponding region according to the region identifier does not exceed the range of the region, thereby improving the rationality of the defect. During defect generation, each of the specified at least one region identifier is associated with a defect quantization parameter describing the second spatial information of the defect, as well as defect surface information describing the details of the defect. In this way, using a mask image and a normal chip image as model input, the mask image guides the generation of defects in at least one region of the normal chip image, making the generated defects conform to the chip structure and further improving the rationality of the defects. Moreover, during defect generation, defects can be added in parallel in the corresponding regions according to the defect quantization parameters, defect surface information, and first spatial information corresponding to at least one region identifier, and the abnormal chip image with defects in each region is output. This achieves the quantification and fine control of regional defects while ensuring the efficiency of defect generation, thereby better guiding the detection task to learn defect features and improving the accuracy of defect detection.

[0097] See Figure 1 The following is a flowchart illustrating a chip defect sample generation method provided in this application embodiment, which mainly includes the following steps:

[0098] S101: Perform region segmentation on the normal chip image to obtain the initial mask image.

[0099] In one embodiment, a normal chip image is input into a trained target segmentation model for region segmentation, and the Canny edge detection algorithm is used to extract the contours of each region in the normal chip image. The edge detection results are then fused with the region segmentation results to obtain an initial mask image.

[0100] Among them, the target segmentation model can be the Segment Anything Model (SAM). As an unsupervised segmentation model, the SAM model can utilize the prompt-free segmentation capability of SAM to explore the structure of the input normal chip image based on the knowledge learned from unlabeled samples, automatically identify the boundaries of different regions, and achieve pixel-level region segmentation.

[0101] In one embodiment, the normal chip image can be preprocessed before being input into the target segmentation model. Specifically, the normal chip image is converted into a grayscale chip image to reduce the amount of image data and computational complexity. Then, Gaussian filtering is used to denoise the chip grayscale image to remove random noise from the image. Finally, histogram equalization and other methods are used to increase the contrast of the image to improve the distinguishability of different regions in the normal chip image.

[0102] In one embodiment, a normal chip image is a chip image that does not contain defects, and the image format includes, but is not limited to, BMP, JPEG, etc.

[0103] It should be noted that the embodiments of this application do not impose restrictive requirements on the algorithm for region segmentation. In addition to using the SAM model, traditional image processing algorithms (such as threshold segmentation, region growing, etc.) or YOLO series models can also be used.

[0104] S102: Based on the preset chip structure image, mark each region contained in the initial mask image with a region identifier, and associate each region identifier with the first spatial information of the corresponding region.

[0105] The chip structure differs on different wafer models, such as Figure 2 The diagram shows schematics of two different chip structures. Therefore, for each wafer type, a chip structure image reflecting the chip structure can be pre-drawn by a draftsman, such as... Figure 3 As shown. Thus, after obtaining the initial mask image, template matching (such as the Sum of Squared Differences (SSD) matching algorithm) can be performed with the chip structure image of the corresponding model. Each segmented region is then uniquely labeled with a region identifier. Simultaneously, a region attribute table is constructed to record the first spatial information of the corresponding region associated with each region identifier. This first spatial information serves as the spatial constraint for the defect and includes at least the geometric information of the region, such as area, perimeter, coordinate range, bounding box coordinates, and Euler number.

[0106] Taking the area identifier as ID number as an example, the marking results are: ID=001 represents the electrode area, ID002 represents the light-emitting area, ID003 represents the MESA area, ID004 represents the cutting channel, etc.

[0107] After region labeling is completed, connected region analysis is performed on the initial mask image, mapping each pixel to a corresponding region identifier to obtain a region identifier matrix. This matrix has the same size as the initial mask image. Through mapping, a one-to-one relationship is established between each segmented region and the actual chip structure, facilitating targeted defect generation within a defined region under specific conditions in the later stages.

[0108] It should be noted that there are no restrictions on the method of drawing the chip structure image; for example, it can be drawn using CAD or Photoshop.

[0109] S103: Generate a target mask image corresponding to at least one specified region identifier based on the regions contained in the initial mask image.

[0110] In one embodiment, at least one region for which defects are to be generated can be specified using a region identifier matrix, based on actual detection requirements.

[0111] For example, when generating defects in the electrode region, the pixel with ID=001 is set to true, and the pixels of other region identifiers are set to false, resulting in a Boolean mask of the region identifier matrix. The electrode region is used as the foreground using the Boolean mask, and the Otsu thresholding algorithm is used to binarize the electrode region and non-electrode region of the initial mask image. Morphological dilation and erosion operations are then performed on the binarized image to fill small holes, resulting in the target mask image.

[0112] For example, when generating defects in the electrode area and the luminescent area, the pixels with ID=001 and ID=002 are set to true, and the pixels of other area identifiers are set to false, resulting in a Boolean mask of the area identifier matrix. The electrode area and the luminescent area are used as the foreground using the Boolean mask. The Otsu thresholding algorithm is used to binarize the initial mask image into electrode area, luminescent area and non-electrode area, and non-luminescent area. Morphological dilation and erosion operations are performed on the binarized image to fill small holes, resulting in the target mask image.

[0113] In the target mask image, the pixel value of at least one region where the defect is to be generated is 1, and the pixel value of other regions is 0.

[0114] S104: Obtain the defect quantization parameters and defect surface information corresponding to at least one region identifier.

[0115] Among them, the defect quantification parameter is used to describe the second spatial information of the defect, and the defect surface information is used to describe the detailed information of the defect.

[0116] In one embodiment, the defect quantization parameters are pre-defined for the region where the defect is to be generated, and include at least the defect location, defect size (e.g., length, width, radius), defect shape (e.g., circle, matrix, irregular shape), and defect grayscale value g. The defect location is defined as the relative coordinates (in pixels) of the defect center, with the region's centroid as the origin, to ensure that the defect is within the coordinate range of the specified region (e.g., the electrode area). The defect size is used to ensure that the defect does not exceed the specified region (e.g., the electrode area).

[0117] When the defect dimensions are length and width, the length must be less than or equal to half of the area.

[0118] In one embodiment, the defect surface information is natural language text automatically generated based on the GPT-4 model, describing the details of the defect, including but not limited to defect type, defect color, defect shape, etc.

[0119] For example, the defect surface information is as follows: the area with ID=002 has silver scratches and burrs on the edge; another example is that the area with ID=003 has a triangular notch and an edge roughness Ra=3.2μm.

[0120] It should be noted that the embodiments of this application do not impose restrictive requirements on the generation order of the defect surface information corresponding to each region identifier. For example, it can be generated in advance before defect generation, or it can be integrated into the sample generation model.

[0121] S105: Input the target mask image and the normal chip image into the trained sample generation model, and use the target mask image to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information and first spatial information corresponding to the corresponding region identifier, and output abnormal chip images with defects in at least one region.

[0122] During defect generation, targeted regional generation is performed using the target mask image corresponding to the region identifier. Additionally, specific quantized defects are generated within a specific region using defect quantization parameters, defect surface information, and first spatial information.

[0123] In one embodiment, the sample generation model is a diffusion model. In actual production lines, model deployment typically considers cost and computing power, generally choosing lightweight models with a small number of parameters. For example, the diffusion model uses the AnomalyAny model, which utilizes the attention-guided optimization and detail-hinting techniques of the AnomalyAny model. In the latent space of Stable Diffusion, by introducing region-level control and multimodal (such as image, text, and geometric parameters) input capabilities, quantization defects are generated in different regions to ensure that the generated defects conform to the chip structure.

[0124] See Figure 4 The flowchart for generating defects by region provided in this application embodiment mainly includes the following steps:

[0125] S1051: Encode the normal chip image into a latent vector, and perform noise processing on the latent vector to obtain an initial vector.

[0126] Specifically, a variational autoencoder (VAE) can be used to compress and represent a normal chip image into a latent vector in low dimension. Then, a denoising diffusion probabilistic model (DDPM) can be used to add and remove controllable noise from the latent vector to obtain an initial vector.

[0127] By encoding normal chip images into low-dimensional latent vectors, image data compression is achieved, thereby improving the defect generation speed.

[0128] S1052: Multiply the initial vector pixel by pixel with the target mask image to obtain the target vector, and determine at least one region in the normal chip image based on the target vector.

[0129] The initial vector is multiplied pixel by pixel with the target mask image to obtain the target vector. The non-zero regions in the target vector are the regions where defects are to be generated, thus forcing the diffusion process to introduce abnormal defects only in at least one specified region.

[0130] Since the target mask image is a binary image, it is used to specify the region of interest that needs to be focused on. After being multiplied with the latent vector, it can suppress the non-region of interest, thereby guiding the model to generate defects only in the specified region, realizing the control of the defect region, and thus ensuring the rationality of defect generation.

[0131] S1053: For at least one region, perform the following respectively: determine the range of the defect based on the first spatial information corresponding to the region, and add the defect within the range based on the defect quantization parameters and defect surface information corresponding to the region.

[0132] During the defect generation process, the range of the defect is determined based on the first spatial information corresponding to the region where the defect is to be generated, so as to ensure that the defect boundary maintains a certain distance from the boundary of the region, avoid the defect from exceeding the region range, and improve the accuracy of defect generation. Furthermore, the defect quantization parameters and defect surface information are encoded into semantic vectors by the CLIP encoder, and a quantized defect that conforms to the description is generated based on the semantic vector.

[0133] The defect quantification parameters include at least defect location, defect size, defect shape, and defect grayscale value. The defect quantification generation process includes the following methods:

[0134] (1) When the defect shape is circular, within the area, the defect area is determined with the defect location as the center and the defect size as the radius, and the pixel value associated with the defect surface information is set to the defect gray value within the defect area.

[0135] (2) When the defect shape is rectangular, within the area, the defect area is determined with the defect position as the upper left corner vertex and the defect size as the length and width. Within the defect area, the pixel value associated with the defect surface information is set as the defect gray value.

[0136] (3) When the defect shape is irregular, a set of irregular points is randomly generated within the area. The area enclosed by the point set is taken as the defect area, and the pixel value associated with the defect surface information is set as the defect gray value within the defect area.

[0137] By setting defect quantization parameters, the location, size, shape, and other attributes of defects can be described, thereby quantifying defects in the chip and ensuring the accuracy of subsequent defect detection. At the same time, the defect quantization results are semantically correlated with defect detail information through defect grayscale values, thereby precisely controlling the accuracy of defect generation.

[0138] In one embodiment, when defects need to be generated in multiple regions of an image, they can be generated in parallel by multiple threads, thereby further improving the speed of defect generation.

[0139] S1054: Use an image with defects added to at least one region as an abnormal chip image.

[0140] The abnormal chip image contains at least one defect.

[0141] S106: Use the defect quantization parameters as labels for the abnormal chip images to generate defect samples.

[0142] In defect detection tasks, the training samples of defect detection models need to be labeled with defect region information. Therefore, when the sample generation model outputs abnormal chip images containing defects, it can also output the defect quantization parameters as labels for the defect regions in the abnormal chip images.

[0143] In the embodiments of this application, region identifiers are marked on each region segmented from a normal chip image to achieve regional defect generation. During defect generation, the first spatial information of the defect location region is used as a constraint to ensure that the generated defect does not exceed the region range and conforms to the chip structure, thereby improving the rationality of the defect. At the same time, by specifying at least one region identifier associated with a defect quantification parameter describing the second spatial information of the defect, the quantification and fine control of regional defects are achieved, thereby better guiding the detection task to learn defect features and improving the accuracy of defect detection.

[0144] In one embodiment, before outputting the abnormal chip image, the generated defect can be located and verified to ensure the accuracy of defect generation.

[0145] See Figure 5This is a schematic diagram of the defect location and verification process, which mainly includes the following steps:

[0146] S1055: Extract the attention weight matrix from the abnormal chip image and normalize the attention weight matrix.

[0147] S1056: For each defect in the abnormal chip image, filter out at least one pixel within the range of the defect whose attention weight is greater than a preset weight threshold.

[0148] For example, the preset weight threshold is 0.5, and this application embodiment does not impose any limiting requirements.

[0149] S1057: Determine the overlap rate between the defect and the corresponding region based on at least one selected pixel and the pixels contained in the region where the defect is located in the target mask image.

[0150] The overlap rate is the ratio of the intersection of at least one selected pixel and the pixels contained in the defect region to the union of at least one selected pixel and the pixels contained in the defect region.

[0151] S1058: Determine whether the overlap rate is lower than the preset overlap threshold. If yes, execute S1059; otherwise, execute S10510.

[0152] Optionally, the overlap threshold can be set according to actual needs, such as setting it to 90%.

[0153] S1059: Remove abnormal chip images.

[0154] S10510: Retain abnormal chip image.

[0155] Since attention weights reflect the areas of concern for defects during model generation, the defect areas understood by the model are obtained by filtering at least one pixel whose attention weight is greater than a preset weight threshold. The corresponding area in the target mask image accurately records the range where the defect should be. The overlap rate between the defect and the corresponding area is calculated by comparing at least one pixel with the pixels contained in the corresponding area, thus verifying the rationality of the defect distribution and avoiding defect position shifts, thereby ensuring the accuracy of defect generation.

[0156] In one embodiment, the sample defect model can output not only abnormal chip images and defect quantification parameters, but also a defect compliance assessment report. Specifically, during defect generation, after adding a defect to a specified area, the attributes of the defect can be evaluated based on a preset chip process knowledge base and the first spatial information of the corresponding area to generate a defect compliance assessment report. The chip process knowledge base contains the definition of defects, and the defect compliance assessment report indicates whether the generated defect conforms to the definition of a defect.

[0157] For example, taking the generation of defects in the electrode region as an example, assuming that the area of ​​the click region is determined to be 2μm based on the first spatial information of the electrode region. 2 The contamination area of ​​the indicator electrode region in the chip process knowledge base must be greater than or equal to 0.5 μm. 2 Only then does it constitute a defective product. Therefore, when the area of ​​the dirt defect generated in the electrode area is 0.6 μm... 2 At that time, the defect compliance report indicated that the generated contamination defect met the defect definition and constituted a defective product. The area of ​​the contamination defect generated in the electrode area was 0.45 μm. 2 At that time, the defect compliance report pointed out that the generated dirt defect did not meet the defect definition and constituted a good product.

[0158] Taking a normal chip image as an example, which is an image acquired under four light sources, such as Figure 6A The image shown is a schematic diagram of an abnormal chip after adding contamination defects. Figure 6B The image shown is a schematic diagram of an abnormal chip image of a good product after adding contamination defects. Although both types of abnormal chip images contain contamination defects, they can be used as samples of different categories for training in defect detection tasks.

[0159] In one embodiment, the training process of the sample generation model is as follows: Figure 7 As shown, it mainly includes the following steps:

[0160] S701: Construct a training sample pair set, each training sample pair containing a normal sample image, a defective sample image, a region mask image, and descriptive information of the defect in the defective sample image;

[0161] The descriptive information includes defect surface information and defect quantification parameters.

[0162] S702: Input multiple training samples into the initial generation model in batches for iterative training to obtain the sample generation model.

[0163] Each iteration performs the following operations:

[0164] S7021: After noise processing, the latent vector of the normal sample image is multiplied with the region mask image to obtain the defect diffusion vector.

[0165] The vector encoder can be a VAE encoder, and the noise processor can be a DDPM model.

[0166] S7022: Determine the region of interest in the normal sample image based on the defect diffusion vector, and add defects to the region of interest based on the description information to obtain the target defect image.

[0167] S7023: Based on the defect difference between the target defect image and the corresponding defect sample image, and at least one constraint term of chip process knowledge, obtain the target defect generation loss, and adjust the model parameters with the target defect generation loss.

[0168] In one embodiment, the defect difference between the target defect image and the defect sample image can be represented by the mean squared error. Based on this, at least one constraint term is added to obtain the loss function of the sample generation model. Thus, after adding at least one constraint term, when the generated defect exceeds the defect area, the target generation loss increases significantly. The model adjusts its parameters according to the target generation loss to avoid this situation and ensure the rationality of defect generation.

[0169] Taking a single constraint as an example, the formula for the loss function is as follows:

[0170] L = L MSE +λ·max(0,A1-A2) Formula 1

[0171] Among them, L MSE A1 represents the size of the generated defect, A2 represents the size of the defect area indicated in the chip process knowledge, and λ is the penalty coefficient.

[0172] During the training of the sample generation model, the model is forced to perform defect mutations only in the specified regions by inputting a region mask image, thereby achieving regional defect generation. At the same time, chip process knowledge is used as a constraint term in the loss function to avoid defects exceeding the range, thus further ensuring the rationality of defect generation.

[0173] join Figure 8This document presents a system structure diagram for generating chip defect samples according to an embodiment of the present application. The system mainly includes an image preprocessing module, a region mapping module, and a defect quantization generation module. The image preprocessing module performs preprocessing operations on the input normal chip image, including grayscale conversion, filtering and noise reduction, and contrast enhancement, to improve image clarity and provide a good foundation for the subsequent region mapping and defect quantization generation modules. The region mapping module uses an unsupervised segmentation model (such as SAM with prompt-free input) to segment the preprocessed normal chip image into regions. Combined with a pre-generated chip structure image, it assigns a unique region identifier to each region corresponding to a chip structure (such as electrode area, light-emitting area, segmentation channel, MESA area, etc.) using template matching. Each region identifier is then topologically associated with the first spatial information (such as area, perimeter, coordinate range, bounding box coordinates, Euler number, etc.) of the corresponding region. The first spatial information of each region can be used for subsequent evaluation of the attributes of the generated defects. The defect quantification generation module includes two parts: targeted region generation based on region identifiers, and defect generation within a specific structural region based on set defect quantification parameters. During the defect quantification generation process, for regularly shaped defects (such as circles, rectangles, etc.), the defect outline can be directly generated using mathematical formulas based on the defect size; for irregularly shaped defects, a random generation algorithm combined with morphological operations can be used to simulate the defect outline. Furthermore, the set grayscale values ​​are semantically correlated with the defect surface information to adjust the pixel values ​​of the defect area, thereby highlighting the contrast difference between the defect area and the surrounding normal area, and obtaining an abnormal chip image with defects. After defect location verification, the verified abnormal chip image, the corresponding defect quantification parameters, and the defect compliance assessment report are output as results to facilitate the training and verification of subsequent defect detection tasks.

[0174] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0175] Based on the same technical concept, this application provides an electronic device, including but not limited to laptops, desktop computers, and testing terminals, which can implement the steps of the above-described chip defect sample generation method and achieve the same technical effect.

[0176] See Figure 9The electronic device includes a processor 901, a memory 902, and a communication interface 903, which are connected via a bus 904.

[0177] Communication interface 903 is used for image transmission and reception;

[0178] The memory 902 stores a computer program, and the processor 901 executes the program according to the computer program. Figure 1 The steps of the method for generating chip defect samples.

[0179] It should be noted that, Figure 9 This is just one example; other electronic devices, not shown, may also include components such as memory, power supply, crystal oscillator, camera, and display screen.

[0180] In this embodiment, the memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and programs required for running instant messaging functions; the data storage area may store various instant messaging information and operation instruction sets. The memory may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any other medium capable of carrying or storing a desired computer program having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory may be a combination of the above-mentioned memories. The processor may include one or more central processing units (CPUs), GPUs, or digital processing units, etc.

[0181] This application does not limit the specific connection medium between the communication interface, memory, and processor. In this application, the bus between the communication interface, memory, and processor is depicted with a thick line in the figure. The connection methods between other components are only for illustrative purposes and are not intended to be limiting. Buses can be divided into address buses, data buses, control buses, etc. For ease of description, only one thick line is used to describe them in the figure, but this does not mean that there is only one bus or one type of bus.

[0182] This application also provides a computer-readable storage medium for storing instructions that, when executed, can complete the steps of any of the chip defect sample generation methods described in the foregoing embodiments.

[0183] This application also provides a computer program product for storing a computer program that performs the steps of any of the chip defect sample generation methods described in the foregoing embodiments.

[0184] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0188] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating chip defect samples, characterized in that, The method includes: Perform region segmentation on the normal chip image to obtain the initial mask image; Based on the preset chip structure image, each region contained in the initial mask image is marked with a region identifier, and each region identifier is associated with the first spatial information of the corresponding region; wherein, the first spatial information serves as the spatial constraint of the defect. Based on the regions contained in the initial mask image, generate a target mask image corresponding to at least one specified region identifier; Obtain the defect quantization parameters and defect surface information corresponding to each of the at least one region identifier, wherein the defect quantization parameters are used to describe the second spatial information of the defect, and the defect surface information is used to describe the detailed information of the defect; The target mask image and the normal chip image are input into a trained sample generation model. The target mask image is used to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information and the first spatial information corresponding to the corresponding region identifier. The model outputs abnormal chip images with defects in each of the at least one region. The defect quantification parameters are used as labels for the abnormal chip images to generate defect samples.

2. The method as described in claim 1, characterized in that, The step of using the target mask image to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information, and the first spatial information corresponding to the corresponding region identifier, and outputting abnormal chip images with defects in each of the at least one region, includes: The normal chip image is encoded into a latent vector, and the latent vector is subjected to noise processing to obtain an initial vector; The initial vector is multiplied pixel by pixel with the target mask image to obtain the target vector, and at least one region in the normal chip image is determined based on the target vector; For the at least one region, the following steps are performed: determining the range of the defect based on the first spatial information corresponding to the region, and adding a defect within the range based on the defect quantization parameters and defect surface information corresponding to the region; The image with defects added to at least one region is used as the abnormal chip image.

3. The method as described in claim 2, characterized in that, The defect quantification parameters include at least defect location, defect size, defect shape, and defect grayscale value. Adding defects within the specified range based on the defect quantification parameters and surface information of the region includes: When the defect shape is circular, within the area, with the defect location as the center and the defect size as the radius, the defect area within the area is determined, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area; When the defect shape is rectangular, within the area, the defect area is determined with the defect position as the top left vertex and the defect size as the length and width, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area; When the defect shape is irregular, a set of irregular points is randomly generated within the area, the area enclosed by the set of points is taken as the defect area, and the pixel value associated with the defect surface information is set as the defect gray value within the defect area.

4. The method according to any one of claims 1-3, characterized in that, After outputting the abnormal chip image, the method further includes: Extract the attention weight matrix of the abnormal chip image and normalize the attention weight matrix. For at least one defect in the abnormal chip image, perform the following respectively: Filter out at least one pixel whose attention weight is greater than a preset weight threshold within the range of the defects; The overlap rate between the defect and the corresponding region is determined based on at least one selected pixel and the pixels contained in the region where the defect is located in the target mask image. When the overlap rate is lower than a preset overlap threshold, the abnormal chip image is discarded.

5. The method according to any one of claims 1-3, characterized in that, The training components of the sample generation model include: Construct a training sample pair set, each training sample pair containing a normal sample image, a defective sample image, a region mask image, and descriptive information of the defect in the defective sample image; Multiple training samples are input into the initial generation model in batches for iterative training to obtain the sample generation model. The following operations are performed in each iteration: After noise processing, the latent vector of the normal sample image is multiplied with the region mask image to obtain the defect diffusion vector; The region of interest in the normal sample image is determined based on the defect diffusion vector, and a defect is added to the region of interest based on the description information to obtain the target defect image; Based on the defect difference between the target defect image and the corresponding defect sample image, and at least one constraint term of chip process knowledge, the target defect generation loss is obtained, and the model parameters are adjusted using the target defect generation loss.

6. An electronic device, characterized in that, It includes a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus; The communication interface is used to send and receive images; The memory stores a computer program, and the processor performs the following operations according to the computer program: Perform region segmentation on the normal chip image to obtain the initial mask image; Based on the preset chip structure image, each region contained in the initial mask image is marked with a region identifier, and each region identifier is associated with the first spatial information of the corresponding region; wherein, the first spatial information serves as the spatial constraint of the defect. Based on the regions contained in the initial mask image, generate a target mask image corresponding to at least one specified region identifier; Obtain the defect quantization parameters and defect surface information corresponding to each of the at least one region identifier, wherein the defect quantization parameters are used to describe the second spatial information of the defect, and the defect surface information is used to describe the detailed information of the defect; The target mask image and the normal chip image are input into a trained sample generation model. The target mask image is used to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information and the first spatial information corresponding to the corresponding region identifier. The model outputs abnormal chip images with defects in each of the at least one region. The defect quantification parameters are used as labels for the abnormal chip images to generate defect samples.

7. The electronic device as claimed in claim 6, characterized in that, The processor uses the target mask image to guide the addition of defects in at least one region of the normal chip image according to the defect quantization parameters, defect surface information, and the first spatial information corresponding to the corresponding region identifier, and outputs abnormal chip images with defects in each of the at least one region. The specific operation is as follows: The normal chip image is encoded into a latent vector, and the latent vector is subjected to noise processing to obtain an initial vector; The initial vector is multiplied pixel by pixel with the target mask image to obtain the target vector, and at least one region in the normal chip image is determined based on the target vector; For the at least one region, the following steps are performed: determining the range of the defect based on the first spatial information corresponding to the region, and adding a defect within the range based on the defect quantization parameters and defect surface information corresponding to the region; The image with defects added to at least one region is used as the abnormal chip image.

8. The electronic device as claimed in claim 7, characterized in that, The defect quantization parameters include at least defect location, defect size, defect shape, and defect grayscale value. The processor adds defects within the specified range based on the defect quantization parameters and defect surface information corresponding to the region. Specifically, the operation is as follows: When the defect shape is circular, within the area, with the defect location as the center and the defect size as the radius, the defect area within the area is determined, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area; When the defect shape is rectangular, within the area, the defect area is determined with the defect position as the top left vertex and the defect size as the length and width, and the pixel value associated with the defect surface information is set to the defect grayscale value within the defect area; When the defect shape is irregular, a set of irregular points is randomly generated within the area, the area enclosed by the set of points is taken as the defect area, and the pixel value associated with the defect surface information is set as the defect gray value within the defect area.

9. The electronic device as claimed in any one of claims 6-8, characterized in that, After outputting the image of the abnormal chip, the processor also performs: Extract the attention weight matrix of the abnormal chip image and normalize the attention weight matrix. For at least one defect in the abnormal chip image, perform the following respectively: Filter out at least one pixel whose attention weight is greater than a preset weight threshold within the range of the defects; The overlap rate between the defect and the corresponding region is determined based on at least one selected pixel and the pixels contained in the region where the defect is located in the target mask image. When the overlap rate is lower than a preset overlap threshold, the abnormal chip image is discarded.

10. The electronic device as claimed in any one of claims 6-8, characterized in that, The processor trains the sample generation model in the following manner: Construct a training sample pair set, each training sample pair containing a normal sample image, a defective sample image, a region mask image, and descriptive information of the defect in the defective sample image; Multiple training samples are input into the initial generation model in batches for iterative training to obtain the sample generation model. The following operations are performed in each iteration: After noise processing, the latent vector of the normal sample image is multiplied with the region mask image to obtain the defect diffusion vector; The region of interest in the normal sample image is determined based on the defect diffusion vector, and a defect is added to the region of interest based on the description information to obtain the target defect image; Based on the defect difference between the target defect image and the corresponding defect sample image, and at least one constraint term of chip process knowledge, the target defect generation loss is obtained, and the model parameters are adjusted using the target defect generation loss.

Images