Image generation method, defect detection method, and defect detection device formed thereby

Through the diffusion model, the transistor defect image is noise-processed and denoised, and combined with the probability density and the realistic discriminant loss function, the problem of large differences in defect detection in the existing technology is solved, and high-quality defect image generation is achieved and detection.

CN119048475APending Publication Date: 2024-11-29XIAN JIAOTONG LIVERPOOL UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411172303.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, transistor defect detection relies on manual detection, which is highly subjective and costly. Deep learning algorithms are difficult to deploy due to insufficient defect samples. The samples generated by traditional image generation algorithms are very different from real samples. The algorithm training process based on deep learning is unstable. .

Method used

The diffusion model is used to perform noise processing on the real transistor defect image, and denoising is directed to generate high-quality defect samples through density reduction loss. Combining the probability density distribution and the realistic discriminant loss function, a variety of defect images are generated that are close to the reality.

Benefits of technology

It realizes the generation of a large number of high-quality transistor defect image samples, improves the accuracy and stability of defect detection, and is suitable for automated inspection in industrial quality inspection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048475A_ABST
    Figure CN119048475A_ABST
Patent Text Reader

Abstract

The invention relates to an image generation method, a defect detection method and defect detection equipment formed by the image generation method and the defect detection method. The method comprises the following steps: S1, an original sample capturing step: enabling an image acquisition device to capture a real product image after transistor production; s2, a defect generation sample forming step: taking the captured real product defect image as an original sample, performing noise increase on the original sample through a diffusion model, and then performing de-noising processing on an area with reduced probability distribution density through a density reduction loss guide diffusion model to generate a new product defect image; and S3, a product detection step: taking the new product defect image as a defect generation sample, determining the defect position of the defect generation sample, marking the defect of the defect generation sample, and detecting the product after transistor production by using the marked defect generation sample. According to the method, a large amount of high-quality transistor defect image data can be generated, the method can be applied to product defect detection in subsequent production, and considerable training data can be provided for other target detection algorithms.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to an image generation method, a defect detection method, and a defect detection device constructed thereby. Background Art

[0002] Transistors are vital components in modern electronic devices, and chips are the integration of transistors. Transistors are electronic devices used to control current, and they are made of semiconductor materials. As an important component of electronic devices in industrial production, transistors have functions such as amplifying signals, switching circuits, and controlling voltages. Therefore, it is necessary to determine surface defects such as fractures, holes, and wear through industrial quality inspection processes. Although the industrial production automation product line in the existing technology is becoming increasingly mature, the industrial quality inspection of transistors, especially surface defect detection, still mostly relies on manual inspection, which is highly subjective, costly, and prone to visual fatigue.

[0003] Transistor defect detection algorithms based on deep learning mostly use target detection algorithm frameworks such as YOLO (You Only Look Once). The process of the YOLO algorithm is to first manually collect transistor image samples, then manually annotate detection boxes (Region Proposals), and finally train the target detection network to locate anomalies. However, this detection framework often cannot be deployed smoothly due to insufficient transistor defect samples, high collection difficulty, and inability to collect all defect types.

[0004] The mainstream algorithms for generating transistor defect images in the current prior art are as follows: 1) Traditional image processing algorithm, which synthesizes defective samples by randomly sampling patches from texture images, rotating, cropping, filtering, and then pasting them on normal samples. Although the algorithm is simple to operate, there is a huge difference between the synthesized samples and the real samples, and it cannot be well used in actual industrial production. 2) Generation algorithm based on deep learning: Transistor generation algorithms based on deep neural networks can be mainly divided into two categories: the first category is based on the variational autoencoder (VAE) algorithm, which is simple to deploy and easy to train, but the generated samples are fuzzy. The second category is based on the generative adversarial network (GAN) algorithm, which can generate high-quality samples, but the training process is extremely unstable. This is why the present invention comes. Summary of the invention

[0005] The embodiments of the present application provide a transistor defect detection method, an image data generation method and a defect detection device constituted thereby, which can realize the generation of transistor defect product images and can generate a large number of diverse and high-quality transistor defect image generation samples, thereby facilitating the training of transistor image data and subsequent product defect detection.

[0006] In a first aspect, an embodiment of the present application provides a transistor defect detection method, characterized by comprising:

[0007] S1, capturing original samples step: enabling an image acquisition device to capture a real product image of the transistor after production;

[0008] S2, defect generation sample formation step: taking the captured real product defect image as the original sample, adding noise to the original sample through the diffusion model, and then guiding the diffusion model to the area with reduced probability distribution density through the density reduction loss to perform denoising processing, so as to generate a new product defect image;

[0009] S3, product inspection step: using the new product defect image as a defect generation sample, determining the defect location of the defect generation sample, and marking the defects of the defect generation sample, and using the marked defect generation sample to inspect the transistor product after production.

[0010] In some possible embodiments, in step S2 of the method, the original sample further undergoes an image preprocessing step, and the image preprocessing step includes image cleaning and image cropping; the image cleaning includes classifying the original sample according to the defect type in the product image, and the number of images of the original sample in each defect type is not less than 50; the image cropping includes cropping the original sample to a uniform size.

[0011] In some possible embodiments, in step S2 of the method, the backbone network of the diffusion model adopts a UNet structure, and the original sample is forwardly added with noise according to the Markov process of formula (F1):

[0012]

[0013] In formula (F1) t is the time step t

[0014] Among them, x 0 is the original sample without noise; the q(·) operation represents the Markov process; represents Gaussian distribution, x t Obey Gaussian distribution; β t is the variance at time step t, and is not a constant; I is the variance of the added Gaussian noise;

[0015] In some possible embodiments, the probability density distribution used for denoising in step S2 of the method is constructed by a Gaussian model. Assuming that there are k types of defects in the product image, the probability distribution of k types of transistor defect images constructed by the multivariate Gaussian model is (F2):

[0016]

[0017] in is a Gaussian model with a mean of μk and a variance of ∑k; x is the x-th type of transistor defect image.

[0018] In some possible embodiments, in step S2 of the method, a density reduction loss function (F3) is determined according to the probability distribution:

[0019]

[0020] Where: f 1 (·) is the trained classifier; x i For the kth i The i-th transistor defect image in the defect class; N is the total number of transistor defect images in the class; τ represents the temperature coefficient.

[0021] In some possible embodiments, the image generation process for removing noise in step S2 of the method is performed step by step in the manner (F4):

[0022]

[0023] where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; 1 is a hyperparameter; is the normalized gradient; is the density reduction loss function.

[0024] In some possible embodiments, in step S2 of the method, a realistic discrimination loss function (F5) is determined according to the probability distribution:

[0025]

[0026] Where: f 2 (·) is the trained discriminator (binary classifier); S * The category is true or false, that is, {0, 1}x i For the kth iThe i-th transistor defect image in the defect class; N is the total number of transistor defect images in the class; τ represents the temperature coefficient.

[0027] In some possible embodiments, the image generation process for removing noise in step S2 of the method is performed step by step in the manner (F6):

[0028]

[0029] where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; 1 , 2 All are hyperparameters; is the normalized gradient; is the density reduction loss function.

[0030] In some possible embodiments, the method for detecting transistor products after production by using the marked defect generation samples in step S3 of the method is a YOLO target detection method.

[0031] In a second aspect, an embodiment of the present invention provides a method for generating image data based on a diffusion model, characterized in that the method comprises the following steps:

[0032] (S21) using a diffusion model to gradually increase noise processing on the original image data;

[0033] (S22) The diffusion model is guided by the density reduction loss to perform denoising on the area where the probability distribution density is reduced, thereby generating the required image data.

[0034] In some possible embodiments, the backbone network of the diffusion model in the method adopts a UNet structure, and the original sample is forwardly added with noise according to the Markov process of formula (F1):

[0035]

[0036] In formula (F1) t is the time step t

[0037] Among them, x 0 is the original sample without noise; the q(·) operation represents the Markov process; represents Gaussian distribution, x t Obey Gaussian distribution; β t is the variance at time step t, and is not a constant; I is the variance of the added Gaussian noise;

[0038] Preferably, the probability density distribution used for denoising in step S2 of the method is constructed by a Gaussian model. Assuming that there are k types of defects in the product image, the probability distribution of k types of transistor defect images constructed by the multivariate Gaussian model is (F2):

[0039]

[0040] in is a Gaussian model with a mean of μk and a variance of ∑k; x is the x-th type of transistor defect image;

[0041] Preferably, in step S2 of the method, a density reduction loss function (F3) is determined according to the probability distribution:

[0042]

[0043] Where: f 1 (·) is the trained classifier; x i For the kth i The i-th transistor defect image in the class defect; N is the total number of transistor defect images in the class; τ represents the temperature coefficient;

[0044] Preferably, the image generation process for removing noise in step S2 of the method is stepwise generated in the manner (F4):

[0045]

[0046] where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; 1 is a hyperparameter; is the normalized gradient; is the density reduction loss function;

[0047] Preferably, in step S2 of the method, a pseudo-realistic discrimination loss function (F5) is determined according to the probability distribution:

[0048]

[0049] Where: f 2 (·) is the trained discriminator (binary classifier); S * The category is true or false, that is, {0, 1}x i For the kth i The i-th transistor defect image in the class defect; N is the total number of transistor defect images in the class; τ represents the temperature coefficient;

[0050] Preferably, the image generation process for removing noise in step S2 of the method is performed step by step in the manner (F6):

[0051]

[0052] where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; 1 , 2 All are hyperparameters; is the normalized gradient; is the density reduction loss function.

[0053] In a third aspect, an embodiment of the present invention further provides a defect detection device for transistor production, characterized in that the device includes an image acquisition device, a processor and a memory,

[0054] The image acquisition device is used to capture the real product image of the transistor after production;

[0055] The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method described are implemented.

[0056] In a fourth aspect, an embodiment of the present invention further provides an image generation model training device, comprising a processor,

[0057] The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method described are implemented.

[0058] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a processor, the processor executes the described method.

[0059] In a sixth aspect, an embodiment of the present invention further provides a computer program product, wherein the computer program product comprises software codes, and the software codes are used to execute the steps of the described method.

[0060] The present application provides a transistor defect detection method, an image data generation method and a defect detection device constructed thereby. By using a probability density driven diffusion model to generate a large number of transistor defect samples, a large number of high-quality transistor defect product images that are close to real defects can be obtained, which can be applied to product defect detection of transistor products or integrated circuit products, and can also be used as training material for defective product image data, as input for target detection algorithms such as YOLO, etc. Since the generated defective product image data is more complete, the neural network system and target detection constructed thereby can provide considerable generated product defect images, facilitate target detection, and improve the product's rapid defect detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 A method flow chart of a method for generating image data based on a diffusion model provided in an embodiment of the present application;

[0063] Figure 2 A model framework diagram of the image data generation method based on the diffusion model provided in an embodiment of the present application;

[0064] Figure 3 A sample image of an original transistor defect image provided in an embodiment of the present application.

[0065] Figure 4 A comparison diagram of the effects of transistor defect image samples generated by the traditional image generation method and the training provided by the embodiment of the present application;

[0066] Figure 5 An overall flow chart of a transistor defect detection method provided in a specific embodiment of the present application;

[0067] Figure 6 A specific flow chart of using YOLO for target detection provided in a specific embodiment of the present application;

[0068] Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided for a specific embodiment of the present application;

[0069] Figure 8 A structural principle diagram of an image generation model training device provided in a specific embodiment of the present application;

[0070] Fig. 9 A structural block diagram of a defect detection device for transistor production provided in a specific embodiment of the present application. DETAILED DESCRIPTION

[0071] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0072] It should be understood that in the description of the specification and claims of this application, if there are terms such as "first", "second", etc., they are only used to distinguish the objects described, and do not have any order or technical meaning. Therefore, the objects defined as "first", "second", etc. may explicitly or implicitly include one or more of the objects. In addition, "one" or "a" and similar words do not indicate a quantitative limitation, but indicate the existence of at least one, and "multiple" means no less than two.

[0073] Since there are not enough actual defective product images of transistors in the prior art, it is necessary to collect defective product images for a long time, and there is a huge difference between the generated samples of transistor defect images in the prior art and the real samples, or the generated defective samples are blurred and cannot be used in target detection. The present invention provides an image generation method based on a diffusion model, which can provide more considerable defective product image data with more defect types, and can provide a reference for fully automated detection equipment in industrial quality inspection.

[0074] The diffusion model (Denoising Diffusion Probabilistic Models, DDPM) is a new generation of generative models. This model gradually adds noise to the data set to form a noisy image. Then it removes the noise inversely and learns how to reverse this process through machine learning to uniquely generate the original image or other high-quality data. The training and deployment of the diffusion model includes two steps. The first stage is the noise addition step, which is to gradually add noise to the original image in order to provide a learning target for the subsequent denoising operation. The second stage is the denoising step, which is to establish a deep neural network to learn the noise and restore the image through gradual denoising. However, the generation process of the diffusion model's gradual denoising is uncontrollable, and sometimes there is a big gap between the generated defect samples and the real defect samples.

[0075] The embodiment of the present invention is based on a diffusion model. By using a probability density descent control algorithm to generate a diffusion model during image generation by denoising, the present invention solves the common shortcomings of traditional image generation algorithms and deep learning-based defect generation algorithms, such as poor authenticity of generated defect images, a single pattern, and poor controllability. The improved technology of the diffusion model of the present invention can be applied to the field of artificial intelligence visual technology, especially to industries such as industrial defect detection or computer vision.

[0076] The image data generation method based on the diffusion model of the present invention is described in detail below.

[0077] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for generating image data based on a diffusion model, comprising the following steps:

[0078] (S21) using a diffusion model to gradually increase noise processing on the original image data;

[0079] (S22) The diffusion model is guided by the density reduction loss to perform denoising on the area where the probability distribution density is reduced, thereby generating the required image data.

[0080] Among them, step (S21) is a noise adding step, and step (S22) is an improved noise removing step.

[0081] The following is a specific description of the application scenario of transistor defect image generation. Before the diffusion model-based image data generation method, the original image data needs to be preprocessed. These image data preprocessing include image data collection, organization, data cleaning, data enhancement and other operations.

[0082] The specific data collection and sorting work can be as follows: First, collect transistor samples with various defects from the transistor production line. The defective sample collection is required to ensure that the types are as diverse as possible. Then, use an image acquisition device such as a camera (high-resolution camera, industrial camera, etc.) to calibrate the camera, and then place the transistor in the center of the photo area to take a photo to obtain the original defect image of the real sample. Finally, the original defect images of the real samples obtained are screened and the real defect images that do not meet the operation are cleaned to obtain k types of transistor defect images, requiring no less than 50 images for each class, so that more considerable original defect image data can be obtained.

[0083] The specific data enhancement operation can be to crop all the real defect images after the above cleaning to a uniform size, such as a resolution of 1024×1024, and perform center cropping to remove redundant background. The data enhancement operation can be performed before the data enters the network, or it can be processed uniformly in advance.

[0084] In the noise adding step (S21), the generation framework adopts the basic diffusion model (DDPM). The backbone network of this diffusion model adopts the UNet structure. The input of the network is the training sample image x and the time step t. Then the diffusion model gradually adds noise to the original image. Specifically, the forward noise adding process q(·) can be written as:

[0085]

[0086] Where: -x 0 is the original image without noise. -q(·) is a Markov process. - is a Gaussian distribution. -β t is the variance at time step t, which is an adjustable hyperparameter. I is the variance of the added Gaussian noise.

[0087] Markov process is a type of random process. The diffusion model applied in this case is based on Markov chain. The improvement of the diffusion model of the present invention is that the denoising step adds the modeling of probability distribution density, and uses the density reduction loss to guide the diffusion model to the area where the probability distribution density is reduced.

[0088] In the denoising step (S22), the probability density modeling is first performed on the original real defect image data of the transistor sample, and then the density reduction loss is used to guide the diffusion model to optimize the low-density area of ​​the probability distribution to generate diversified defect samples.

[0089] The specific generation framework of this application adopts the basic diffusion model (DDPM). The backbone network of this diffusion model adopts the UNet structure to learn the denoising process and generate the required defect samples. The detailed process is as follows:

[0090] (a) Use the multivariate Gaussian model to construct the probability distribution of training sample images: The goal of this process is to construct the probability distribution of transistor samples with different types of defects, which will be used to guide the generation of the diffusion model. First, use the multivariate Gaussian model to construct the probability distribution of k-type transistor defect sample images:

[0091]

[0092] in The mean is μ k , the variance is ∑ k The larger S(x, k) is, the smaller the probability density of x is.

[0093] (b) Construct a density reduction loss function and guide the diffusion model generation through the low density reduction loss function: After obtaining the probability distribution of each type of defect sample, define the density reduction loss:

[0094]

[0095] Where: -f 1 (·) is the trained classifier. -x i For the kth i The i-th image in the class defect. -N is the total number of classes. -τ represents the temperature coefficient.

[0096] Then the loss-guided diffusion model is used to generate the loss-guided diffusion model:

[0097]

[0098] in:

[0099] -x t-1 is the noisy image at time t-1, x t is the noisy image at time t.

[0100] -μ θ is the mean predicted by the diffusion model, which is associated with the noise at time t.

[0101] - is the standard deviation of the diffusion model prediction, associated with the noise at time t.

[0102] -λ 1 is a hyperparameter.

[0103] is the normalized gradient.

[0104] At this time, the diffusion model will tend to sample from the direction of the low-density area and generate samples with diverse defects. Here, the low-density area is an area with relatively low probability density of the Gaussian distribution fitted by the training data, without a clear boundary, and the range of this area will change with the number of iterations of the denoising process. The algorithm of the present invention mainly guides the diffusion model to generate samples in the direction of decreasing probability density. After multiple iterations, the generated samples will be at the edge of the distribution, so that the range of the low-density area will change with the number of iterations of the denoising process.

[0105] (c) Constructing a realistic discriminant loss function, adding a realistic discriminant loss function to guide the diffusion model generation on the basis of the low-density descent loss function: In order to ensure that the generated samples have high authenticity, the embodiment of the present invention provides a realistic discriminant loss function to optimize the generation network. In order to make the generated samples more realistic, the method also constructs a realistic discriminant loss function to guide the generation process. The specific loss function is:

[0106]

[0107] Among them, -f 2(·) is the trained discriminator (binary classifier).

[0108] -S * Consistent with the S form in (b), the categories are true and false, that is, {0, 1}.

[0109] -Xi, τ and other parameters remain the same as in (b).

[0110] Then, based on (b), a realistic discriminant loss function is added to guide the diffusion model generation process:

[0111]

[0112] Except λ 2 Except for the newly introduced hyperparameters, other symbols remain consistent with those in (b).

[0113] like Figure 3 and Figure 4 As shown in the figure, the traditional image generation algorithm cannot construct the real distribution of sample images well, the types of defective product images are lacking, and the generated image quality is also fuzzy. The present invention adopts an improved diffusion model-based image data generation method, which uses the trained diffusion model to generate a large number of diverse and high-quality transistor defect samples. For the same original image, the transistor defect image samples generated by training are as follows: Figure 4 As shown. The present invention adds probability density reduction to the original diffusion model so that the diffusion model generates more diverse samples. In addition, the probability density reduction loss contains category information, which makes the samples generated by the present invention not generate pseudo-abnormal samples with noise like the basic diffusion model. On the other hand, in order to make the generated samples more realistic, the present invention also introduces a discriminative loss, which borrows the idea of ​​generative adversarial to make the generated samples more realistic.

[0114] Based on the above diffusion model-based image data generation method, a defective product detection method or detection equipment for transistor production can be constructed. Figure 5 As shown, the present invention can provide a transistor defect detection method, comprising:

[0115] S1, capturing original samples step: enabling an image acquisition device to capture a real product image of the transistor after production;

[0116] S2, defect generation sample formation step: taking the captured real product defect image as the original sample, adding noise to the original sample through the diffusion model, and then guiding the diffusion model to the area with reduced probability distribution density through the density reduction loss to perform denoising, so as to generate a new product defect image;

[0117] S3, product inspection step: using the new product defect image as a defect generation sample, determining the defect location of the defect generation sample, and marking the defects of the defect generation sample, and using the marked defect generation sample to inspect the transistor product after production.

[0118] The S1 step can be implemented by a standard module of an image acquisition device. For example, a digital camera or a video camera is used to capture images by scanning line by line or frame by frame. This is the most common way to acquire images and is suitable for most scenarios. In special scenarios, infrared sensors or thermal infrared cameras can also be used to capture infrared or thermal infrared images that are invisible to the human eye. Alternatively, the three-dimensional shape and texture information of an object can be obtained by utilizing technologies such as structured light, time of flight (TOF) or stereo cameras.

[0119] The defect generation sample formation step in step S2 has been described in detail in the aforementioned image generation step and will not be repeated here. Since the density reduction loss is used to guide the diffusion model to perform denoising processing on the area with reduced probability distribution density, compared with the transistor defect generation algorithm based on traditional image processing, this method uses the simulated discriminant loss to guide the diffusion model to generate defect samples with higher authenticity, which can adapt well to the actual generation environment. Based on generation algorithms such as GAN and VAE, this method obtains a large number of defect samples with the help of the denoising generation framework of the diffusion model. The defect samples are of higher quality than the fuzzy samples generated by VAE. In addition, the training process and generation process are more stable than GAN.

[0120] The product inspection step in step S3 can be done by manually marking the detection box, and then inputting the data and labels into target detectors such as YOLO to detect anomalies. In actual production, transistor defect samples are very difficult to obtain. This method can batch generate synthetic samples similar to real samples, providing a stable training data source for the subsequent defect detection stage. Figure 6As shown in the figure, the specific product inspection step S3 can be divided into: S31, data processing and manual labeling steps; S32, training YOLO series target detectors; S33, defect detection. In step S31, a large number of synthetic samples are first generated in batches according to the synthesis scheme. Subsequently, the synthetic samples are mixed with a small number of real samples, and each picture is manually labeled at the detection frame level to form a new data set. The new data set is then divided, that is, the number of training set data: the number of validation set data is equal to 8:2. Before training and testing, the data is rotated, color enhanced, center cropped and other processing methods are used to obtain augmented data. In step S32, the YOLO v5 model is used to perform fully supervised target detection training on the processed data. The training goal is that the model can accurately locate the defect area with the detection frame. After the model is trained, the model parameters are saved according to the number of training iterations with a step size of 10. In step S33, the trained models with different parameters are used to detect the validation set data respectively, and the models with higher detection performance are selected and deployed on the actual detection production line.

[0121] Reference Figure 7 , Figure 7 It is a schematic diagram of the framework of the computer-readable storage medium of the specific implementation method of the present application. The computer-readable storage medium 1 of the embodiment of the present application stores a computer program instruction, and the method provided by the present application is implemented when the program instruction is executed. Among them, the computer program instruction can form a program file and be stored in the above-mentioned computer-readable storage medium in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the various implementation methods of the present application. The aforementioned computer-readable storage medium 1 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0122] More specifically, Figure 8 The structural block diagram of an image generation model training device provided by an embodiment of the present invention is shown. The image generation model training device 2 includes a processor and a memory, the processor and the memory are coupled, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method provided by the present application are implemented. Specifically, it can be as follows Figure 8 As shown, in this embodiment, the image generation model training device 2 includes a memory, a processor and a communication link. The processor is coupled to the memory and the communication circuit respectively. The memory stores program data. The processor implements the image generation method in this application by executing the program data in the memory.

[0123] The processor may also be referred to as a processor (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0124] like Fig. 9 As shown, an embodiment of the present invention provides a defect detection device for transistor production, the device comprising an image acquisition device, a processor and a memory, the image acquisition device is used to capture the real product image after the transistor is produced; the processor and the memory are coupled, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the defect detection method of the present application are implemented.

[0125] More specifically, the processor includes a neural network chip, which can be used as a device for some data processing classification reasoning and data processing functions, such as Fig. 9 As shown, in addition to the neural network chip, other common components (for example, central processing unit (CPU), graphics processing unit (GPU), application processor (AP), sensors, communication modules, etc.) and so on.

[0126] Such an electronic device may be included in various types of electronic devices such as personal computers (PCs), server devices, mobile devices, and embedded devices. Such a device may correspond to a hardware component included in, for example, a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robot, a medical device, etc., which performs, for example, speech recognition, image recognition, and image classification using a neural network. Therefore, the device may correspond to a dedicated hardware accelerator installed on the above-mentioned electronic device, or to a hardware accelerator such as a neural processor (NPU), a tensor processor (TPU), or a neural engine (Neural Engine) (a neural engine is a dedicated module for driving a neural network), etc.).

[0127] The memory may be hardware that stores various neuromorphic data processed in the neural network chip. The memory may be DRAM, etc. The memory may include at least one of a volatile memory or a non-volatile memory. Non-volatile memories include ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, phase change random access memory (PRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memories include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, etc. In one embodiment, the memory may include at least one of HDD, SSD, CF, SD, micro SD, mini SD, xD, and a memory stick.

[0128] The device can analyze input data in real time based on a neural network to extract valid information, based on the extracted information. For example, the device can be applied to robotic devices (such as drones and advanced driver assistance systems (ADAS)), smart TVs, smart phones, medical devices, mobile devices, image display devices, measuring devices, and IoT devices, etc. In addition, the device can be installed on at least one of various other types of electronic devices. The device may include a processor, RAM, a neural network chip, a memory, various sensors including an image capture device, and a communication module. It may also include an input / output module, a security module, a power control device, and the like. The processor can control the overall operation of the electronic device. The processor may include one processor core (single-core) or multiple processor cores (multi-core). The processor may process or execute programs and / or data stored in a memory. In some example embodiments, the processor may control the function of the neural network chip by executing a program stored in a memory. The processor may be implemented as a CPU, a GPU, an AP, and the like.

[0129] RAM can temporarily store programs, data or instructions. For example, programs and / or data stored in the memory can be temporarily stored in RAM according to the control or boot code of the processor. RAM can be implemented as dynamic RAM (DRAM), static RAM (SRAM), etc.

[0130] The neural network chip can perform calculations of the neural network based on the received input data, and generate an information signal based on the calculation results. The neural network may include, but is not limited to, CNN, RNN, FNN, deep belief network, restricted Boltzmann machine, etc. According to an example embodiment, the neural network chip is a hardware accelerator dedicated to the neural network or a device including a hardware accelerator, and may include a neuromorphic processor according to the above-mentioned example embodiment.

[0131] The information signal may include one of various types of recognition signals (such as a voice recognition signal, an object recognition signal, an image recognition signal, and a biometric information recognition signal). For example, the neural network chip may receive frame data included in a video stream as input data, and generate a recognition signal of an object included in an image represented by the frame data from the frame data. In other example embodiments, the neural network chip may receive various types of input data according to the type or function of the electronic device on which the electronic device is installed, and may generate a recognition signal according to the input data.

[0132] The memory is a storage location for storing data, and can store an operating system (OS), various programs, and various data. In one embodiment, the memory can store intermediate results generated during the computational processing of the neural network chip.

[0133] The sensor may collect peripheral information of the electronic device on which the electronic device is mounted. The sensor may sense or receive a signal (e.g., a video signal, an audio signal, a magnetic signal, a bio-signal, a touch signal, etc.) from the outside of the electronic device and convert the sensed or received signal into data. For example, the sensor may include at least one of various types of sensing devices (such as a microphone, an imaging device, an image sensor, a light detection and ranging (LIDAR) sensor, an ultrasonic sensor, an infrared sensor, a biosensor, and a touch sensor).

[0134] The sensor may provide the converted data as input data to the neural network chip. For example, the sensor may include an image sensor that generates a video stream by photographing the external environment of the electronic device, and sequentially provides continuous data frames of the video stream to the neural network chip as input data. In other example embodiments, the sensor may provide various types of data to the neural network chip.

[0135] The communication module may be provided with various wired or wireless interfaces capable of communicating with an external device. For example, the communication module may include a communication interface capable of accessing a wired local area network (LAN), a wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi)), a wireless personal area network (WPAN) (such as Bluetooth), a wireless universal serial bus (wireless USB), Zigbee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC) or a mobile cellular network (such as third generation (3G), fourth generation (4G) and long term evolution (LTE)).

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A transistor defect detection method, characterized in that: include: S1, capturing original samples step: enabling an image acquisition device to capture a real product image of the transistor after production; S2, defect generation sample formation step: taking the captured real product defect image as the original sample, adding noise to the original sample through the diffusion model, and then guiding the diffusion model to the area with reduced probability distribution density for denoising through density reduction loss modeling, to generate a new product defect image; S3, product inspection step: using the new product defect image as a defect generation sample, determining the defect location of the defect generation sample, and marking the defects of the defect generation sample, and using the marked defect generation sample to inspect the transistor product after production.

2. The detection method according to claim 1, characterized in that: In step S2 of the method, the original samples are further subjected to an image preprocessing step, wherein the image preprocessing step includes image cleaning and image cropping; the image cleaning includes image classification of the original samples according to the defect type in the product image, and the number of images of the original samples in each defect type is not less than 50; The image cropping includes performing image cropping of the original sample, and cropping the original sample into a uniform size.

3. The detection method according to claim 1, characterized in that: In step S2 of the method, the backbone network of the diffusion model adopts the UNet structure, and the original sample is forward-added with noise according to the Markov process of formula (F1): In formula (F1) t is the time step t; Where x0 is the original sample without noise; the q(·) operation represents the Markov process; represents Gaussian distribution, x t Obey Gaussian distribution; β t is the variance at time step t, and is not a constant; I is the variance of the added Gaussian noise; Preferably, the probability density distribution used for denoising in step S2 of the method is constructed by a Gaussian model. Assuming that there are k types of defects in the product image, the probability distribution of k types of transistor defect images constructed by the multivariate Gaussian model is (F2): in The mean is μ k , the variance is ∑ k Gaussian model of; x is the x-th type of transistor defect image; Preferably, in step S2 of the method, a density reduction loss function (F3) is determined according to the probability distribution: Where: f1(·) is the trained classifier; xi is the kth i The i-th transistor defect image in the class defect; N is the total number of transistor defect images in the class; τ represents the temperature coefficient; Preferably, the image generation process for removing noise in step S2 of the method is stepwise generated in the manner (F4): where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; λ1 is a hyperparameter; is the normalized gradient; is the density reduction loss function; Preferably, in step S2 of the method, a pseudo-realistic discrimination loss function (F5) is determined according to the probability distribution: Among them: f2(·) is the trained discriminator, which is a binary classifier; S * The category is true or false, that is, {0, 1}x i For the kth i The i-th transistor defect image in the class defect; N is the total number of transistor defect images in the class; τ represents the temperature coefficient; Preferably, the image generation process for removing noise in step S2 of the method is performed step by step in the manner (F6): where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; λ1 and λ2 are both hyperparameters; is the normalized gradient; is the density reduction loss function.

4. The detection method according to claim 1, characterized in that: The method of using the marked defects to generate samples to detect transistor products after production in step S3 of the method is the YOLO target detection method.

5. A method for generating image data based on a diffusion model, characterized in that The method comprises the following steps: (S21) using a diffusion model to gradually increase noise processing on the original image data; (S22) The diffusion model is guided to perform denoising to the area where the probability distribution density is reduced through density reduction loss modeling to generate the required image data.

6. The image data generating method according to claim 5, characterized in that: The backbone network of the diffusion model in the method adopts the UNet structure, and the original sample is forward-added with noise according to the Markov process of formula (F1): In formula (F1) t is the time step t Where x0 is the original sample without noise; the q(·) operation represents the Markov process; represents Gaussian distribution, x t Obey Gaussian distribution; β t is the variance at time step t, and is not a constant; I is the variance of the added Gaussian noise; Preferably, the probability density distribution used for denoising in step S2 of the method is constructed by a Gaussian model. Assuming that there are k types of defects in the product image, the probability distribution of k types of transistor defect images constructed by the multivariate Gaussian model is (F2): in The mean is μ k , the variance is ∑ k Gaussian model of; x is the x-th type of transistor defect image; Preferably, in step S2 of the method, a density reduction loss function (F3) is determined according to the probability distribution: Where: f1(·) is the trained classifier; x i For the kth i The i-th transistor defect image in the class defect; N is the total number of transistor defect images in the class; τ represents the temperature coefficient; Preferably, the image generation process for removing noise in step S2 of the method is stepwise generated in the manner (F4): where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; λ1 is a hyperparameter; is the normalized gradient; is the density reduction loss function; Preferably, in step S2 of the method, a pseudo-realistic discrimination loss function (F5) is determined according to the probability distribution: Among them: f2(·) is the trained discriminator, which is a binary classifier; S * The category is true or false, that is, {0,1}x i For the kth i The i-th transistor defect image in the class defect; N is the total number of transistor defect images in the class; τ represents the temperature coefficient; Preferably, the image generation process for removing noise in step S2 of the method is performed step by step in the manner (F6): where x t-1 is the noisy image at time t-1, x t is the noisy image at time t; μ θ is the mean predicted by the diffusion model; is the standard deviation predicted by the diffusion model; λ1 and λ2 are both hyperparameters; is the normalized gradient; is the density reduction loss function.

7. A defect detection device for transistor production, characterized in that: The device comprises an image acquisition device, a processor and a memory, The image acquisition device is used to capture the real product image of the transistor after production; The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

8. An image generation model training device, characterized in that: Including processor, The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method described in any one of claims 5 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a processor, the processor performs the method according to any one of claims 5 to 6.

10. A computer program product, characterized in that The computer program product comprises software code for executing the steps of the method according to any one of claims 5-6.

Citation Information

Cited By

  • PCB welding spot detection method and device, electronic equipment and storage medium

    CN120047402A

  • Defect sample generation method and device, equipment and storage medium

    CN121415183A

  • A method, apparatus, device, and storage medium for generating defect samples.

    CN121415183B