Defect sample generation method and device

Abstract

The invention provides a defect sample generation method and equipment, and the method comprises the steps: obtaining a defect-free image of a to-be-detected product, adding noise to the defect-free image through a diffusion model, and obtaining a noise image corresponding to the defect-free image; estimating estimated noise corresponding to the added noise through a neural network model; calculating disturbance noise corresponding to the estimated noise through the disturbance coefficient and the estimated noise; performing de-noising processing on the noise image through the diffusion model and the disturbance noise to obtain a disturbed image; and merging the defect-free image, the disturbed image and the mask to obtain a defect sample of the to-be-detected product. According to the method, the defect sample is generated through the defect-free image, the defect sample does not need to be collected in advance, and the defect type of the defect sample does not need to be labeled, so that the time consumed in the collection and labeling process of the defect sample is shortened, and the generation efficiency of the defect sample is correspondingly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of defect detection, and particularly relates to a defect sample generation method and device. BACKGROUND

[0002] In the field of industrial defect detection, it is often necessary to detect whether a product has defects through defect samples. At present, defect samples are usually generated through deep learning. In the process of generating defect samples, the defect samples need to be collected first, and then the types of the defect samples are labeled by artificial labeling. Then, the labeled defect samples are used to train a deep learning model to obtain a trained deep learning model, and then the trained deep learning model is used to generate defect samples.

[0003] However, this method requires a large amount of time for collecting and labeling defect samples, and therefore the generation efficiency of defect samples is low. SUMMARY

[0004] To solve the above problems, the embodiments of the present application provide a defect sample generation method to improve the generation efficiency of defect samples.

[0005] The first aspect of the embodiments of the present application provides a defect sample generation method, which comprises the following steps:

[0006] obtaining a defect-free image comprising a product to be inspected;

[0007] adding noise to the defect-free image through a diffusion model to obtain a noise image corresponding to the defect-free image;

[0008] estimating an estimated noise corresponding to the noise through a neural network model;

[0009] calculating a perturbation noise corresponding to the estimated noise through a perturbation coefficient and the estimated noise corresponding to the noise;

[0010] performing denoising processing on the noise image through the diffusion model and the perturbation noise to obtain a perturbed image;

[0011] obtaining a defect sample of the product to be inspected by merging the defect-free image, the perturbed image and a mask.

[0012] Through the above scheme, defect samples can be generated through defect-free images, and there is no need to pre-collect defect samples and label the defect types of the defect samples, thereby reducing the time consumed by the processes of collecting and labeling the defect samples, and accordingly improving the generation efficiency of the defect samples.

[0013] In another aspect of the embodiments of the present application, the denoising processing of the noisy image by the diffusion model and the perturbation noise comprises:

[0014] In the case that the denoising processing comprises at least two denoising steps, at least two random numbers corresponding to the denoising steps are generated by a preset random number generation algorithm.

[0015] The random number corresponding to the target denoising step is compared with a preset threshold, and the target denoising step is any one of the denoising steps.

[0016] If the comparison result of the random number corresponding to the target denoising step and the preset threshold indicates that the target denoising step needs to be perturbed, the perturbation noise corresponding to the target denoising step is removed from the image to be denoised by the diffusion model.

[0017] If the comparison result of the random number corresponding to the target denoising step and the preset threshold indicates that the target denoising step does not need to be perturbed, the estimated noise corresponding to the target denoising step is removed from the image to be denoised by the diffusion model.

[0018] In the case that the target denoising step is the first denoising step of the denoising processing, the image to be denoised is the noisy image, and in the case that the target denoising step is other denoising step of the denoising processing, the image to be denoised is the image obtained after the previous denoising step is executed.

[0019] The above scheme can determine whether the target denoising step needs to be perturbed according to the comparison result of the random number corresponding to each denoising step and the preset threshold, and only in the case of need, the perturbation noise corresponding to the target denoising step is removed from the image to be denoised. In the case of no need, the estimated noise corresponding to the target denoising step is removed from the image to be denoised, that is, the target denoising step is not perturbed, so that the deviation of the image obtained after the reverse process from the product to be inspected can be avoided, and the authenticity of the defect sample can be improved.

[0020] In another aspect of the embodiments of the present application, the calculation of the perturbation noise corresponding to the estimated noise by the perturbation coefficient and the noise comprises:

[0021] In the case that the diffusion model adds noise to the defect-free image by at least two adding steps, the weight coefficients of the estimated noises corresponding to the noises added by the at least two adding steps are determined according to a preset weight coefficient distribution method.

[0022] multiply the estimated noise, the weight coefficient corresponding to the estimated noise and the perturbation coefficient to obtain a perturbation noise corresponding to the estimated noise.

[0023] Through the above scheme, the perturbation noise corresponding to the estimated noise can be obtained based on the estimated noise and the perturbation coefficient. Moreover, the scheme can adjust the frequency of adding the perturbation by adjusting the size of the preset threshold, so as to control the intensity of the perturbation.

[0024] In another aspect of the embodiments of the present application, before the merging processing of the defect-free image, the perturbed image and the mask, the method further comprises:

[0025] determining a two-dimensional fractal noise corresponding to the defect-free image according to the image length and the image width of the defect-free image, and the Berlin noise and the preset number of superposition layers of the Berlin noise;

[0026] determining the mask by binarizing the two-dimensional fractal noise.

[0027] Through the above scheme, the corresponding mask can be determined. Since the two-dimensional fractal noise has self-similarity and multi-scale characteristics, the mask has similarity at different scales, and is more consistent with the contour characteristics of the defect at different scales.

[0028] In another aspect of the embodiments of the present application, after the binarizing of the two-dimensional fractal noise to determine the mask, the method further comprises:

[0029] filling the broken part in the mask by dilating the mask;

[0030] removing the redundant part in the mask by eroding the filled mask.

[0031] Through the above scheme, the optimization of the mask can be realized, so as to improve the authenticity of the defect sample.

[0032] In another aspect of the embodiments of the present application, the merging processing of the defect-free image, the perturbed image and the mask to obtain the defect sample of the product under test comprises:

[0033] determining a first image region in the perturbed image corresponding to the mask by merging the region in the mask with the perturbed image;

[0034] determining a second image region in the defect-free image corresponding to the mask by merging the region outside the mask with the defect-free image;

[0035] obtaining the defect sample by merging the first image region and the second image region.

[0036] In another aspect of the embodiments of the present application, after the defect sample is obtained by merging the first image region and the second image region, the method further includes:

[0037] determining a first shortest distance between a first pixel point in the first image region in the defect sample and the mask boundary, the first pixel point being any one of the pixel points in the first image region;

[0038] adjusting the transparency of the first pixel point according to the first shortest distance, wherein the shorter the first shortest distance, the higher the transparency of the first pixel point.

[0039] According to the above scheme, the transparency of the pixels in the first image region can be adjusted, the transition of the region inside the mask to the outside is realized, and thus the fusion effect of the defect region and the background region is improved, and the fusion of the defect region and the background region is more natural.

[0040] In another aspect of the embodiments of the present application, after the defect sample is obtained by merging the first image region and the second image region, the method further includes:

[0041] determining a second shortest distance between a second pixel point in the second image region in the defect sample and the mask boundary, the second pixel point being any one of the pixel points in the second image region;

[0042] adjusting the transparency of the second pixel point according to the second shortest distance, wherein the longer the second shortest distance, the higher the transparency of the second pixel point.

[0043] According to the above scheme, the transparency of the pixels in the second image region can be adjusted, the transition of the region outside the mask to the inside is realized, and thus the fusion effect of the defect region and the background region is further improved, and the fusion of the defect region and the background region is more natural.

[0044] In another aspect of the embodiments of the present application, the obtaining of the defect-free image of the product to be inspected includes:

[0045] obtaining a shooting image obtained after a product to be inspected is shot by defect detection;

[0046] locating a region of interest in the shooting image through a preset locating algorithm;

[0047] cutting the shooting image through the located region of interest to obtain a defect-free image of the product to be inspected.

[0048] With the progress of industrial production technology, the production process of products is gradually improved, and the yield of products is gradually improved, so it is easier to obtain the products to be inspected through defects, and it is easier to obtain the defect-free image through the above scheme.

[0049] The second aspect of the embodiment of the application provides a defect sample generation device, comprising:

[0050] a controller, and configured to:

[0051] obtain a defect-free image comprising a product to be inspected;

[0052] add noise to the defect-free image through a diffusion model to obtain a noise image corresponding to the defect-free image;

[0053] estimate an estimated noise corresponding to the noise through a neural network model;

[0054] calculate a perturbation noise corresponding to the estimated noise through a perturbation coefficient and the estimated noise corresponding to the noise;

[0055] perform denoising processing on the noise image through the diffusion model and the perturbation noise to obtain a perturbed image;

[0056] obtain a defect sample of the product to be inspected by merging the defect-free image, the perturbed image and a mask.

[0057] The embodiment of the application provides a defect sample generation method and device, which improves the reverse process of the diffusion model. Through the improvement, in the reverse process, the perturbation noise added on the basis of the estimated noise is used to perform denoising processing on the image obtained through the forward process, so as to obtain a perturbed image comprising defects. Then, the defect sample is obtained by merging the defect-free image, the perturbed image and a mask.

[0058] The scheme of the embodiment of the application generates the defect sample through the defect-free image, without pre-acquiring the defect sample and labeling the defect type of the defect sample, so that the time consumed by the acquisition and labeling process of the defect sample is reduced, and the generation efficiency of the defect sample is correspondingly improved.

[0059] In addition, in the reverse process, the de-noising is performed by using the disturbance noise, and the disturbance noise is added on the basis of the estimated noise. In this case, each region of the obtained disturbed image usually includes defects. Therefore, if the mask is arranged at different positions in the merging process of the defect-free image, the disturbed image and the mask, the defects included in the obtained defect sample are usually different, so that diversified defect samples can be obtained, that is, diversified defect samples can be obtained by performing the method provided in the present application once, so that the generation efficiency of the defect sample can be further improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 Flowchart for generating defect samples by deep learning;

[0061] Figure 2 Example diagram of the working process of the diffusion model;

[0062] Figure 3 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0063] Figure 4 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0064] Figure 5 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0065] Figure 6 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0066] Figure 7 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0067] Figure 8 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0068] Figure 9 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0069] Figure 10 Schematic diagram of merging a defect-free image, a disturbed image and a mask according to some example embodiments;

[0070] Figure 11 Workflow schematic diagram of the defect sample generation method according to some example embodiments;

[0071] Figure 12 Workflow diagram of a defect sample generation method according to some example embodiments;

[0072] Figure 13 Flowchart of a defect sample generation method according to some example embodiments. DETAILED DESCRIPTION

[0073] The embodiments will be described in detail with reference to the drawings, wherein the same reference numerals will be used for similar elements throughout several views. The following description is made with reference to the accompanying drawings, in which the same or similar components have the same reference numerals. The following description of embodiments is not representative of all embodiments consistent with the present application. Rather, it is merely an example of a system and method consistent with some aspects of the present application as detailed in the claims.

[0074] It should be noted that the brief description of terms in the present application is only for the convenience of understanding the following description of embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood in accordance with their ordinary and customary meanings.

[0075] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0076] The terms "include" and "have" and any variations thereof are intended to cover but not exclusively include, for example, a product or device that includes a list of components does not necessarily have to include all of the components clearly listed, but can include other components not clearly listed or inherent to such products or devices.

[0077] In some embodiments, the defect sample can be generated by deep learning, and the process is as shown in Figure 1 First, the defect sample is collected, and the defect type of the defect sample is manually labeled, and the defect sample is stored in the corresponding defect database. Then, the defect sample in the defect database is used to train a deep learning generation model, and the trained deep learning generation model is used to generate new defects by using a deep learning generation algorithm and the defect sample in the defect database. Finally, the new defects are combined with normal samples to obtain new defect samples, and the defect type of the new defect samples is the same as that of the defect samples used to train the deep learning generation model, thereby realizing the generation of defect samples.

[0078] However, the method needs to collect defect samples and manually label the types of the defect samples, and the collection and labeling process consumes a large amount of time, resulting in a low generation efficiency of the defect samples.

[0079] To solve this problem, an embodiment of the present application provides a defect sample generation method. The defect sample generation method improves the reverse process of a diffusion model (Denoising Diffusion Probabilistic Models, DDPM), and generates defect samples of a product to be inspected by using a non-defect image of the product to be inspected and the improved diffusion model.

[0080] For ease of understanding, the diffusion model is introduced as follows.

[0081] The core idea of the diffusion model is to diffuse data to a noise distribution by gradually adding noise, and then gradually remove noise to restore the original data through a reverse process.

[0082] The diffusion model includes a forward process (also referred to as a diffusion process) and a reverse process (also referred to as a denoising process), wherein the forward process is to obtain a noise image corresponding to an original image by gradually adding noise to the original image, and each step included in the forward process is referred to as an adding step; the reverse process is to estimate the noise added in each step of the forward process, and then use the estimated noise to denoise the noise image through multiple steps to obtain the original image, and each step included in the reverse process is referred to as a denoising step.

[0083] Referring to FIG. 1, Figure 2 which shows an example diagram of a working process of a diffusion model, in this example, Figure 2 the upper half of FIG. 1 represents the forward process, and the direction indicated by the arrow is used to represent the execution order of each step in the forward process, wherein x0 corresponds to an original image, x1 is an image obtained after the first adding step, x t is an image obtained after the tth adding step, and the forward process has T steps in total.

[0084] In addition, Figure 2 the lower half of FIG. 1 represents the reverse process, and the direction indicated by the arrow is used to represent the execution order of each step in the reverse process, wherein y0 is a noise image including noise obtained after the completion of the forward process, y1 is an image obtained after the first denoising step, y t is an image obtained after the tth denoising step, and the similarity between the image and the original image is high.

[0085] The diffusion model before improvement can be obtained by Figure 2The reverse process is shown in the lower half of FIG. 1. Denoising is performed using the estimated noise to obtain an image that is similar to the original image.

[0086] The scheme provided by the embodiments of the present application can add disturbance to the estimated noise by improving the reverse process. The diffusion model performs the reverse process using the estimated noise with the added disturbance. Since the disturbance is added in the reverse process, the image obtained after the reverse process is completed has a low similarity to the original image, which can be referred to as an abnormal image.

[0087] In this case, if the original image is a defect-free image including the product to be inspected, the abnormal image includes various defects that can occur in the product to be inspected. Therefore, the defect sample can be obtained based on the abnormal image.

[0088] The scheme of the embodiments of the present application generates a defect sample from a defect-free image, without the need to pre-collect defect samples and label the defect types of the defect samples. Therefore, the time consumed in the process of collecting and labeling the defect samples is reduced, and the generation efficiency of the defect sample is correspondingly improved.

[0089] The defect sample generation method is described below through various embodiments.

[0090] Referring to Figure 3 In the defect sample generation method provided by an embodiment of the present application, the following steps are included.

[0091] In step S100, a defect-free image including a product to be inspected is obtained.

[0092] In the embodiments of the present application, referring to Figure 4 The defect-free image can be obtained through the following operations, as shown in the workflow schematic diagram of FIG. 2.

[0093] In step S110, a photographed image obtained after a product to be inspected passes defect detection is obtained.

[0094] If a product to be inspected passes defect detection, it indicates that the product to be inspected does not include defects or includes defects that are too subtle to be ignored.

[0095] In step S120, a region of interest in the photographed image is located through a preset positioning algorithm.

[0096] The region of interest refers to a region corresponding to the defect sample to be generated this time. The method for locating the region of interest in the photographed image can include position matching based on a traditional template or a deep learning positioning algorithm (for example, a target detection algorithm), and the present application does not limit this.

[0097] Step S130: Cut the captured image using the located region of interest to obtain a defect-free image of the product to be inspected.

[0098] With the advancement of industrial production technology, the production process of products is gradually improving, and the yield of products is also gradually increasing. Therefore, it is easier to obtain defective products to be inspected. Correspondingly, it is easier to obtain defect-free images through the above steps.

[0099] Step S200: adding noise to a non-defective image including the product to be inspected by using a diffusion model to obtain a noise image corresponding to the non-defective image.

[0100] The diffusion model is a trained diffusion model, and the defect-free image can be used as training data during the training process. Through training, the diffusion model can, when receiving a defect-free image, obtain a noise image corresponding to the defect image through a forward process, and then, through an unmodified reverse process, denoise the noise image by estimating the noise to obtain a denoised image. The denoised image has a high similarity to the defect-free image.

[0101] Furthermore, this step involves gradually adding noise to the defect-free image according to at least two pre-defined addition steps, which constitutes the forward process of the diffusion model. In this step, a defect-free image of the product to be inspected can be input into the diffusion model. After the diffusion model completes the forward process, the image obtained is the noise image corresponding to the defect-free image.

[0102] Assume that the image corresponding to x0 is a defect-free image of the product to be inspected, and x t-1 is the image obtained after the t-1th adding step, x t is the image obtained after the t-th adding step, then x t-1 to x t The conditional probability distribution of the diffusion process is expressed as follows:

[0103]

[0104] By deducing from formula (1), we can obtain the following formula:

[0105]

[0106] In formula (1) and formula (2), α t =1-β t , x0 represents a defect-free image of the product to be inspected; t represents the number of rounds of adding steps, which is round t, and the maximum number of rounds is T; x t represents the image obtained after the t-th addition step; is a set of pre-set parameters, exemplary, can be {0.1, 0.2, …, 0.8, …}; N() represents a normal distribution, q(x t |x t-1 ) represents the conditional probability distribution from x t-1 to x t in the forward process, q(x t |x0) represents the conditional probability distribution of the diffusion process from x0to x t , and I represents a unit matrix.

[0107] Step S300, estimating the estimated noise corresponding to the noise through the neural network model.

[0108] Wherein, the neural network model is pre-trained, and the estimated noise corresponding to each adding step obtained through the trained neural network model is close to the noise added in each corresponding adding step. In this case, the diffusion model can obtain an image extremely close to the defect-free image when performing the reverse process through the estimated noise.

[0109] Step S400, calculating the perturbation noise corresponding to the estimated noise through the perturbation coefficient and the estimated noise corresponding to the noise.

[0110] In the embodiment of the present application, the perturbation coefficient can be pre-set, and then the perturbation noise corresponding to the estimated noise is calculated through the perturbation coefficient and the estimated noise.

[0111] Step S500, performing denoising processing on the noise image through the diffusion model and the perturbation noise to obtain the perturbed image.

[0112] According to the foregoing introduction of the diffusion model, if the reverse process of the diffusion model is not improved, the corresponding estimated noise will be gradually subtracted from the noise image in each denoising step in the reverse process, so that an image extremely close to the original image without performing the forward process can be obtained.

[0113] And in the embodiment of the present application, the reverse process of the diffusion model is improved, and the improved reverse process performs denoising processing on the noise image through the perturbation noise, and the perturbation noise adds perturbation on the basis of the estimated noise. Therefore, the similarity between the obtained perturbed image and the original image is low, and compared with the defect-free image, the perturbed image is usually an abnormal image with defects in each region.

[0114] Step S600, obtaining the defect sample of the product to be inspected by merging the defect-free image, the perturbed image and the mask.

[0115] Since the disturbed image is usually an abnormal image with defects in each region, the sample obtained by merging the defect-free image, the disturbed image and the mask is the defect sample containing defects.

[0116] The embodiment of the present application provides a defect sample generation method, which improves the reverse process of the diffusion model. In the reverse process, the disturbed noise added on the basis of the estimated noise is used to perform denoising processing on the image obtained through the forward process, so that a disturbed image containing defects is obtained. Then, the defect sample is obtained by merging the defect-free image, the disturbed image and the mask.

[0117] The scheme of the embodiment of the present application generates the defect sample from the defect-free image, and does not need to pre-collect the defect sample and label the defect type of the defect sample, thereby reducing the time consumed by the collection and labeling process of the defect sample, and correspondingly improving the generation efficiency of the defect sample.

[0118] In addition, in the reverse process, the denoising processing is performed by the disturbed noise, and the disturbed noise adds the disturbance on the basis of the estimated noise. In this case, each region of the obtained disturbed image usually contains defects. Therefore, if the mask is set at different positions in the merging process of the defect-free image, the disturbed image and the mask, the defects contained in the obtained defect sample are usually different, so that diversified defect samples can be obtained. That is, once the method provided in the embodiment of the present application is performed, diversified defect samples can be obtained, thereby further improving the generation efficiency of the defect sample.

[0119] Referring to Figure 5 In another embodiment of the present application, the step S500 discloses that the denoising processing is performed on the noise image by the diffusion model and the disturbed noise to obtain the disturbed image, comprising the following steps:

[0120] In the case that the denoising processing process includes at least two denoising steps, the random number generation algorithm is used to generate at least two random numbers corresponding to the denoising steps, respectively.

[0121] In the case that the denoising processing process includes at least two denoising steps, the random number generation algorithm is used to generate at least two random numbers corresponding to the denoising steps, respectively.

[0122] If the random number generation algorithm randomly generates a random number between 0 and 1, the preset threshold value can be a positive number between 0 and 1, for example, 0.7. Of course, the preset threshold value can also be other numerical values, which are not limited in the embodiment of the present application.

[0123] The comparison result of the random number corresponding to the target denoising step and the preset threshold value is used to indicate whether the target denoising step needs to be disturbed. In the embodiments of the present application, it can be preset that the target denoising step needs to be disturbed when the random number corresponding to the target denoising step is greater than the preset threshold value; or, it can also be preset that the target denoising step does not need to be disturbed when the random number corresponding to the target denoising step is greater than the preset threshold value.

[0124] In step S530, if the comparison result of the random number corresponding to the target denoising step and the preset threshold value indicates that the target denoising step needs to be disturbed, the disturbance noise corresponding to the target denoising step is removed from the to-be-denoised image through the diffusion model.

[0125] In the case that the target denoising step is the first denoising step of the denoising process, the to-be-denoised image is a noise image; in the case that the target denoising step is other denoising step of the denoising process, the to-be-denoised image is an image obtained after the previous denoising step is performed.

[0126] In step S540, if the comparison result of the random number corresponding to the target denoising step and the preset threshold value indicates that the target denoising step does not need to be disturbed, the estimated noise corresponding to the target denoising step is removed from the to-be-denoised image through the diffusion model.

[0127] If the disturbance noise corresponding to each denoising step in the reverse process is removed from the to-be-denoised image, the disturbance amplitude may be too large, resulting in that the image obtained after the reverse process ends deviates from the product to be inspected.

[0128] To solve the problem, in the embodiments of the present application, a random number generation algorithm is used to generate a random number corresponding to each denoising step, and then the comparison result of the random number and the preset threshold value is used to determine whether the target denoising step needs to be disturbed, and only in the case of need, the disturbance noise corresponding to the target denoising step is removed from the to-be-denoised image. In the case of no need, the estimated noise corresponding to the target denoising step is removed from the to-be-denoised image, that is, the target denoising step is not disturbed.

[0129] Through the embodiments of the present application, the disturbance amplitude can be avoided to be too large, and the deviation of the image obtained after the reverse process from the product to be inspected can be avoided to be too large, which is beneficial to improve the authenticity of the defect sample.

[0130] Further, in the embodiments of the present application, the size of the preset threshold value can be adjusted to adjust the frequency of disturbance addition, so as to control the disturbance intensity.

[0131] Referring to Figure 6 In another embodiment of the present application, the step S400 discloses that the disturbance noise corresponding to the estimated noise is calculated by the disturbance coefficient and the estimated noise corresponding to the noise, including the following steps:

[0132] Step S410: When the diffusion model adds noise to the defect-free image through at least two adding steps, determine the weight coefficients of the estimated noise corresponding to the noise added in the at least two adding steps according to a preset weight coefficient allocation method.

[0133] In some embodiments, the weight coefficients of the estimated noises corresponding to the noises added in each adding step may be set to 1.

[0134] Of course, the weight coefficients of the estimated noise corresponding to the noise added in each adding step may also be set to other values, which is not limited in the embodiment of the present application.

[0135] Step S420: Calculate the product of the estimated noise, the weight coefficient corresponding to the estimated noise, and the disturbance coefficient to obtain the disturbance noise corresponding to the estimated noise.

[0136] In one example, the weight coefficient of the estimated noise corresponding to the noise added in each adding step is 1, and the disturbance noise can be determined by the following formula:

[0137]

[0138] Where τ is the perturbation coefficient, ∈ θ (x t ,t) is the estimated noise corresponding to the t-th denoising step, is the disturbance noise corresponding to the t-th denoising step, and is the parameter of the neural network model used to estimate the estimated noise.

[0139] If θ is set as the parameter of the neural network model, after the t-th step denoising process through the unimproved reverse process, the mean of the noise distribution of the obtained image can be expressed as the following formula:

[0140]

[0141] Accordingly, after the t-th step denoising process through the unimproved reverse process, the probability distribution of the noise of the obtained image can be expressed as the following formula:

[0142] p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),∑ θ (x t ,t)) Formula (5).

[0143] Among them, ε represents the noise added by the adding step corresponding to the t-th denoising step, ∈ θ(x t ,t) is the estimated noise corresponding to the t-th de-noising step, μ θ (x t ,t) represents the mean of the noise distribution of the image obtained after the t-th de-noising step by the unimproved inverse process, Σ θ (x t ,t) represents the variance of the probability distribution of the noise of the image obtained after the t-th de-noising step by the unimproved inverse process, p θ (x t-1 |x t ) represents the conditional probability distribution of the noise of the image obtained after the t-th de-noising step by the unimproved inverse process.

[0144] The mean of the noise distribution of the image obtained after the t-th de-noising step by the improved inverse process can be represented by the following formula:

[0145]

[0146] wherein, represents the mean of the noise distribution of the image obtained after the t-th de-noising step by the improved inverse process.

[0147] Further, the following formula can also be obtained:

[0148]

[0149] In the above formula, Σ θ (x t ,t)τ 2 represents the variance of the probability distribution of the noise of the image obtained after the t-th de-noising step by the improved inverse process, represents the conditional probability distribution of the noise of the image obtained after the t-th de-noising step by the improved inverse process.

[0150] In the scheme provided in the present application, the defect sample is determined based on the merging processing of the defect-free image, the perturbed image and the mask. Referring to Figure 7 In another embodiment of the present application, the following steps are further provided:

[0151] Step S700, before the merging processing of the defect-free image, the perturbed image and the mask, the two-dimensional fractal noise corresponding to the defect-free image is determined according to the image length and the image width of the defect-free image, and the Berlin noise and the preset number of superposition layers of the Berlin noise.

[0152] Exemplarily, the two-dimensional fractal noise can be generated by the following formula:

[0153]

[0154] In the above formula, F(h, w) represents the two-dimensional fractal noise, h represents the image length of the defect-free image, w represents the image width of the defect-free image, Perlin represents the Perlin noise, f0 represents the base frequency, a0 represents the base amplitude, l represents the frequency growth factor, p represents the amplitude attenuation factor, and n represents the preset number of superimposed layers of the Perlin noise.

[0155] In step S800, a mask is determined by binarizing the two-dimensional fractal noise.

[0156] In some embodiments, in the binarization process, the value inside the mask is set to 1, and the value outside the mask is set to 0.

[0157] The embodiments of the present application can generate two-dimensional fractal noise corresponding to the defect-free image, and then determine the corresponding mask by binarizing the two-dimensional fractal noise. Since the two-dimensional fractal noise has self-similarity and multi-scale characteristics, the mask has similarity at different scales, and is more consistent with the contour characteristics of defects at different scales.

[0158] In addition, in the Figure 7 embodiments of the present application do not limit the actual execution process.

[0159] Further, after determining the mask by binarizing the two-dimensional fractal noise, referring to Figure 8 , the method further includes:

[0160] In step S900, the broken part in the mask is filled by dilating the mask.

[0161] The broken part in the mask generally refers to a small discontinuous region or gap included in the mask. The mask generation process may be affected by noise and other reasons, which may cause the existence of the broken part in the mask.

[0162] The dilating process of the mask can fill the broken part, so that the mask is optimized.

[0163] In step S1000, the mask after filling is eroded to remove the redundant part in the mask.

[0164] The redundant part in the mask can include parts not belonging to the region within the mask, which can be caused by inaccurate selection of the binary threshold, noise interference, etc., so that some background regions that should be shielded are incorrectly marked as the region within the mask, or can include some isolated small regions of the region within the mask, such as small noise, isolated pixels, etc.

[0165] By performing an erosion operation on the filled mask, the redundant part in the mask can be removed, and the mask is further optimized.

[0166] The above scheme can optimize the mask, thereby improving the authenticity of the defect sample.

[0167] Referring to Figure 9 In another embodiment of the present application, the defect sample of the product to be inspected can be obtained by merging the non-defect image, the perturbed image and the mask through the following operations:

[0168] In step S610, a first image region in the perturbed image corresponding to the region within the mask is determined by merging the region within the mask and the perturbed image.

[0169] In a feasible design of the present application, the merging of the region within the mask and the perturbed image can be realized by multiplying the pixels included in the region within the mask and the pixels included in the perturbed image element by element.

[0170] In step S620, a second image region in the non-defect image corresponding to the region outside the mask is determined by merging the region outside the mask and the non-defect image.

[0171] In a feasible design of the present application, the merging of the region outside the mask and the non-defect image can be realized by multiplying the pixels included in the region outside the mask and the pixels included in the non-defect image element by element.

[0172] In step S630, the defect sample is obtained by merging the first image region and the second image region.

[0173] The scheme for obtaining the defect sample can be represented by the following formula:

[0174] S = M1 O A + M2 O X formula (9).

[0175] In the above formula, S represents the defect sample, A represents the perturbed image obtained by the improved diffusion model, M1 represents the region within the mask, X represents the non-defect image, M2 represents the region outside the mask, O represents element-by-element multiplication, M1 O A represents the region within the mask in the perturbed image (i.e. the first image region), and M2 O X represents the region outside the mask in the non-defect image (i.e. the second image region).

[0176] To make clear the way of the merging processing in the embodiments of the present application, the following disclosure Figure 10 . Figure 10 is a schematic diagram of the merging processing, in which the upper half part shows that the region M1 (the region within the mask is M1 which is the white region in the corresponding image) within the mask is merged with the perturbed image A to obtain the first image region P1, the lower half part shows that the region M2 (the region outside the mask is M2 which is the white region in the corresponding image) outside the mask is merged with the defect-free image X to obtain the second image region P2, then the first image region P1 is merged with the second image region P2, and finally the defect sample S is obtained.

[0177] In another embodiment of the present application, after the defect sample is obtained by merging the first image region and the second image region, referring to Figure 11 , the following operations are further included.

[0178] Step S1100, determining a first shortest distance from a first pixel point in the first image region in the defect sample to the mask boundary, the first pixel point being any one of the pixel points in the first image region;

[0179] Step S1200, adjusting the transparency of the first pixel point according to the first shortest distance, wherein the shorter the first shortest distance is, the higher the transparency of the first pixel point is.

[0180] The first shortest distance from any pixel point in the mask to the mask boundary is represented by the following formula:

[0181]

[0182] In the formula, M represents the mask, p represents any pixel point in the mask, q represents the pixel point of the mask boundary, ||p-q|| represents the Euclidean distance from the pixel point p in the mask to the mask boundary, represents the first shortest distance from the pixel point p in the mask to the mask boundary.

[0183] Correspondingly, the transparency of the first pixel point can be adjusted by the following formula:

[0184]

[0185] wherein a(p) represents the transparency, and n represents the number of the first pixel points. When is 0, it means that the boundary of the mask, at this time a(p) is 1, which means that the first pixel point in this case is completely transparent; when is n, it means that the mask boundary is at the position of n pixels inward, at this time a(p) is 1, which means that the first pixel point in this case is completely opaque.

[0186] After the defect sample of the product to be inspected is obtained by merging the defect-free image, the perturbed image and the mask, the defect sample can have a problem that the fusion of the defect region and the background region in the defect sample is unnatural. Therefore, the embodiments of the present application can adjust the transparency of the pixels in the first image region by the above steps, realize the transition of the region in the mask to the outside, and thus improve the fusion effect of the defect region and the background region, so that the fusion of the defect region and the background region is more natural.

[0187] In another embodiment of the present application, after the defect sample is obtained by merging the first image region and the second image region, referring to Figure 12 , the following operations are further included:

[0188] Step S1300, determining a second shortest distance between a second pixel point in the second image region in the defect sample and the mask boundary;

[0189] Step S1400, adjusting the transparency of the second pixel point according to the second shortest distance, wherein the longer the second shortest distance is, the higher the transparency of the second pixel point is.

[0190] Through the embodiments of the present application, the transparency of the pixels in the second image region can be adjusted to realize the transition of the region outside the mask to the inside, and further improve the fusion effect of the defect region and the background region, so that the fusion of the defect region and the background region is more natural.

[0191] In order to clarify the process of generating the defect sample by the method provided by the embodiments of the present application, another embodiment of the present application provides Figure 13 . Figure 13 A process diagram for generating a defect sample by the defect sample generation method provided by each of the above embodiments of the present application is shown.

[0192] Referring to Figure 13 , in the process of generating the defect sample, first, a defect-free image including a product to be inspected is obtained; then, the diffusion model can be trained by the defect-free image, and the trained diffusion model can process the defect-free image in turn through the forward process and the reverse process, and the image obtained after processing has a high similarity with the defect-free image; the reverse process of the diffusion model is improved, and the perturbed image is generated by the improved diffusion model; the perturbed image is merged with the region in the mask to obtain a first image region, and the defect-free image is merged with the region outside the mask to obtain a second image region, and then the first image region and the second image region are merged to obtain the corresponding defect sample.

[0193] In addition, in another embodiment of the present application, after the defect sample is obtained, the defect sample can also be stored in a corresponding database, so as to facilitate subsequent detection of the product to be inspected based on the database.

[0194] In another embodiment of the present application, in order to facilitate the staff to view, after obtaining the defect sample, the defect sample can also be pushed to the display device for the display device to display the defect sample.

[0195] Corresponding to the above-mentioned embodiments, another embodiment of the present application provides a defect sample generation device, which can generate a defect sample based on a non-defective image of a product to be inspected.

[0196] In the embodiments of the present application, the defect sample generation device generally refers to a device capable of obtaining a non-defective image and generating a defect sample by running a diffusion model using the non-defective image. For example, the defect sample generation device can include, but is not limited to, a tablet computer, a handheld computer, a personal computer (PC), a mobile terminal, and the like.

[0197] The defect sample generation device comprises:

[0198] a controller configured to:

[0199] obtain a non-defective image of a product to be inspected;

[0200] add noise to the non-defective image according to at least two adding steps of a preset diffusion model to obtain a noise image corresponding to the non-defective image;

[0201] estimate, by a neural network model, estimated noise corresponding to the noise added by each of the at least two adding steps;

[0202] calculate, by a perturbation coefficient and the estimated noise corresponding to the noise added by each of the at least two adding steps, perturbation noise corresponding to the estimated noise;

[0203] perform, by the diffusion model and the perturbation noise, denoising processing on the noise image according to at least two denoising steps to obtain a perturbed image;

[0204] obtain a defect sample of the product to be inspected by merging the non-defective image, the perturbed image, and a mask.

[0205] By using the defect sample generation device provided in the embodiments of the present application, a defect sample can be generated based on a non-defective image, and there is no need to pre-collect a defect sample and label the defect type of the defect sample, thereby reducing the time consumed in the process of collecting and labeling the defect sample, and correspondingly improving the generation efficiency of the defect sample.

[0206] For the convenience of explanation, the above description has been made in connection with specific embodiments. However, the above description discussed in some embodiments is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Various modifications and variations can be derived from the above teachings by those skilled in the art. The selection and description of the above embodiments are made in order to better explain the content of the disclosure, so that those skilled in the art can better use the embodiments.

Claims

1. A defect sample generation method, characterized in that: include: Acquire defect-free images including the product to be inspected; adding noise to the defect-free image using a diffusion model to obtain a noise image corresponding to the defect-free image; estimating an estimated noise corresponding to the noise through a neural network model; Calculating disturbance noise corresponding to the estimated noise by using a disturbance coefficient and the estimated noise corresponding to the noise; Performing denoising processing on the noisy image using the diffusion model and the disturbance noise to obtain a disturbed image; The defect sample of the product to be inspected is obtained by merging the defect-free image, the disturbed image and the mask.

2. The method according to claim 1, characterized in that The step of performing denoising on the noisy image using the diffusion model and the disturbance noise to obtain a disturbed image includes: In a case where the denoising process includes at least two denoising steps, generating random numbers corresponding to the at least two denoising steps respectively by a preset random number generation algorithm; Comparing the random number corresponding to a target denoising step with a preset threshold, the target denoising step being any one of the denoising steps; If a comparison result between the random number corresponding to the target denoising step and a preset threshold indicates that the target denoising step needs to be disturbed, removing the disturbance noise corresponding to the target denoising step from the image to be denoised by using the diffusion model; If a comparison result between the random number corresponding to the target denoising step and a preset threshold indicates that the target denoising step does not require disturbance, removing the estimated noise corresponding to the target denoising step from the image to be denoised by using the diffusion model; Wherein, when the target denoising step is the first denoising step of the denoising process, the image to be denoised is the noisy image; when the target denoising step is another denoising step of the denoising process, the image to be denoised is the image obtained after executing the previous denoising step.

3. The method according to claim 1, characterized in that The calculating the disturbance noise corresponding to the estimated noise by using the disturbance coefficient and the estimated noise corresponding to the noise includes: In a case where the diffusion model adds noise to the defect-free image through at least two adding steps, determining, according to a preset weight coefficient allocation method, weight coefficients of estimated noise corresponding to the noise added in the at least two adding steps respectively; A product of the estimated noise, a weight coefficient corresponding to the estimated noise, and the disturbance coefficient is calculated to obtain disturbance noise corresponding to the estimated noise.

4. The method according to claim 1, wherein Before the merging process is performed on the defect-free image, the disturbed image and the mask, the method further includes: determining, according to an image length and an image width of the defect-free image, the Perlin noise and a preset number of superimposed layers of the Perlin noise, a two-dimensional fractal noise corresponding to the defect-free image; The mask is determined by performing binarization processing on the two-dimensional fractal noise.

5. The method according to claim 4, characterized in that After the mask is determined by performing binarization processing on the two-dimensional fractal noise, the method further includes: Filling the broken parts in the mask by dilating the mask; The redundant portion of the mask is removed by performing an erosion operation on the filled mask.

6. The method according to claim 1, characterized in that The step of obtaining a defect sample of the product to be inspected by merging the defect-free image, the disturbed image, and the mask comprises: Determining a first image region in the perturbed image corresponding to the mask by merging the region in the mask with the perturbed image; Determining a second image region in the defect-free image corresponding to the region outside the mask by merging the region outside the mask with the defect-free image; The defect sample is obtained by merging the first image area and the second image area.

7. The method according to claim 6, characterized in that After obtaining the defect sample by merging the first image area and the second image area, the method further includes: determining a first shortest distance between a first pixel point in the first image area in the defect sample and the mask boundary, where the first pixel point is any pixel point in the first image area; The transparency of the first pixel is adjusted according to the first shortest distance, wherein the shorter the first shortest distance is, the higher the transparency of the first pixel is.

8. The method according to claim 6, characterized in that After obtaining the defect sample by merging the first image area and the second image area, the method further includes: determining a second shortest distance between a second pixel point in the second image area of ​​the defect sample and the mask boundary, where the second pixel point is any pixel point in the second image area; The transparency of the second pixel is adjusted according to the second shortest distance, wherein the longer the second shortest distance is, the higher the transparency of the second pixel is.

9. The method according to any one of claims 1 to 8, characterized in that The obtaining of a defect-free image of the product to be inspected comprises: Acquiring an image obtained by photographing the product to be inspected that has passed the defect inspection; Locating a region of interest in the captured image using a preset positioning algorithm; The captured image is sheared using the located region of interest to obtain a defect-free image of the product to be inspected.

10. A defect sample generating device, characterized in that: include: controller, and is configured to: Acquire defect-free images including the product to be inspected; adding noise to the defect-free image using a diffusion model to obtain a noise image corresponding to the defect-free image; estimating an estimated noise corresponding to the noise through a neural network model; Calculating disturbance noise corresponding to the estimated noise by using a disturbance coefficient and the estimated noise corresponding to the noise; Performing denoising processing on the noisy image using the diffusion model and the disturbance noise to obtain a disturbed image; The defect sample of the product to be inspected is obtained by merging the defect-free image, the disturbed image and the mask.

Images