Mini-LED wafer surface defect data enhancement method and device based on deep learning

Abstract

The present invention discloses a method and device for enhancing Mini-LED wafer surface defect data based on deep learning, which relates to the field of image processing, including: constructing a diffusion model based on a network of each defect type, the diffusion model includes a noise addition process and a denoising process, training the network to obtain a trained network of each defect type; randomly sampling a Gaussian noise map, and inputting it into the trained network of each defect type to obtain the predicted noise corresponding to each defect type, and generating a generated defect image of each defect type, and constructing a generated defect image library; randomly selecting a generated defect image from the generated defect image library and performing a random transformation to obtain a transformed generated defect image, and synthesizing the transformed randomly selected generated defect image with a background image through image space processing to obtain a synthesized image. The present invention solves the problems of instability and poor effect of existing image data enhancement technologies.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for enhancing Mini-LED wafer surface defect data based on deep learning. Background Art

[0002] If there is an imbalance in the labeled data in the dataset, the data types with a small number of samples will easily overfit during the training process. When making a Mini-LED defect image dataset, the number of samples of certain defect types may be relatively scarce, resulting in an imbalanced labeling problem in the Mini-LED defect image dataset. Traditional solutions focus on simple transformations of existing images, usually including basic image processing techniques such as rotation, flipping, translation, and adding noise. The images generated by this method are relatively simple and cannot simulate complex defect situations. In recent years, generative models based on deep learning technology have made significant progress, but they are unstable and have poor results.

[0003] The diffusion model combined with UNet provides a new idea for solving the problem of unbalanced labeled data in the dataset. It has better stability and can generate more realistic images. Although the diffusion model combined with UNet has advantages in generating images, it is not effective in learning and generating subtle features. Summary of the invention

[0004] The purpose of this application is to propose a Mini-LED wafer surface defect data enhancement method and device based on deep learning to address the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for enhancing Mini-LED wafer surface defect data based on deep learning, comprising the following steps:

[0006] Obtain defect images of each defect type on the surface of the Mini-LED wafer and construct training data for each defect type, obtain defect-free background images and store them in the background image library;

[0007] Build a Diffusion model of the network, The network includes an encoding layer, a decoding layer, and an SPPF module located between the encoding layer and the decoding layer. The diffusion model includes a denoising process and a denoising process. In the denoising process, real noise is added to the defect image in the training data of each defect type to obtain the corresponding denoised image of each defect type. In the denoising process, the corresponding defect type is used. The network denoises the noise image corresponding to the defect image of each defect type to obtain the predicted noise; based on the real noise and the predicted noise, the diffusion model is used to denoise the noise image of each defect type. The network is trained to obtain the trained defect type network;

[0008] Randomly sample the Gaussian noise map, use the Gaussian noise map as the noise map of the current time step, and perform the following denoising process: Input the noise map of the current time step into the trained The network is constructed to obtain the predicted noise corresponding to each defect type. The noise map of the next time step is calculated based on the predicted noise corresponding to each defect type and the noise map of the current time step. The noise map of the next time step is used as the noise map of the current time step. The above denoising process is repeated until the generated defect image of each defect type is obtained, and the generated defect image library is constructed.

[0009] A generated defect image is randomly selected from a generated defect image library, and the randomly selected generated defect image is processed by a random transformation matrix to obtain a transformed randomly selected generated defect image. A background image is randomly selected from a background image library, and the transformed randomly selected generated defect image is synthesized with the background image through image space processing to obtain a synthesized image and determine the position of the defect in the synthesized image.

[0010] Preferably, the encoding layer includes five first convolution modules connected in sequence, the decoding layer includes four decoding units connected in sequence, each decoding unit includes a multi-scale feature fusion module and a second convolution module connected in sequence, the fifth first convolution module in the encoding layer is connected to the SPPF module, the SPPF module is connected to the multi-scale feature fusion module of the first decoding unit in the decoding layer, and the input features of each multi-scale feature fusion module are the output features of the SPPF module or the output features of the second convolution module of the previous decoding unit and the output features of the first four first convolution modules in the decoding layer.

[0011] Preferably, both the first convolution module and the second convolution module adopt a convolution structure based on a volume attention mechanism, and the calculation process is as follows:

[0012] ;

[0013] Among them, X represents the input features of the convolution structure based on the volume attention mechanism, represents the combined operation of the sequentially connected convolution operation with a kernel size of 1×1, batch normalization, and SiLU activation function. Represents a convolution operation with a convolution kernel size of 1×1; represents the time step embedding operation; , and Respectively represent the first intermediate feature map, the second intermediate feature map and the third intermediate feature map; is the output feature of the convolution structure based on the volume attention mechanism, Represents the function corresponding to the volume attention module;

[0014] The expression of the time step embedding operation is as follows:

[0015] ;

[0016] in, represents the tth time step, represents the index of the embedding dimension, is the embedding dimension, Represents the encoding result of the even index position of the t-th time step, Represents the encoding result of the odd index position of the tth time step;

[0017] The calculation process of the volume attention module is as follows:

[0018] The three dimensions of the input features of the volume attention module are average pooled separately, as shown in the following formula:

[0019] ;

[0020] Among them, V represents the input feature of the volume attention module, c, h and w represent the index of the number of channels, the index of the height and the index of the width respectively, C, H and W represent the number of channels, height and width of the input feature of the volume attention module respectively, represents the set of real numbers, , and Represent the first pooling feature, the second pooling feature and the third pooling feature respectively;

[0021] Secondly, the first pooling feature, the second pooling feature, and the third pooling feature are transposed and then concatenated, and then subjected to a convolution operation with a convolution kernel size of 1×1, batch normalization, and Activation function, get the first fusion feature and the second fusion feature, as shown in the following formula:

[0022] ;

[0023] in, and Respectively represent the first fusion feature and the second fusion feature, T represents the transposed matrix, represents the concatenation operation, BN represents batch normalization, express Activation function, ;

[0024] Next, the first fusion feature and the second fusion feature Perform segmentation operation and transform the first fusion feature First sub-feature and the second sub-feature , the second fusion feature Divided into the third sub-feature and the fourth sub-characteristic ;

[0025] Then, the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature are respectively subjected to the convolution operation with a convolution kernel size of 1×1 and The activation function obtains the first attention vector, the second attention vector, the third attention vector, and the fourth attention vector. The specific operations are as follows:

[0026] ;

[0027] in, , , and denote the first attention vector, the second attention vector, the third attention vector and the fourth attention vector respectively;

[0028] Finally, the first attention vector and the second attention vector Perform matrix multiplication to obtain the first product result , the third attention vector and the fourth attention vector Perform matrix multiplication to obtain the second product result , the first product result and the second product result After element-by-element addition The activation function gets the first weight , then get the first weight Then multiply the input features of the volume attention module element by element to obtain the output features of the volume attention module. The specific operations are as follows:

[0029] ;

[0030] Where Y represents the output feature of the volumetric attention module.

[0031] As a preference, the input feature of the multi-scale feature fusion module is expressed as ,in, represents the output features of the qth first convolutional module of the encoding layer, Represents the output features of the previous layer structure of the multi-scale feature fusion module, , q=1, 2, 3, 4, , They represent the maximum height, maximum width and maximum number of channels of the output features of the qth first convolutional module of the encoding layer, respectively. They respectively represent the maximum value of the height, width, and number of channels of the output features of the previous layer structure of the multi-scale feature fusion module;

[0032] First, the size of the input feature of the multi-scale feature fusion module is adjusted to be consistent with the size of the second convolution module of the next layer by upsampling or downsampling, and the first adjusted feature is obtained, which is expressed as , , ;

[0033] Then, the number of channels of the first adjustment feature is adjusted to be the same through a convolution operation with a convolution kernel size of 1×1 to obtain the second adjustment feature, as shown in the following formula:

[0034] ,

[0035] Among them, the second adjustment feature is expressed as ;

[0036] The number of channels of the second adjustment feature is adjusted to 1 through a convolution operation with a convolution kernel size of 1×1, and the third adjustment feature is obtained, as shown in the following formula:

[0037] ;

[0038] Among them, the third adjustment feature is expressed as ;

[0039] The third adjusted feature is globally averaged pooled to obtain the global pooled feature, as shown in the following formula:

[0040] ;

[0041] Among them, the global pooling feature is expressed as , w and h represent the index of width and height respectively;

[0042] Finally, the global pooling features are sequentially subjected to a convolution operation with a convolution kernel size of 1×1 and The combination of activation functions and a convolution operation with a convolution kernel size of 1×1 and The combination of activation functions gives the second weight and with the second adjustment feature Perform element-by-element multiplication to obtain weighted features, and then add the weighted features element-by-element to obtain the output features of the multi-scale feature fusion module, as shown in the following formula:

[0043] ;

[0044] The second weight is expressed as , the weighted feature is expressed as , Represents the output features of the multi-scale feature fusion module.

[0045] As a preference, the noise adding process in the diffusion model is as follows:

[0046] The noise image corresponding to the defect image of each defect type at the u-1th time step is superimposed with Gaussian noise once to obtain the noise image corresponding to the defect image of each defect type at the u-1th time step, as shown in the following formula:

[0047] ;

[0048] in, represents the noise image corresponding to the defect image of each defect type at the u-1th time step, represents the noise image corresponding to the defect image of each defect type at the u-th time step, For the The weight coefficient of the noise at time steps, Indicates The noise added conforms to the standard normal distribution;

[0049] Through the above formula, it can be deduced that the defect images in the training data of each defect type Through superposition Gaussian noise of the tth time step is obtained , as shown below:

[0050] ;

[0051] in, , , u=1,2,…,t, Indicates t The product of Z represents the real noise.

[0052] The denoising process in the diffusion model is as follows:

[0053] Randomly sampled Gaussian noise image , the Gaussian noise map As the t-th noise map input to each defect type trained Network, get the corresponding predicted noise for each defect type , then according to and Calculate the noise map at the t-1th time step , as shown below:

[0054] ;

[0055] in, Indicates t-1 The result of multiplication is For the The weight coefficient of the noise at each time step;

[0056] Repeat the above process until the generated defect image of each defect type is obtained. until.

[0057] Preferably, randomly selecting a generated defect image from a generated defect image library, processing the randomly selected generated defect image through a random transformation matrix to obtain a transformed randomly selected generated defect image, randomly selecting a background image from a background image library, synthesizing the transformed randomly selected generated defect image with the background image through image space processing to obtain a synthesized image and determining the position of the defect in the synthesized image, specifically comprising:

[0058] The mth generated defect image of the randomly selected cth defect type is transformed as follows:

[0059] ,

[0060] In the formula, It means generating the mth defect image from the randomly selected cth defect type. represents the mth generated defect image of the cth defect type after transformation, Represents the width and height of the mth defect image generated after the transformation of the cth defect type; is a random transformation matrix, which generates rotation parameters randomly , translation parameters( , ), scaling parameters( , ) to construct the random transformation matrix, which is defined as follows:

[0061] ;

[0062] Finally, a background image is randomly selected from the background image library and synthesized with each generated defect image of each defect type after transformation to obtain a synthesized image and determine the position of the defect in the synthesized image, as shown in the following formula:

[0063] ;

[0064] Where n represents the total number of randomly selected defect types, represents the total number of generated defect images of the randomly selected c-th defect type, represents a randomly selected background image from a background image library; represents a composite image, Indicates the coordinates of the upper left corner of the defect image in the background image, and is a randomly generated position offset parameter, and the position coordinates of the defect in the background image are .

[0065] In a second aspect, the present invention provides a Mini-LED wafer surface defect data enhancement device based on deep learning, comprising:

[0066] A data acquisition module is configured to acquire defect images of each defect type on the surface of the Mini-LED wafer and construct training data for each defect type, acquire defect-free background images and store them in a background image library;

[0067] The model building module is configured to build a Diffusion model of the network, The network includes an encoding layer, a decoding layer, and an SPPF module located between the encoding layer and the decoding layer. The diffusion model includes a denoising process and a denoising process. In the denoising process, real noise is added to the defect image in the training data of each defect type to obtain the corresponding denoised image of each defect type. In the denoising process, the corresponding defect type is used. The network denoises the noise image corresponding to the defect image of each defect type to obtain the predicted noise; based on the real noise and the predicted noise, the diffusion model is used to denoise the noise image of each defect type. The network is trained to obtain the trained defect type network;

[0068] The denoising module is configured to randomly sample the Gaussian noise map, use the Gaussian noise map as the noise map of the current time step, and perform the following denoising process: input the noise map of the current time step into the trained The network is constructed to obtain the predicted noise corresponding to each defect type. The noise map of the next time step is calculated based on the predicted noise corresponding to each defect type and the noise map of the current time step. The noise map of the next time step is used as the noise map of the current time step. The above denoising process is repeated until the generated defect image of each defect type is obtained, and the generated defect image library is constructed.

[0069] The synthesis module is configured to randomly select a generated defect image from a generated defect image library, process the randomly selected generated defect image through a random transformation matrix to obtain a transformed randomly selected generated defect image, randomly select a background image from a background image library, synthesize the transformed randomly selected generated defect image with the background image through image space processing to obtain a synthesized image and determine the position of the defect in the synthesized image.

[0070] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0071] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0072] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] (1) The Mini-LED wafer surface defect data enhancement method based on deep learning proposed in the present invention designs a multi-scale feature fusion module by combining the ideas of jump connection and multi-scale feature fusion. The multi-scale feature fusion module improves the model's expression ability of different scales by fusing features of different scales.

[0075] (2) The Mini-LED wafer surface defect data enhancement method based on deep learning proposed in the present invention designs a convolution structure based on volumetric attention by combining the volumetric attention mechanism. The convolution structure based on volumetric attention can capture spatial correlation more accurately.

[0076] (3) The Mini-LED wafer surface defect data enhancement method based on deep learning proposed in the present invention designs a VMSA-UNet network by combining a convolutional structure based on volumetric attention and a multi-scale feature fusion module. Combining the VMSA-UNet network with a diffusion model can more effectively remove noise according to different scales and spatial dimensions and generate images with richer details. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 A schematic diagram of a process of a Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0079] Figure 2 A flowchart of a method for enhancing Mini-LED wafer surface defect data based on deep learning according to an embodiment of the present application;

[0080] Figure 3 The Mini-LED wafer surface defect data enhancement method based on deep learning in the embodiment of the present application is Schematic diagram of the network;

[0081] Figure 4 A schematic diagram of the noise adding process of the diffusion model of the Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0082] Figure 5 A schematic diagram of a convolution structure based on a volumetric attention mechanism for a Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0083] Figure 6 A schematic diagram of a volume attention module of a Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0084] Figure 7 A schematic diagram of a multi-scale feature fusion module of a Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0085] Figure 8A schematic diagram of a denoising process of a diffusion model of a Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0086] Fig. 9 A schematic diagram of randomly generated synthetic images of a Mini-LED wafer surface defect data enhancement method based on deep learning according to an embodiment of the present application;

[0087] Fig.10 Schematic diagram of a Mini-LED wafer surface defect data enhancement device based on deep learning according to an embodiment of the present application;

[0088] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] Figure 1 A method for enhancing Mini-LED wafer surface defect data based on deep learning provided by an embodiment of the present application is shown, comprising the following steps:

[0091] S1, obtain defect images of each defect type on the surface of the Mini-LED wafer and construct training data for each defect type, obtain defect-free background images and store them in the background image library.

[0092] Specifically, refer to Figure 2 First, image data is collected from the Mini-LED production line. In one embodiment, the image collection process uses a CCD camera, which has resolution, equipped with a telecentric lens and a LED coaxial light source with adjustable brightness.

[0093] Secondly, a defect-free image that meets the process requirements is selected from the collected image data, and the image size is cropped to 640×640 to obtain a background image, and the background image is added to the background image library.

[0094] Then, the defect types on the surface of the Mini-LED wafer include foreign matter, residual glue, incompleteness / discontinuity, lumps, dirt and scratches, among which the data of the three defect types of scratches, foreign matter and incompleteness / discontinuity are relatively small. Therefore, the embodiment of the present application selects the image data of the three defect types of scratches, foreign matter and incompleteness / discontinuity from the collected image data, segments the defective part images from the image data of these three defect types, and pre-processes the defective part images, specifically including removing useless defective part images, adjusting image contrast, adjusting image clarity and other operations to obtain the pre-processed defective part images.

[0095] Finally, the preprocessed defective part image is scaled to a size of 640×640 through image processing technology, specifically including: if the size of the preprocessed defective part image is less than 640×640, the image is enlarged by bilinear interpolation; if the size of the preprocessed defective part image is larger than 640×640, the image is reduced by downsampling. Through processing, defect images of each defect type with consistent size are obtained and training data of the corresponding defect type is formed, so as to facilitate batch processing of the subsequent diffusion model.

[0096] S2, constructs a Diffusion model of the network, The network includes an encoding layer, a decoding layer, and an SPPF module located between the encoding layer and the decoding layer. The diffusion model includes a denoising process and a denoising process. In the denoising process, real noise is added to the defect image in the training data of each defect type to obtain the corresponding denoised image of each defect type. In the denoising process, the corresponding defect type is used. The network denoises the noise image corresponding to the defect image of each defect type to obtain the predicted noise; based on the real noise and the predicted noise, the diffusion model is used to denoise the noise image of each defect type. The network is trained to obtain the trained defect type network.

[0097] In a specific embodiment, the encoding layer includes five first convolution modules connected in sequence, the decoding layer includes four decoding units connected in sequence, each decoding unit includes a multi-scale feature fusion module and a second convolution module connected in sequence, the fifth first convolution module in the encoding layer is connected to the SPPF module, the SPPF module is connected to the multi-scale feature fusion module of the first decoding unit in the decoding layer, and the input features of each multi-scale feature fusion module are the output features of the SPPF module or the output features of the second convolution module of the previous decoding unit and the output features of the first four first convolution modules in the decoding layer.

[0098] In a specific embodiment, both the first convolution module and the second convolution module adopt a convolution structure based on a volume attention mechanism, and the calculation process is as follows:

[0099] ;

[0100] Among them, X represents the input features of the convolution structure based on the volume attention mechanism, represents the combined operation of the sequentially connected convolution operation with a kernel size of 1×1, batch normalization, and SiLU activation function. Indicates a convolution operation with a convolution kernel size of 1×1; represents the time step embedding operation; , and Respectively represent the first intermediate feature map, the second intermediate feature map and the third intermediate feature map; is the output feature of the convolution structure based on the volume attention mechanism, Represents the function corresponding to the volume attention module;

[0101] The expression of the time step embedding operation is as follows:

[0102] ;

[0103] in, represents the tth time step, represents the index of the embedding dimension, is the embedding dimension, Represents the encoding result of the even index position of the t-th time step, Represents the encoding result of the odd index position of the tth time step;

[0104] The calculation process of the volume attention module is as follows:

[0105] The three dimensions of the input features of the volume attention module are average pooled separately, as shown in the following formula:

[0106] ;

[0107] Among them, V represents the input feature of the volume attention module, c, h and w represent the index of the number of channels, the index of the height and the index of the width respectively, C, H and W represent the number of channels, height and width of the input feature of the volume attention module respectively, represents the set of real numbers, , and Represent the first pooling feature, the second pooling feature and the third pooling feature respectively;

[0108] Secondly, the first pooling feature, the second pooling feature, and the third pooling feature are transposed and then concatenated, and then subjected to a convolution operation with a convolution kernel size of 1×1, batch normalization, and Activation function, get the first fusion feature and the second fusion feature, as shown in the following formula:

[0109] ;

[0110] in, and Respectively represent the first fusion feature and the second fusion feature, T represents the transposed matrix, represents the concatenation operation, BN represents batch normalization, express Activation function, ;

[0111] Next, the first fusion feature and the second fusion feature Perform segmentation operation and transform the first fusion feature First sub-feature and the second sub-feature , the second fusion feature Divided into the third sub-feature and the fourth sub-characteristic ;

[0112] Then, the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature are respectively subjected to the convolution operation with a convolution kernel size of 1×1 and The activation function obtains the first attention vector, the second attention vector, the third attention vector, and the fourth attention vector. The specific operations are as follows:

[0113] ;

[0114] in, , , and denote the first attention vector, the second attention vector, the third attention vector and the fourth attention vector respectively;

[0115] Finally, the first attention vector and the second attention vector Perform matrix multiplication to obtain the first product result , the third attention vector and the fourth attention vector Perform matrix multiplication to obtain the second product result , the first product result and the second product result After element-by-element addition The activation function gets the first weight , then get the first weight Then multiply the input features of the volume attention module element by element to obtain the output features of the volume attention module. The specific operations are as follows:

[0116] ;

[0117] Where Y represents the output feature of the volumetric attention module.

[0118] In a specific embodiment, the input feature of the multi-scale feature fusion module is expressed as ,in, represents the output features of the qth first convolutional module of the encoding layer, Represents the output features of the previous layer structure of the multi-scale feature fusion module, , q=1, 2, 3, 4, , They represent the maximum height, maximum width and maximum number of channels of the output features of the qth first convolutional module of the encoding layer, respectively. They respectively represent the maximum value of the height, width, and number of channels of the output features of the previous layer structure of the multi-scale feature fusion module;

[0119] First, the size of the input feature of the multi-scale feature fusion module is adjusted to be consistent with the size of the second convolution module of the next layer by upsampling or downsampling, and the first adjusted feature is obtained, which is expressed as , , ;

[0120] Then, the number of channels of the first adjustment feature is adjusted to be the same through a convolution operation with a convolution kernel size of 1×1 to obtain the second adjustment feature, as shown in the following formula:

[0121] ,

[0122] Among them, the second adjustment feature is expressed as ;

[0123] The number of channels of the second adjustment feature is adjusted to 1 through a convolution operation with a convolution kernel size of 1×1, and the third adjustment feature is obtained, as shown in the following formula:

[0124] ;

[0125] Among them, the third adjustment feature is expressed as ;

[0126] The third adjusted feature is globally averaged pooled to obtain the global pooled feature, as shown in the following formula:

[0127] ;

[0128] Among them, the global pooling feature is expressed as , w and h represent the index of width and height respectively;

[0129] Finally, the global pooling features are sequentially subjected to a convolution operation with a convolution kernel size of 1×1 and The combination of activation functions and a convolution operation with a convolution kernel size of 1×1 and The combination of activation functions gives the second weight and with the second adjustment feature Perform element-by-element multiplication to obtain weighted features, and then add the weighted features element-by-element to obtain the output features of the multi-scale feature fusion module, as shown in the following formula:

[0130] ;

[0131] The second weight is expressed as , the weighted feature is expressed as , Represents the output features of the multi-scale feature fusion module.

[0132] In a specific embodiment, the noise adding process in the diffusion model is as follows:

[0133] The noise image corresponding to the defect image of each defect type at the u-1th time step is superimposed with Gaussian noise once to obtain the noise image corresponding to the defect image of each defect type at the u-1th time step, as shown in the following formula:

[0134] ;

[0135] in, represents the noise image corresponding to the defect image of each defect type at the u-1th time step, represents the noise image corresponding to the defect image of each defect type at the u-th time step, For the The weight coefficient of the noise at time steps, Indicates The noise added conforms to the standard normal distribution;

[0136] Through the above formula, it can be deduced that the defect images in the training data of each defect type Through superposition Gaussian noise of the tth time step is obtained , as shown below:

[0137] ;

[0138] in, , , u=1,2,…,t, Indicates t The product of , Z represents the real noise.

[0139] Specifically, refer to Figure 3 , design a network. The network consists of an encoding layer and a decoding layer, which are connected by Module connection.

[0140] In the coding layer, the noise map corresponding to the defect image of each defect type input First, the number of channels is changed to 64 through the first convolution module, and then the size of the feature map is halved through downsampling; then, the number of channels is changed to 128 through the second convolution module, and then the size of the feature map is halved through downsampling; then, the number of channels is changed to 256 through the third convolution module, and then the size of the feature map is halved through downsampling; then, the number of channels is changed to 512 through the fourth convolution module, and then the size of the feature map is halved through downsampling; finally, the number of channels is changed to 1024 through the fifth convolution module; then the output features of the fifth convolution module are passed to the SPPF module;

[0141] In the decoding layer, the output features of the first four first convolution modules of the encoding layer are first input together with the output features of the SPPF module into the multi-scale feature fusion (MSFF) module in the first decoding unit, and then input into the first second convolution module after fusion; then, the output features of the first four first convolution modules of the encoding layer are input together with the output features of the first second convolution module into the MSFF module in the second decoding unit, and then input into the second second convolution module after fusion; similarly, the output features of the first four first convolution modules of the encoding layer are input together with the output features of the second second convolution module into the MSFF module in the third decoding unit, and then input into the output features of the third second convolution module after fusion; then, the output features of the first four first convolution modules of the encoding layer are input together with the output features of the third second convolution module into the MSFF module in the fourth decoding unit, and then input into the fourth second convolution module after fusion; finally, the fourth second convolution module outputs the predicted noise .

[0142] Furthermore, the The network participates in building the diffusion model, which includes the noise addition process and the noise removal process. Figure 4 ,exist During the training process of the network, noise that conforms to the standard normal distribution is added to the defect image of each defect type through the noise addition process until the noise image corresponding to each defect type corresponding to the added real noise is obtained. In the denoising process, the noise image corresponding to the defect image of each defect type is input into the defect image of each defect type. The network generates the predicted noise in the denoising process. and prediction noise Build a The loss function in the network training process uses the average error to minimize the loss function. The back propagation method and gradient descent method are used to calculate the loss of each defect type. The parameters of the network are adjusted, and the above iterative process is repeated until the preset number of iterative training times is reached, and the trained defect types of each defect are obtained. network.

[0143] refer to Figure 5 and Figure 6 , the first convolution module and the second convolution module both use the same convolution structure based on volumetric attention mechanism (VA Conv), and the convolution structure based on volumetric attention mechanism uses volumetric attention mechanism ( ) and time step embedding operations, and the time step embedding operation uses sine and cosine coding. The multi-scale feature fusion module is used to perform multi-scale fusion on the output features of the first four first convolution modules and the output features of the previous layer of the multi-scale feature fusion module. Figure 7 ,The multi-scale feature fusion module adopts the idea of ​​skip connection and ,multi-scale feature fusion to fuse features of different scales.

[0144] S3 randomly samples the Gaussian noise map, uses the Gaussian noise map as the noise map of the current time step, and performs the following denoising process: input the noise map of the current time step into the trained noise map of each defect type The network is constructed to obtain the predicted noise corresponding to each defect type. The noise map of the next time step is calculated based on the predicted noise corresponding to each defect type and the noise map of the current time step. The noise map of the next time step is used as the noise map of the current time step. The above denoising process is repeated until the generated defect image of each defect type is obtained, and a generated defect image library is constructed.

[0145] In a specific embodiment, the denoising process in the diffusion model is as follows:

[0146] Randomly sampled Gaussian noise image , the Gaussian noise map As the t-th noise map input to each defect type trained Network, get the corresponding predicted noise for each defect type , then according to and Calculate the noise map at the t-1th time step , as shown below:

[0147] ;

[0148] in, Indicates t-1 The result of multiplication is For the The weight coefficient of the noise at each time step;

[0149] Repeat the above process until the generated defect image of each defect type is obtained. until.

[0150] Specifically, refer to Figure 8 , during inference, randomly sampled Gaussian noise maps , the Gaussian noise map Input to each defect type trained Network, get the corresponding predicted noise for each defect type , then according to and Calculate , and repeat the above process until the generated defect image of each defect type is obtained. So far, the defect images for each defect type are generated The size of is consistent with the size of the defect image of the training data, which is 640 × 640. The generated defect images of all defect types are stored in the generated defect image library.

[0151] S4, randomly selects a generated defect image from the generated defect image library, processes the randomly selected generated defect image through a random transformation matrix to obtain a transformed randomly selected generated defect image, randomly selects a background image from the background image library, synthesizes the transformed randomly selected generated defect image with the background image through image space processing to obtain a synthesized image and determine the position of the defect in the synthesized image.

[0152] In a specific embodiment, step S4 specifically includes:

[0153] The mth generated defect image of the randomly selected cth defect type is transformed as follows:

[0154] ,

[0155] In the formula, It means generating the mth defect image from the randomly selected cth defect type. represents the mth generated defect image of the cth defect type after transformation, Represents the width and height of the mth defect image generated after the transformation of the cth defect type; is a random transformation matrix, which generates rotation parameters randomly , translation parameters( , ), scaling parameters( , ) to construct the random transformation matrix, which is defined as follows:

[0156] ;

[0157] Finally, a background image is randomly selected from the background image library and synthesized with each generated defect image of each defect type after transformation to obtain a synthesized image and determine the position of the defect in the synthesized image, as shown in the following formula:

[0158] ;

[0159] Where n represents the total number of randomly selected defect types, represents the total number of generated defect images of the randomly selected c-th defect type, represents a randomly selected background image from a background image library; represents a composite image, Indicates the coordinates of the upper left corner of the defect image in the background image, and is a randomly generated position offset parameter, and the position coordinates of the defect in the background image are .

[0160] Specifically, refer to Fig. 9 , randomly select the defect type and quantity of the generated defect image from the generated defect image library, and synthesize the randomly selected different generated defect images of different defect types into the same background image. First, obtain the size of the transformed generated defect image and ;

[0161] Second, randomly generate coordinates , as the position coordinates of the upper left corner of the defect image in the background image during the image synthesis process;

[0162] Finally, the size of the acquired defect image and randomly generated coordinates The position coordinates of the defect in the synthetic image can be obtained as , and then further annotate the synthetic image into YOLO format.

[0163] Further references Fig.10 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a Mini-LED wafer surface defect data enhancement device based on deep learning. The device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0164] The embodiment of the present application provides a Mini-LED wafer surface defect data enhancement device based on deep learning, comprising:

[0165] The data acquisition module 1 is configured to acquire defect images of each defect type on the surface of the Mini-LED wafer and construct training data for each defect type, acquire defect-free background images and store them in a background image library;

[0166] Model building module 2 is configured to build a Diffusion model of the network, The network includes an encoding layer, a decoding layer, and an SPPF module located between the encoding layer and the decoding layer. The diffusion model includes a denoising process and a denoising process. In the denoising process, real noise is added to the defect image in the training data of each defect type to obtain the corresponding denoised image of each defect type. In the denoising process, the corresponding defect type is used. The network denoises the noise image corresponding to the defect image of each defect type to obtain the predicted noise; based on the real noise and the predicted noise, the diffusion model is used to denoise the noise image of each defect type. The network is trained to obtain the trained defect type network;

[0167] Denoising module 3 is configured to randomly sample the Gaussian noise map, use the Gaussian noise map as the noise map of the current time step, and perform the following denoising process: input the noise map of the current time step into the trained The network is constructed to obtain the predicted noise corresponding to each defect type. The noise map of the next time step is calculated based on the predicted noise corresponding to each defect type and the noise map of the current time step. The noise map of the next time step is used as the noise map of the current time step. The above denoising process is repeated until the generated defect image of each defect type is obtained, and the generated defect image library is constructed.

[0168] The synthesis module 4 is configured to randomly select a generated defect image from the generated defect image library, process the randomly selected generated defect image through a random transformation matrix to obtain a transformed randomly selected generated defect image, randomly select a background image from the background image library, and synthesize the transformed randomly selected generated defect image with the background image through image space processing to obtain a synthesized image and determine the position of the defect in the synthesized image.

[0169] Fig.11 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Fig.11 As shown, the electronic device of this embodiment includes: a processor 1101 and a memory 1102; wherein the memory 1102 is used to store computer-executable instructions; the processor 1101 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0170] Optionally, the memory 1102 may be independent or integrated with the processor 1101 .

[0171] When the memory 1102 is independently provided, the electronic device further includes a bus 1103 for connecting the memory 1102 and the processor 1101 .

[0172] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 1101 executes the computer execution instructions, the above method is implemented.

[0173] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 1101, the above method is implemented.

[0174] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0175] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0176] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0177] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 1101 to perform some steps of the methods of various embodiments of the present application.

[0178] It should be understood that the processor 1101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 1101 may be any conventional processor 1101, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 1101, or may be executed by a combination of hardware and software modules in the processor 1101.

[0179] The memory 1102 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0180] The bus 1103 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1103 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 1103 in the drawings of the present application is not limited to only one bus 1103 or one type of bus 1103.

[0181] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0182] An exemplary storage medium is coupled to the processor 1101, so that the processor 1101 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 1101. The processor 1101 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 1101 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0183] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Mini-LED wafer surface defect data enhancement method based on deep learning, characterized in that: The following steps are involved: Obtain defect images of each defect type on the surface of the Mini-LED wafer and construct training data for each defect type, obtain defect-free background images and store them in the background image library; A diffusion model of a VMSA-UNet network based on each defect type is constructed. The VMSA-UNet network includes a coding layer, a decoding layer, and an SPPF module located between the coding layer and the decoding layer. The coding layer includes five first convolution modules connected in sequence. The decoding layer includes four decoding units connected in sequence. Each decoding unit includes a multi-scale feature fusion module and a second convolution module connected in sequence. The fifth first convolution module in the coding layer is connected to the SPPF module. The SPPF module is connected to the multi-scale feature fusion module of the first decoding unit in the decoding layer. The input features of each multi-scale feature fusion module are the output features of the SPPF module or the output features of the second convolution module of the previous decoding unit and the output features of the first four first convolution modules in the decoding layer. The first convolution module and the second convolution module both adopt a convolution structure based on a volume attention mechanism, and the calculation process is as follows: Wherein, X represents the input feature of the convolution structure based on the volumetric attention mechanism, CBS(·) represents the combined operation of the convolution operation with a convolution kernel size of 1×1, batch normalization processing and SiLU activation function connected in sequence, Conv represents the convolution operation with a convolution kernel size of 1×1; t-Embedding(t) represents the time step embedding operation; M1, M2 and M3 represent the first intermediate feature map, the second intermediate feature map and the third intermediate feature map respectively; M4 is the output feature of the convolution structure based on the volumetric attention mechanism, and VA represents the function corresponding to the volumetric attention module; The expression of the time step embedding operation is as follows: Wherein, t represents the t-th time step, i represents the index of the embedding dimension, d is the embedding dimension, E(t,2i) represents the encoding result of the even index position of the t-th time step, and E(t,2i+1) represents the encoding result of the odd index position of the t-th time step; the diffusion model includes a denoising process and a denoising process, in which real noise is added to the defect image in the training data of each defect type to obtain a denoised image corresponding to the defect image of each defect type, and in the denoising process, a VMSA-UNet network corresponding to the defect type is used to denoise the denoised image corresponding to the defect image of each defect type to obtain predicted noise; the VMSA-UNet network of each defect type in the diffusion model is trained based on the real noise and the predicted noise to obtain a trained VMSA-UNet network of each defect type; Randomly sample a Gaussian noise image, use the Gaussian noise image as the noise image of the current time step, and perform the following denoising process: input the noise image of the current time step into the trained VMSA-UNet network of each defect type to obtain the predicted noise corresponding to each defect type, calculate the noise image of the next time step according to the predicted noise corresponding to each defect type and the noise image of the current time step, use the noise image of the next time step as the noise image of the current time step, repeat the above denoising process until the generated defect image of each defect type is obtained, and construct a generated defect image library; A generated defect image is randomly selected from the generated defect image library, and the randomly selected generated defect image is processed by a random transformation matrix to obtain a transformed randomly selected generated defect image. A background image is randomly selected from the background image library, and the transformed randomly selected generated defect image is synthesized with the background image by image space processing to obtain a synthesized image and determine the position of the defect in the synthesized image.

2. The method for enhancing Mini-LED wafer surface defect data based on deep learning according to claim 1, characterized in that: The calculation process of the volume attention module is as follows: The three dimensions of the input features of the volume attention module are average pooled respectively, as shown in the following formula: Wherein, V represents the input feature of the volume attention module, c, h and w represent the index of the number of channels, the index of the height and the index of the width respectively, C, H and W represent the number of channels, the height and the width of the input feature of the volume attention module respectively, represents the set of real numbers, Z x , Z y and Z z Respectively represent the first pooling feature, the second pooling feature and the third pooling feature; Secondly, the first pooling feature, the second pooling feature, and the third pooling feature are transposed and then concatenated, and the first fusion feature and the second fusion feature are obtained by performing a convolution operation with a convolution kernel size of 1×1, batch normalization, and ReLU activation function, respectively, as shown in the following formula: Wherein, f1 and f2 represent the first fusion feature and the second fusion feature respectively, T represents the transposed matrix, [·] represents the concatenation operation, BN represents the batch normalization process, δ represents the ReLU activation function, r1=4, r2=4; Next, the first fusion feature f1 and the second fusion feature f2 are segmented to divide the first fusion feature f1 into the first sub-feature and the second sub-feature Divide the second fusion feature f2 into the third sub-feature and the fourth sub-characteristic Then, the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature are respectively subjected to a convolution operation with a convolution kernel size of 1×1 and a ReLU activation function to obtain the first attention vector, the second attention vector, the third attention vector and the fourth attention vector. The specific operations are as follows: Among them, g x , g y and denote the first attention vector, the second attention vector, the third attention vector and the fourth attention vector respectively; Finally, the first attention vector g x and the second attention vector Perform matrix multiplication to obtain the first product result g1 and the third attention vector g y and the fourth attention vector Perform matrix multiplication to obtain the second product result g2, add the first product result g1 and the second product result g2 element by element, and then obtain the first weight g through the Simgoid activation function, and then multiply the first weight g by the input feature of the volume attention module element by element to obtain the output feature of the volume attention module. The specific operation is as follows: Wherein, Y represents the output feature of the volume attention module.

3. The method for enhancing Mini-LED wafer surface defect data based on deep learning according to claim 1, characterized in that: The input feature of the multi-scale feature fusion module is expressed as in, represents the output feature of the qth first convolutional module of the encoding layer, represents the output feature of the previous layer structure of the multi-scale feature fusion module, H q , W q and C q respectively represent the maximum value of the height, width and number of channels of the output feature of the qth first convolutional module of the encoding layer, H p , W p and C p Respectively represent the maximum value of the height, the maximum value of the width and the maximum value of the number of channels of the output feature of the upper layer structure of the multi-scale feature fusion module; First, the size of the input feature of the multi-scale feature fusion module is adjusted to be consistent with the size of the second convolution module of the next layer by upsampling or downsampling, and the first adjusted feature is obtained, which is expressed as Then, the number of channels of the first adjustment feature is adjusted to be the same through a convolution operation with a convolution kernel size of 1×1, and a second adjustment feature is obtained, as shown in the following formula: The second adjustment feature is expressed as The number of channels of the second adjustment feature is adjusted to 1 through a convolution operation with a convolution kernel size of 1×1, and a third adjustment feature is obtained, as shown in the following formula: Wherein, the third adjustment feature is expressed as The third adjusted feature is globally averaged pooled to obtain a global pooled feature, as shown in the following formula: Among them, the global pooling feature is expressed as w and h represent the index of width and height respectively; Finally, the global pooling feature is sequentially subjected to a convolution operation with a convolution kernel size of 1×1 and a combination of a ReLU activation function and a convolution operation with a convolution kernel size of 1×1 and a Sigmoid activation function to obtain a second weight S, which is then combined with the second adjustment feature X″ in Perform element-by-element multiplication to obtain weighted features, and then add the weighted features element-by-element to obtain the output features of the multi-scale feature fusion module, as shown in the following formula: Among them, the second weight is expressed as The weighted feature is expressed as Represents the output features of the multi-scale feature fusion module.

4. The method for enhancing Mini-LED wafer surface defect data based on deep learning according to claim 1, characterized in that: The noise adding process in the diffusion model is as follows: The noise image corresponding to the defect image of each defect type at the u-1th time step is superimposed with Gaussian noise once to obtain the noise image corresponding to the defect image of each defect type at the u-1th time step, as shown in the following formula: Among them, X u-1 represents the noise image corresponding to the defect image of each defect type at the u-1th time step, X u represents the noise image corresponding to the defect image of each defect type at the u-th time step, β u is the weight coefficient of the noise at the u-th time step, Z u represents the noise added for the uth time that conforms to the standard normal distribution; It can be deduced from the above formula that the defect image X0 in the training data of each defect type is superimposed with Gaussian noise t times to obtain the noise image X at the tth time step. t , as shown below: Among them, α u =1-β u , represents t α u The product of Z represents the real noise. The denoising process in the diffusion model is as follows: Randomly sampled Gaussian noise image The Gaussian noise map As the tth noise map, it is input into the trained VMSA-UNet network of each defect type to obtain the corresponding predicted noise of each defect type. Then according to and Calculate the noise map at the t-1th time step As shown below: in, represents t-1 α u The result of multiplication, β t is the weight coefficient of the noise at the tth time step; Repeat the above process until the generated defect image of each defect type is obtained. until.

5. The method for enhancing Mini-LED wafer surface defect data based on deep learning according to claim 1, characterized in that: Randomly selecting a generated defect image from the generated defect image library, processing the randomly selected generated defect image through a random transformation matrix to obtain a transformed randomly selected generated defect image, randomly selecting a background image from the background image library, synthesizing the transformed randomly selected generated defect image with the background image through image space processing to obtain a synthesized image and determining the position of the defect in the synthesized image, specifically including: The mth generated defect image of the randomly selected cth defect type is transformed as follows: In the formula, It means generating the mth defect image from the randomly selected cth defect type. represents the mth generated defect image of the cth defect type after transformation, w ′ and h ′ represents the width and height of the mth defect image generated by the cth defect type after transformation; T k is a random transformation matrix, which generates the rotation parameter θ by random k , translation parameter (t x ,t y ), scaling parameters (S x ,S v ) to construct the random transformation matrix, which is defined as follows: Finally, a background image is randomly selected from the background image library and synthesized with each generated defect image of each defect type after transformation to obtain a synthesized image and determine the position of the defect in the synthesized image, as shown in the following formula: Where n represents the total number of randomly selected defect types, N c represents the total number of generated defect images of the randomly selected c-th defect type, I b represents a randomly selected background image in the background image library; I0 represents a synthetic image, (x′, y′) represents the coordinates of the upper left corner of the defect image in the background image, x l and l is a randomly generated position bias parameter, and the position coordinates of the defect in the background image are (x′, y′, w, h).

6. A Mini-LED wafer surface defect data enhancement device based on deep learning, characterized in that: include: A data acquisition module is configured to acquire defect images of each defect type on the surface of the Mini-LED wafer and construct training data for each defect type, acquire defect-free background images and store them in a background image library; A model building module is configured to build a diffusion model based on a VMSA-UNet network for each defect type, wherein the VMSA-UNet network includes a coding layer, a decoding layer, and an SPPF module located between the coding layer and the decoding layer, wherein the coding layer includes five first convolution modules connected in sequence, the decoding layer includes four decoding units connected in sequence, each decoding unit includes a multi-scale feature fusion module and a second convolution module connected in sequence, the fifth first convolution module in the coding layer is connected to the SPPF module, and the SPPF module is connected to the multi-scale feature fusion module of the first decoding unit in the decoding layer, and the input features of each multi-scale feature fusion module are the output features of the SPPF module or the output features of the second convolution module of the previous decoding unit and the output features of the first four first convolution modules in the decoding layer; the first convolution module and the second convolution module both adopt a convolution structure based on a volume attention mechanism, and the calculation process is as follows: Wherein, X represents the input feature of the convolution structure based on the volumetric attention mechanism, CBS(·) represents the combined operation of the convolution operation with a convolution kernel size of 1×1, batch normalization processing and SiLU activation function connected in sequence, Conv represents the convolution operation with a convolution kernel size of 1×1; t-Embedding(t) represents the time step embedding operation; M1, M2 and M3 represent the first intermediate feature map, the second intermediate feature map and the third intermediate feature map respectively; M4 is the output feature of the convolution structure based on the volumetric attention mechanism, and VA represents the function corresponding to the volumetric attention module; The expression of the time step embedding operation is as follows: Wherein, t represents the t-th time step, i represents the index of the embedding dimension, d is the embedding dimension, E(t,2i) represents the encoding result of the even index position of the t-th time step, and E(t,2i+1) represents the encoding result of the odd index position of the t-th time step; the diffusion model includes a denoising process and a denoising process, in which real noise is added to the defect image in the training data of each defect type to obtain a denoised image corresponding to the defect image of each defect type, and in the denoising process, a VMSA-UNet network corresponding to the defect type is used to denoise the denoised image corresponding to the defect image of each defect type to obtain predicted noise; the VMSA-UNet network of each defect type in the diffusion model is trained based on the real noise and the predicted noise to obtain a trained VMSA-UNet network of each defect type; A denoising module is configured to randomly sample a Gaussian noise image, use the Gaussian noise image as the noise image of the current time step, and perform the following denoising process: input the noise image of the current time step into the trained VMSA-UNet network of each defect type to obtain the predicted noise corresponding to each defect type, calculate the noise image of the next time step according to the predicted noise corresponding to each defect type and the noise image of the current time step, use the noise image of the next time step as the noise image of the current time step, repeat the above denoising process until a generated defect image of each defect type is obtained, and a generated defect image library is constructed; The synthesis module is configured to randomly select a generated defect image from the generated defect image library, process the randomly selected generated defect image through a random transformation matrix to obtain a transformed randomly selected generated defect image, randomly select a background image from the background image library, synthesize the transformed randomly selected generated defect image with the background image through image space processing to obtain a synthesized image and determine the position of the defect in the synthesized image.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images