Nozzle bubble image generation method based on autoencoder and generative adversarial network

Abstract

The present application relates to a nozzle bubble image generation method based on an autoencoder and a generative adversarial network, and belongs to the field of semiconductor manufacturing and data enhancement. The method comprises the following steps: capturing nozzle bubble images in different positions and backgrounds as original images; encoding the position information of the nozzle bubble in the image to obtain a plurality of position codes; expanding the original image to obtain an expanded image; constructing and training an autoencoder based on the original image, the position code and the expanded image; constructing and training a generative adversarial network based on the original image, the expanded image and the encoder output; and passing the original image or the expanded image together with the position code through the autoencoder and the generative adversarial network to obtain a generated image. The present application can generate high-quality bubble images in a specified position, effectively solve the problem of insufficient samples in the industrial production detection environment, greatly expand the diversity of samples, and provide sufficient data for the development of bubble detection algorithms.

Description

Technical Field

[0001] This invention relates to a method for generating nozzle bubble images based on autoencoders and generative adversarial networks, belonging to the fields of semiconductor manufacturing and data enhancement. Background Technology

[0002] In the semiconductor industry, wafer manufacturing processes are extremely complex, and photoresist coating is one of the key processes. This involves spraying photoresist onto the center of the wafer using a nozzle while simultaneously rotating the wafer, utilizing centrifugal force to spread the photoresist evenly across the entire wafer. Due to factors such as the properties of the photoresist and ambient temperature, air bubbles may appear in the photoresist within the nozzle. This causes uneven photoresist film formation and other coating defects, affecting subsequent processes and leading to a decrease in wafer yield and quality. Therefore, it is necessary to accurately detect air bubbles in the nozzle. However, traditional methods struggle to handle flocculent bubbles and complex wavy backgrounds. In contrast, machine learning, especially deep learning, has developed rapidly in recent years and is widely used in various fields of life and industry, showing great potential in solving this problem. The quantity and quality of samples directly affect the performance of machine learning algorithms. Due to the limitations of industrial production inspection environments, collecting high-quality nozzle bubble images is extremely difficult, and relevant samples are scarce, failing to meet the needs of machine learning. Therefore, it is necessary to generate images using algorithms. Current research on image generation is extensive, but its effectiveness is unsatisfactory in the aforementioned scenario, and the generated images cannot be controlled or adjusted.

[0003] Autoencoders, proposed by Rumelhart et al. in 1986, use the data itself as training labels. They first map high-dimensional data to a low-dimensional space through the encoder to achieve data compression and feature extraction, and then use a decoder to restore it to high-dimensional data for data reproduction. Generative Adversarial Networks (GANs), proposed by Goodfellow et al. in 2014, use a generator to generate samples, and then a discriminator to determine whether a sample was generated by the generator. The two learn and train alternately in an adversarial manner, eventually reaching a Nash equilibrium. At this point, the discriminator cannot distinguish whether a sample was generated by the generator, and therefore considers the generator-generated data as real data. Summary of the Invention

[0004] This invention extracts features and encodes information from the original image and a given location using an autoencoder, and then generates bubble images using a generative adversarial network. This invention can generate high-quality bubble images at specified locations, effectively solving the problem of insufficient samples in industrial production inspection environments, greatly expanding sample diversity, and providing sufficient data for the development of bubble detection algorithms.

[0005] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0006] A nozzle bubble image generation method based on autoencoders and generative adversarial networks includes the following steps:

[0007] Capture nozzle bubble images at different locations and against the background, and use them as the original image x;

[0008] The positional information of the nozzle bubble in the image is encoded to obtain multiple position codes p;

[0009] The original image x is augmented to obtain the augmented image e;

[0010] An autoencoder is constructed and trained based on the original image x, the location code p, and the augmented image e.

[0011] A generative adversarial network is constructed and trained based on the original image x, the augmented image e, and the encoder output z.

[0012] The original image x or the augmented image e, together with the location code p, are used as input data and passed sequentially through an autoencoder and a generator G in a generative adversarial network to obtain the generated image g.

[0013] The position information of the nozzle bubble in the image is encoded using a one-hot encoding method, and the encoding length is the same as the length or width of the original image.

[0014] The original image x is expanded by means of translation, rotation, scaling, blurring, cropping, and equalization, and the expansion is limited to a set range.

[0015] The input to the autoencoder is the original image x and the position code p, and the output label is the translation image t in the corresponding augmented image e. The autoencoder consists of an encoder and a decoder. The encoder is composed of a multi-layer convolutional neural network, including four convolutional layers and three fully connected layers. The convolutional layers use 3x3 convolutional kernels. The decoder is composed of a multi-layer deconvolutional neural network with the network structure reversed from that of the encoder.

[0016] When the autoencoder performs the first convolution operation, it splices the position code p to any edge of the original image x with the same encoding length, and fills the remaining edges with zeros. Subsequent convolution operations fill the image edges with zeros. The image information and position information are mapped to the encoding z through the encoder.

[0017] The loss function L of the autoencoder RE for:

[0018]

[0019] Where M×N represents the image size, p ij and Let represent the pixel values ​​of the decoded image d and the corresponding label t at point (i,j), respectively.

[0020] The autoencoder is trained iteratively using gradient descent to minimize the loss function until a given number of iterations or acceptable loss is reached.

[0021] The generative adversarial network consists of a generator G and a discriminator D, wherein the loss function of the discriminator D is:

[0022]

[0023] The loss function of the generator G is:

[0024]

[0025] Where x~P data (x) represents the probability distribution of the true sample, i.e., the original data x and the augmented data e, z ~ P z (z) represents the probability distribution of the image and location code z, n~P n (n) represents the probability distribution of Gaussian noise n;

[0026] Train the generator G to minimize L G To achieve the goal, the discriminator D is trained to maximize L. D With the goal of training both, gradient descent is used alternately until a given number of iterations or allowable loss is reached.

[0027] The present invention has the following beneficial effects and advantages:

[0028] 1. This invention provides a nozzle bubble image generation method based on autoencoder and generative adversarial network, which can generate high-quality bubble images, effectively solves the problem of insufficient samples in industrial production inspection environment, and provides sufficient data for the development of bubble detection algorithms.

[0029] 2. This invention can adjust the position of bubbles in the generated image by setting position codes, thereby improving the diversity and controllability of the generated image. Attached Figure Description

[0030] Figure 1 A schematic diagram of the network structure of this invention;

[0031] Figure 2 The generation method steps of this invention are illustrated in the diagram.

[0032] Figure 3 Schematic diagram of the expanded image of the present invention;

[0033] Figure 4 A schematic diagram of image edge filling during the first-layer convolution operation of the encoder in this invention;

[0034] Figure 5 The image generated by this invention. Detailed Implementation

[0035] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0036] like Figure 1 and Figure 2 As shown, the present invention includes the following steps:

[0037] 1. Image acquisition: Adjust the lighting and other environmental conditions around the spray nozzle, use a high-speed camera to capture images of the nozzle bubbles at different positions and against the background, and select the high-quality images as the original image x;

[0038] 2. Position Encoding: The position information of the nozzle bubble in the image is encoded using a one-hot encoding method. The encoding length is the same as the image length (or width), and several position codes p are selected.

[0039] 3. Manual image augmentation: The original image x is augmented using translation, rotation, scaling, blurring, cropping, and equalization to obtain an augmented image e, which is used for subsequent network training and image generation. All augmentations are limited to a certain range, such as the maximum angle of rotation augmentation being ±π / 6.

[0040] Step 3: The translated image t included in the expanded image e is obtained by moving the original image to the position corresponding to the position code p, such as... Figure 3 As shown;

[0041] 4. Train the autoencoder: Take the original image x and the position code p as input, and the corresponding translated image t as the output label, train the autoencoder to obtain the trained autoencoder model;

[0042] An autoencoder includes an encoder and a decoder. The encoder is composed of a multi-layer convolutional neural network, including four convolutional layers and three fully connected layers. The convolutional layers use 3x3 convolutional kernels. The decoder is composed of a multi-layer deconvolutional neural network with the network structure being the opposite of the encoder.

[0043] During the first convolutional operation, the encoder concatenates the position code p from any edge of the original image x with the same encoding length, while padding the remaining edges with zeros. Subsequent convolutional operations also pad the image edges with zeros. Image information and position information are mapped to the encoding z by the encoder for use in the training of the generative adversarial network and image generation, such as... Figure 4 As shown.

[0044] The loss function during autoencoder training is the reconstruction error, which uses the mean squared error, i.e.:

[0045]

[0046] Where M×N represents the image size, p ij and Let represent the pixel values ​​of the decoded image d and the corresponding label t at point (i,j), respectively.

[0047] The training objective is to minimize the loss function L. RE The gradient descent method is used for iterative training until a given number of iterations or acceptable loss is reached.

[0048] 5. Training the Generative Adversarial Network: The original image x and the augmented image e are mixed as the training set data of the generative adversarial network. The two images and the position code p are encoded by the encoder to obtain the image and position code z. This z, along with Gaussian noise n, is used as the input to the generative network to train the generative adversarial network and obtain the trained generative adversarial network model.

[0049] Generative Adversarial Networks (GANs) use style-GAN, which consists of a generator G and a discriminator D. The input to each level of the generator G is an image, a location code z, and Gaussian noise n.

[0050] The discriminator D loss function is as follows:

[0051]

[0052] The generator G loss function is as follows:

[0053]

[0054] Where x~P data (x) represents the probability distribution of the true sample (original data x and augmented data e), z ~ P z (z) represents the z-distribution probability of the image and location coding, n~P n (n) represents the probability distribution of Gaussian noise n, y=G(x) is the distributor function, y=D(x) is the discriminator function, and the transformation function y=f(x) is generally taken as a logarithmic function.

[0055] Train the generator G to minimize L G To achieve the goal, the discriminator D is trained to maximize L. D With the goal of training both, gradient descent is used alternately until a given number of iterations or allowable loss is reached.

[0056] 6. Generate bubble image: Input the original image x and the augmented image e. The position of the nozzle bubble in the generated image can be adjusted by setting the position code p. Using the encoder G trained in the first two steps, the generated image g can be obtained. Then, high-quality bubble images are manually selected for use, such as... Figure 5 As shown.

Claims

1. A method for generating nozzle bubble images based on autoencoders and generative adversarial networks, characterized in that, Includes the following steps: Capture images of nozzle bubbles at different locations and against different backgrounds as raw images. ; The positional information of the nozzle bubble in the image is encoded to obtain multiple position codes. ; For the original image To expand, you get an expanded image. ; Based on the original image Location code And expanded images Construct and train an autoencoder; Based on the original image Expanding images and the image and position encoding output by the encoder Construct and train a generative adversarial network; Original image and expanded images , with location code The data, taken as input, are passed sequentially through an autoencoder and a generator in a generative adversarial network. The generated image is obtained. ; The input to the autoencoder is the original image. With position code The output labels are the corresponding augmented images of the two. Translation image in , the original image With position code As input, the corresponding translation images of both. As the output label, an autoencoder is trained. The autoencoder consists of an encoder and a decoder. The encoder is composed of a multi-layer convolutional neural network, including four convolutional layers and three fully connected layers. The convolutional layers use 3x3 convolutional kernels. The decoder is composed of a multi-layer deconvolutional neural network with the network structure being the opposite of the encoder. When the autoencoder performs the first convolution operation, the original image The code for splicing the edges of the same length on either side of the code. The remaining edges are zero-padded, and subsequent convolution operations also zero-padded the image edges. Image information and positional information are mapped to image and positional codes by the encoder. middle.

2. The nozzle bubble image generation method based on autoencoder and generative adversarial network according to claim 1, characterized in that, The position information of the nozzle bubble in the image is encoded using a one-hot encoding method, and the encoding length is the same as the length or width of the original image.

3. The nozzle bubble image generation method based on autoencoder and generative adversarial network according to claim 1, characterized in that, The original image was processed using translation, rotation, scaling, blurring, cropping, and equalization techniques. Expansions can be made, and all expansions are limited to the set range.

4. The nozzle bubble image generation method based on autoencoder and generative adversarial network according to claim 1, characterized in that, The loss function of the autoencoder for: ； in, Indicates image size, and Representing the decoded images With corresponding tags exist pixel value; The autoencoder is iteratively trained using gradient descent to minimize the loss function until a given number of iterations or acceptable loss is reached.

5. The nozzle bubble image generation method based on autoencoder and generative adversarial network according to claim 1, characterized in that, The generative adversarial network consists of a generator. With discriminator The discriminator is configured such that... The loss function is: ； The generator The loss function is: ； in, Real samples, i.e., raw data With expanded data The probability distribution, Image and location encoding The probability distribution, Gaussian noise The probability distribution; Training Generator To minimize To train the discriminator To maximize With the goal of training both, gradient descent is used alternately until a given number of iterations or allowable loss is reached.

Images