CycleGAN model-based retaining ring gluing image enhancement detection method

Abstract

The invention discloses a retaining ring gluing image enhancement detection method based on a CycleGAN model. Comprising the steps of collecting an original gluing image; carrying out VGG16 neural network identification and classification; enhancing the image by using the improved CycleGAN network model; and in the gluing area, the gluing state is judged. According to the method, the low-contrast part of the glue area can be effectively enhanced, the detection precision is improved, the edge information of the glue image is enriched, and a more accurate gluing state image can be obtained. The final detection success rate of the image obtained through the method can reach 95%, more accurate information support can be provided for the gluing process of the industrial robot, the working stability between the retaining ring and the polishing head and between the retaining ring and the wafer is guaranteed, the yield in the chip manufacturing process is improved, and the method has actual production significance.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics, image enhancement, CMP polishing, and wafer manufacturing, specifically a method for image enhancement and detection of retaining ring coating based on the CycleGAN model. Background Technology

[0002] In recent years, with the rapid development of technologies such as artificial intelligence, the Internet of Things, 5G communication, and smart cars, the chip industry has ushered in unprecedented development opportunities. Globally, the chip industry has become a key area of ​​development for various countries. Meanwhile, the development of chips in recent years has shown the following characteristics: First, market demand continues to grow. The increasing demand for chips from fields such as smart terminals, data centers, and automotive electronics has driven the rapid development of the chip market. Second, technological innovation is advancing rapidly. Driven by Moore's Law, chip manufacturing processes are constantly being upgraded, and chip performance is continuously improving. Based on this, the chip manufacturing field needs to continuously introduce new technologies to ensure production and achieve synchronous growth in production and demand. Among the series of processes in integrated circuit manufacturing, wafer planarization achieved through chemical mechanical polishing (CMP) is one of the key steps to ensure the final wafer production yield.

[0003] Chemical Mechanical Polishing (CMP) is a crucial step in integrated circuit manufacturing, with many steps significantly impacting the entire process. 1. For dielectric layer removal, CMP removes excess dielectric layers from multilayer wiring structures to facilitate the deposition of the next metal layer. This process requires uniform material removal to ensure the uniformity and reliability of subsequent steps. 2. For dimensional control, the accuracy of CMP directly affects the control of the critical dimension (CD) of the circuit. A good CMP process ensures consistent line widths, thereby improving performance and yield. 3. For surface planarization requirements, CMP technology is used to eliminate surface irregularities generated during photolithography and etching. 4. In advanced manufacturing processes, line widths are increasingly smaller, demanding extremely high surface flatness. CMP can provide sub-angstrom level surface flatness for subsequent processing. 5. CMP is also used for metal layer planarization, which is crucial for forming good electrical contacts and preventing short circuits. Therefore, the CMP process plays a crucial role in wafer manufacturing, and its optimization and innovation have always been one of the research hotspots in the semiconductor industry. Advances in CMP technology have directly driven the development of integrated circuit manufacturing technology, and have a profound impact on improving chip performance, reducing costs, and increasing production efficiency.

[0004] To increase chip production and reduce manufacturing costs, wafer diameters are constantly increasing. However, as wafer diameters increase, warping and deformation are more likely to occur during processing. For wafers with diameters greater than 150mm, "over-grinding" can easily occur at the wafer edges, reducing polishing quality and wafer utilization. To solve this problem, adding a retaining ring to the outside of the wafer can effectively address this issue.

[0005] Adding a retaining ring around the wafer during the CMP process has many advantages.

[0006] 1. Wafer Fixing: The retaining ring is used to fix the wafer, ensuring that the wafer remains flat and firmly located on the polishing head or turntable during the polishing process, preventing the wafer from shifting during CMP and ensuring the uniformity of the polishing effect.

[0007] 2. Prevent wafer damage: The design and materials of the retaining ring prevent physical damage to the wafer during the holding process.

[0008] 3. Pressure transmission: During the CMP process, the retaining ring helps to transmit pressure evenly to the wafer surface, ensuring uniform polishing.

[0009] 4. Preventing chemical leakage: The retaining ring prevents polishing slurry and chemical solutions from leaking during the CMP process, ensuring that the polishing slurry can effectively act on the wafer surface.

[0010] 5. Control the flow of polishing slurry: The design of the retaining ring can affect the flow characteristics of the polishing slurry, which helps to optimize the distribution of the polishing slurry, thereby improving polishing efficiency and uniformity.

[0011] The stability of the retaining ring during polishing is closely related to its bonding effect. Currently, the properties of the retaining ring (PPS) bonded to stainless steel are mainly categorized into physical and chemical properties. Physical properties represent the mechanical properties of the adhesive, while chemical properties depend primarily on the adhesive type. Therefore, in automated adhesive application processes, the mechanical properties of the retaining ring are the primary factor affecting its bonding. Since the force exerted on the retaining ring during operation is essentially equivalent to shear force, ensuring the quality of the adhesive application is paramount to guaranteeing its effectiveness. Achieving good bonding requires considering both the adhesive layer thickness and the actual bonding area. In actual application, when the adhesive layer is less than 0.3mm, the bonding performance is mainly related to the application area. However, current machine adhesive application methods struggle to control both thickness and area effectively, primarily due to the transparent nature of the adhesive, resulting in poor contrast and difficulty in extracting the adhesive outline, severely impacting detection results and the accuracy of the application. Furthermore, traditional image enhancement algorithms, which alter the overall grayscale value of the image, are insufficient to effectively improve the contrast of the adhesive area. Summary of the Invention

[0012] To address the aforementioned shortcomings of existing technologies, this invention obtains the adhesive coating image between the holding ring and the polishing head during CMP polishing using computer image processing. The image is then enhanced using a series of detection methods based on the CycleGAN model mentioned in this invention to obtain a more accurate adhesive coating image. This invention effectively enhances the features of the adhesive coating image, providing reliable information support for subsequent processes such as adhesive coating compliance detection and adhesive coating path algorithms. This ensures the operational stability of the holding ring and improves the wafer yield during wafer manufacturing.

[0013] The technical solution adopted by the present invention to achieve the above objectives is: a retention ring coating image enhancement and detection method based on the CycleGAN model. This method can accurately obtain the coating contour, provide information support for automatic coating devices, and detect coating quality.

[0014] Step 1: Acquire the original image of the adhesive coating, perform region segmentation and image processing operations, and create a dataset;

[0015] Step 2: Pre-train the VGG16 neural network. Use the dataset to perform classification enhancement iterative training on the pre-trained VGG16 model, so that the network can recognize and classify high-contrast and low-contrast images and output them.

[0016] Step 3: Improve the CycleGAN network model by using the image output by the VGG16 model as input for training. This model learns the features of high-contrast images, transforms low-contrast images into high-contrast images, and achieves image enhancement and output.

[0017] Step 4: Extract the glued area from the enhanced image and determine the glued state.

[0018] The process of creating the dataset includes the following steps:

[0019] The original image is divided into several patches as sample instances;

[0020] Edge extraction is performed on each sample instance using the Otsu threshold segmentation algorithm;

[0021] Based on the degree of edge roughness and shape irregularity, images are classified into high-contrast and low-contrast images;

[0022] The sample data was augmented by rotation and mirroring to create a sample dataset.

[0023] The pre-trained VGG16 neural network was trained on the ImageNet dataset to obtain the corresponding neuron parameters for each layer of the network.

[0024] The VGG16 includes 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The parameters of the first 13 network nodes are frozen during pre-training, and the last three fully connected layers are trained using a sample dataset to obtain the network model.

[0025] The improved CycleGAN network model includes two generators and one discriminator.

[0026] Two generators, F and G, each consisting of 3 convolutional layers, 9 residual layers, 2 deconvolutional layers, and 1 final convolutional layer, are used to generate high-contrast images and to compute the cycle consistency of the neural network.

[0027] The discriminator D consists of 5 convolutional layers, using the LeakyReLU activation function to identify whether the image is the original high-contrast image or a generated high-contrast image.

[0028] In the two generators, the low-contrast image identified by the VGG16 model is input into the generator G of the CycleGAN model to generate a high-contrast image. Then, another generator F is used to convert the high-contrast image generated by generator G back into a low-contrast image and compare it with the original low-contrast image to calculate the cycle consistency loss.

[0029] The total loss of CycleGAN is the sum of the loss functions of the generator G, the discriminator D, and the cycle consistency loss function.

[0030] Determining the adhesive application status involves judging the adhesive outline characteristics, including uniformity, bonding area, and offset.

[0031] The present invention has the following beneficial effects and advantages:

[0032] 1. The CycleGAN method proposed in this invention is not only simple to construct and fast to train, but also able to apply high-contrast image features to low-contrast image features, enhance the information of specific regions of the image, thereby significantly improving the gray value of the coated area, enhancing the image contrast, and making the outline of the coated area clearer.

[0033] 2. By using image segmentation methods and feeding the data into the CycleGAN model for training, the high-contrast image obtained under the CycleGAN model can enhance the glue area while keeping the grayscale value of the background area unchanged, thus avoiding the impact on the overall image.

[0034] 3. The CycleGAN model has been simplified to address the need to transform low-contrast images into high-contrast images, reducing model redundancy and improving training speed.

[0035] 4. Using the Log-Cosh loss function instead of the MEA loss function in the original CycelGAN and adopting the LeakyReLU activation function can make the CycelGAN model have better training stability and faster convergence speed for coating image enhancement.

[0036] 5. The enhancement algorithm proposed in this invention outperforms other enhancement algorithms in enhancing the detection performance of glue coating images, effectively improving the accuracy of glue coating quality detection and meeting the basic requirements of engineering testing. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method of the present invention;

[0038] Figure 2 These are the original glue-coating images captured by the computer in this invention;

[0039] Figure 3 This invention utilizes Halcon to segment 20 regions of images;

[0040] Figure 4 This is a simplified CycleGAN model diagram from the present invention;

[0041] Figure 5 This is a comparison image of 20 regions before and after enhancement according to the present invention;

[0042] Figure 6 This is a comparison image of edge detection before and after image enhancement according to the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific implementation methods of the present invention will be further described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention; however, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific implementations disclosed below.

[0044] Industrial robot technology has been increasingly widely applied in the field of intelligent manufacturing. Especially in adhesive coating operations, it can significantly solve problems such as low efficiency, unstable coating quality, and health hazards to operators from harmful gases during the coating process, which are inherent in manual operations. The coating level of the industrial robot directly affects the stability of the retainer ring during operation, and thus the final yield of the chip. This invention addresses the problem of uneven adhesive dispensing by industrial robots during retainer ring coating by proposing a retainer ring adhesive coating detection method based on CycleGAN image recognition. The adhesive coating enhancement detection of this invention mainly consists of four steps:

[0045] Step 1: Use Halcon software to segment the glue distribution image into 20 regions. Use the Otsu thresholding algorithm to extract edge information from each image, analyze the shape and smoothness of the extracted edges, and combine it with the VGG16 algorithm for processing. The processing results are divided into high-contrast images and low-contrast images.

[0046] Step 2: For the low-contrast images obtained in Step 1, train a customized CycleGAN model to convert them into high-contrast images. An improved cycle consistency loss function is used to enhance the model's training stability.

[0047] Step 3: Use a fixed threshold segmentation method to extract the glue outline from the enhanced low-contrast image. Perform dilation and erosion morphological operations on the extracted glue region to obtain the complete glue region. Then, perform edge extraction on the glue region to obtain the complete glue outline.

[0048] Step 4: Judge the quality of the glue based on its outline characteristics. The main indicators include: uniformity, bonding area, and offset.

[0049] The key to the above steps lies in how to solve the following two problems:

[0050] First, how to differentiate the glue distribution images and rationally divide the original image into 20 low-contrast and high-contrast images; second, how to enhance the low-contrast image based on the existing high-contrast and low-contrast images.

[0051] To address the first problem, this invention employs the VGG16 model to distinguish between low-contrast and high-contrast images. The original image is segmented into 20 regions, generating 20 sample images. The Otsu thresholding algorithm is used to extract edges from each sample; images with rough edges and irregular shapes are classified as low-contrast images. Combined with manual judgment, the 20 regions are further categorized into high-contrast and low-contrast images. Subsequently, methods such as rotation and mirroring are used to enhance the sample data, increasing the number of samples and improving the model training results. The VGG16 model is then used to train on both high-contrast and low-contrast images, learning their features. After training, the model is tested on a test machine. The VGG16 model contains multiple convolutional and pooling layers, has a relatively simple structure, is easy to deploy, and can be readily applied to real-world scenarios. It can also extract deep features from images, effectively solving classification problems. Furthermore, since VGG16 has already been pre-trained on the ImageNet dataset, it contains rich image features, which can accelerate training speed, improve model performance, and avoid the divergence problem of models trained from scratch.

[0052] Regarding the second question, this invention uses the CycleGAN model to enhance low-contrast images. CycleGAN is a derivative model of the GAN method proposed in 2017. Currently, CycleGAN models are mainly used for style transfer, realizing the transformation between two source domains, corresponding to the low-contrast and high-contrast images in this invention. However, in this practice, only the low-contrast image needs to be transformed into a high-contrast image, without the reverse process. Therefore, the CycleGAN mentioned in this invention is a simplified model, which reduces one discriminant expression compared to the complete Cycle model, thus facilitating model training. In addition, the CycleGAN model in this invention contains two generators and one discriminator. The generator contains convolutional layers, residual layers, deconvolutional layers, and activation functions to extract features and generate high-contrast images. The discriminator contains convolutional layers, activation functions, and a sigmoid function to distinguish between the generated high-contrast images and the real high-contrast images. The activation function in the discriminator is the LeakyReLU function, which is a customized improvement of the ReLU function in the context of this invention, capable of solving the gradient vanishing problem that may occur.

[0053] Example: The present invention provides an image enhancement and detection method for adhesive coating based on the CycleGAN model, such as... Figure 1 As shown. First, the glue area in the image is divided into 20 regions, as follows. Figure 2 As shown, a trained VGG16 classification model was used to classify the 20 regions to be processed into high-contrast and low-contrast images. Then, a CycleGAN model was used to train the low-contrast images and convert them into high-contrast images. Finally, thresholding and edge detection were used to extract the glue outline, ensuring the detection of glue application quality. Experimental results show that this method can effectively enhance the low-contrast parts of the glue region and improve detection accuracy.

[0054] The specific steps are as follows:

[0055] like Figure 3 As shown, the computer-generated photo of the adhesive coating was divided into 20 regions using Halcon, resulting in the 20 regions shown in the figure below. Due to the limited amount of sample data, the small sample size can lead to underfitting of the neural network during training. Therefore, it is necessary to increase the number of sample images. In this practice, the following methods were mainly used to increase the number of samples: rotation, mirroring, and rotation-mirroring, which increased the data volume by 7 times.

[0056] When training the VGG16 model, it was implemented using the open-source deep learning framework TensorFlow. Since the complete training process of VGG16 requires over 100 million parameters, resulting in excessively long training time, a pre-training-retraining approach was adopted. Pre-training was performed on the ImageNet dataset (the corresponding trained weights have been published, further reducing project time), obtaining the corresponding neuron parameters. These parameters were then frozen to a certain extent before training on the current dataset. Experiments showed that freezing different numbers of layers resulted in different classification effects after training, as detailed in Table 1. This study used the method of freezing the parameters of the first 13 layers of VGG16, training the last three fully connected layers using the sample images provided for this task, which yielded the highest classification accuracy.

[0057]

[0058] In this training process, 3000 sample data were used, including 1500 high-contrast images and 1500 low-contrast images. The VGG16 model was trained using gradient descent, with a batch size of 64 and a learning rate of 0.0001. The final classification accuracy was around 95%, demonstrating good classification performance and avoiding overfitting.

[0059] After classifying real-time images into high-contrast and low-contrast categories using the VGG16 model, a CycleGAN model is trained to learn the features of high-contrast images and transform low-contrast images into high-contrast images. In this training, the CycleGAN model was simplified. Compared to the original CycleGAN with two generators and two discriminants, the modified CycleGAN contains only two generators and one discriminant. The original model and the optimized model are shown below. Figure 4 As shown.

[0060] The two generators, F and G, mainly consist of 3 convolutional layers, 9 residual layers, 2 deconvolutional layers, and 1 final convolutional layer. Their primary function is to calculate the cycle consistency of the neural network. The identified low-contrast image is input into generator G of the CycleGAN model to generate a high-contrast image. Then, generator F converts the high-contrast image generated by generator G back into a low-contrast image and compares it with the original low-contrast image to calculate the cycle consistency loss.

[0061] The discriminator D is responsible for identifying whether an image is the original high-contrast image or a generated high-contrast image. It consists of 5 convolutional layers, using the LeakyReLU activation function to improve the stability of the discriminator during training. The definition of LeakyReLU is shown below.

[0062] The loss function of the entire CycleGAN can be composed of the loss function of the generator G, the loss function of the discriminator D, and the cycle consistency loss function. The loss functions of the generator G and the discriminator D can be illustrated as follows.

[0063] L cyc (G,F)=E x～Pdata(x) [||F(G(x))-x||1]+E y～Pdata(x) [||F(G(y))-y||1]

[0064] y = log(cosh(x))

[0065] Where x represents image information from domain X, y represents image information from domain Y, G(x) represents the Y-domain image information obtained after the X-domain image is processed by generator G, G(y) represents the X-domain image information obtained after the Y-domain image is processed by generator G, F(G(x)) represents the transformation of the Y-domain image obtained by G(x) into the X-domain image by generator F, F(G(y)) represents the transformation of the X-domain image obtained by G(y) into the Y-domain image by generator F, ||||1 is defined as a first normal form, represents the absolute value of the variable, and cosh(x) represents a hyperbolic function.

[0066] In the original CycleGAN, the MEA loss function is used to represent the consistency of the cyclic loss. However, the gradient of the MEA loss function remains the same as the neural network parameters change. When the loss value is very small, the gradient is also very large, which often makes it difficult for the neural network to converge. In this invention, the MEA loss function in the CycleGAN model is replaced with the MSE loss function. The MSE loss function can change the gradient according to the magnitude of the loss value, which can ensure the fast convergence of the neural network and guarantee the convergence stability of the neural network. The cyclic loss function and the final loss function are expressed as follows.

[0067] L gen (G,D,X)=E x～Pdata(x) [(D(G(x))-1) 2 ]

[0068] L dis (G,D,X,Y)=E y～Pdata(y) [(D(y)-1) 2 ]+E x～Pdata(x) [D(G(x))2 ]

[0069] L(G,F,D Y ) = L gen (G,D,X)+L dis (G,D,X,Y)+λL cyc (G,F)

[0070] Where D(y) represents the judgment of the image in the Y domain using discriminator D, and D(G(x)) represents the judgment of the image in the X domain after passing through generator G using discriminator D; L cyc (G,F), L gen (G,D,X),L dis (G,D,X,Y) represent the cycle consistency loss function, generator loss function, and discriminator loss function, respectively; L(G,F,D) Y ) represents the total loss function.

[0071] Currently, the industry commonly uses a fixed segmentation method to extract glue outlines. However, this method produces glue outlines with rough edges and significant errors. Experiments have shown that the success rate of accurately extracting low-contrast glue outlines is only about 25%, far from meeting the inspection requirements of production. However, after our image enhancement, the image contrast is improved, and the glue outline extraction using the fixed segmentation method yields smoother edges and a significantly increased success rate, reaching approximately 99%. Specific improvements are shown below. Figure 5 , 6 As shown. Figure 5 Image (a) shows that the original coated image has low edge contrast, making it difficult to perform edge detection later. Image (b) shows that after the original image is enhanced by the modified CycleGAN, the edge contrast is increased, which is beneficial for edge detection. Figure 6 (a) shows that the edge detection results of the original image are of poor accuracy and cannot reflect the true coating condition. (b) shows that after the original image is enhanced, the improved edge contrast is beneficial for edge detection, and the results can more realistically reflect the coating information of the retaining ring.

[0072] The embodiments described above will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

Claims

1. A method for enhancing and detecting retention loop coated images based on a CycleGAN model, characterized in that, To accurately identify the adhesive application outline and determine the adhesive application quality, the following steps are required: Step 1: Acquire the original image of the adhesive coating, perform region segmentation and image processing operations, and create a dataset; Step 2: Pre-train the VGG16 neural network. Use the dataset to perform classification enhancement iterative training on the pre-trained VGG16 model, so that the network can recognize and classify high-contrast and low-contrast images and output them. Step 3: Improve the CycleGAN network model by using the image output by the VGG16 model as input for training. This model learns the features of high-contrast images, transforms low-contrast images into high-contrast images, and achieves image enhancement and output. Step 4: Extract the glued area from the enhanced image and determine the glued state.

2. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, The process of creating the dataset includes the following steps: segmenting the original image into several patches as sample instances; using the Otsu thresholding algorithm to extract edges from each sample instance; classifying the images into high-contrast and low-contrast images based on edge roughness and shape irregularity; and using rotation and mirroring to augment the sample data and create a sample dataset.

3. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, The pre-trained VGG16 neural network was trained on the ImageNet dataset to obtain the corresponding neuron parameters for each layer of the network.

4. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, The VGG16 includes 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The parameters of the first 13 network nodes are frozen during pre-training, and the last three fully connected layers are trained using a sample dataset to obtain the network model.

5. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, The improved CycleGAN network model includes two generators and one discriminator. Two generators, F and G, each consisting of 3 convolutional layers, 9 residual layers, 2 deconvolutional layers, and 1 final convolutional layer, are used to generate high-contrast images and to compute the cycle consistency of the neural network. The discriminator D consists of 5 convolutional layers, using the LeakyReLU activation function to identify whether the image is the original high-contrast image or a generated high-contrast image.

6. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, In the two generators, the low-contrast image identified by the VGG16 model is input into the generator G of the CycleGAN model to generate a high-contrast image. Then, another generator F is used to convert the high-contrast image generated by generator G back into a low-contrast image and compare it with the original low-contrast image to calculate the cycle consistency loss.

7. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, The total loss of CycleGAN is the sum of the loss functions of the generator G, the discriminator D, and the cycle consistency loss function.

8. The image enhancement and detection method for retaining loop coating based on the CycleGAN model according to claim 1, characterized in that, It determines the adhesive application status based on the characteristics of the adhesive outline, including uniformity, bonding area, and offset.

Images