Nanoimprint defect detection system and method based on deep learning technology

Abstract

The invention relates to a nanoimprint defect detection system and a nanoimprint defect detection method based on a deep learning technology. The method comprises the following steps: step 1, firstly, controlling the brightness of a light source through a light source controller, and collecting a wafer picture with defect detection sample data through a camera; then, the acquisition card transmits the wafer image to a computer; building a design process of a target segmentation training detection system; 2, visual image analysis software is used for preprocessing the wafer image; and step 3, performing defect detection on each picture by using a YOLOv8n-seg target segmentation algorithm for the small picture sample, performing data labeling by using Label, and converting a labeling result into a txt format from a JSON format to serve as a sample data set for model training.

Description

Technical Field

[0001] The present invention relates to a nanoimprint defect detection system and method based on deep learning technology. Background Art

[0002] The diversity of industrial manufacturing, the complexity of the production environment, the non-standard nature of product defects and other factors have brought many challenges to the actual application of machine vision in wafer defect detection. Applying machine vision detection technology to wafer defect detection of nanoimprint lithography technology can intelligently solve the problem of product failure caused by excessive number of final product defects due to factors such as imprint step errors and substandard dust-free production environment during the nanoimprint process, thereby improving the quality and efficiency of wafer nanoimprint inspection.

[0003] Detecting nanoimprint wafer defects is challenging because wafer defects occupy few pixels in the image and provide limited feature information. For example, a wafer defect may only occupy a few pixels and lack clear outlines and details. The quality of the wafer defect detection model depends on the sample data for training, and most defect detection datasets focus on large and medium-sized defect types, which means that existing models may not be optimized for small target defects. In addition, wafer defect detection models usually require fixed-size input images. For example, YOLOv8 is trained using images with a maximum side length of 640 pixels. When a high-resolution image (such as 1920x1080) is input, the model will downsample it to 640x360 before processing, resulting in a decrease in resolution and loss of key information of small objects.

[0004] In addition, the prior art has problems such as poor hardware imaging quality, weak visual imaging, uneven distribution of defect types, difficulty in sample collection in a high-yield environment, unknown defects caused by process instability, optimization of detection of specific defect types, and slow real-time detection of imprinting speed. Therefore, the present invention aims to construct an efficient, accurate and robust defect detection system to improve production efficiency and product quality. Summary of the invention

[0005] In view of the above-mentioned problems in defect detection, the purpose of the present invention is to provide an efficient, accurate and robust small target defect detection system and method to overcome the deficiencies in the prior art. In view of the problems of small wafer defects, low accuracy, low precision, high classification difficulty, slow detection speed, etc., the nanoimprint defect detection method proposed in this patent solves the problem of large number and complexity of defects in the wafer imprint process, can provide good guidance for process production, and help improve product quality and production speed.

[0006] The technical problem to be solved by the present invention is generally to provide a nanoimprint defect detection system and method based on deep learning technology.

[0007] To solve the above problems, the technical solution adopted by the present invention is:

[0008] A nanoimprint defect detection method based on deep learning technology, the method comprising the following steps;

[0009] Step 1: First, control the brightness of the light source through the light source controller, and use the camera to collect wafer images with defect detection sample data; then, the acquisition card transmits the wafer image to the computer; build the design process of the target segmentation training detection system;

[0010] Step 2: Visual image analysis software to pre-process the wafer image;

[0011] Among them, during image analysis, first, for the wafer image of the original defect with a set size of A*A, defined as the original image, the algorithm code is used to segment it into each model training image of the set size a*a as a small image sample (whether it is correct);

[0012] Step 3: Use the YOLOv8n-seg target segmentation algorithm to detect defects in each small image sample, use Labelme to annotate the data, and convert the annotation results from JSON format to txt format as a sample data set for model training;

[0013] Step 4: randomly divide the sample data set into training set and test set, train and evaluate the performance of the model; the detection results are instance segmented and post-processed, and finally the defect detection results of each slice are output;

[0014] Step 5: merge all the segmented slice image results to form a complete large image, that is, the original image; process and reason on the large image to obtain the complete defect detection results of each original image;

[0015] Step six: Analyze and count the defects of all original images, store the output data analysis results, and make targeted corrections and guide the process operation flow based on the defect statistics.

[0016] Furthermore, the process operation flow includes image acquisition, image processing, image analysis, model training and result output.

[0017] Furthermore, in step 5, when generating the detection list of the entire original image, the slices in the last row and the slices in the last column are overlapped to ensure that the object can be detected in at least one slice;

[0018] When merging the detection results of slices, remove the objects with duplicate detections.

[0019] Further, in step 1, the design process of the target segmentation training detection system is as follows;

[0020] S1.1, prepare; build the GPU operating environment required by the Yolov8 algorithm, install dependent libraries, and convert wafer defect data images into png format;

[0021] S1.2, data preparation; select defect images from the collected original images, create labels for the training set, annotate the data set with labelme, manually annotate the selected defect images, mark the type and location of the defects, generate a json file, and run the json format conversion to txt format;

[0022] Defects include black spots, halos, smears, and scratches;

[0023] S1.3, data division; random division of training set and test set: randomly divide the labeled data set into two parts, one for training the model and the other for verifying the effect of the model;

[0024] S1.4, training model; first, build and improve the segmentation network, modify the YOLOv8n-seg network architecture according to actual needs, so that the YOLOv8n-seg network architecture is suitable for crack detection tasks; then, set the training parameters, and set the key parameters before training the model; secondly, use the labeled training set to train the improved YOLOv8n-seg network. During the training process, record the changes in the loss value and draw the loss function attenuation curve to observe the model convergence;

[0025] Key parameters include learning rate, batch size, and number of iterations;

[0026] S1.5, adjust parameters and test training results: After the model training is completed, use the test set to test the performance of the model; if the loss value stops decreasing or reaches the predetermined convergence condition, the model is considered to have converged; otherwise, return to S1.1, continue to adjust the training parameters and retrain;

[0027] S1.6, evaluate the performance of the wafer defect detection model; collect statistics on the defect detection results in the test set of S1.3, calculate the detection accuracy, evaluate the wafer defect detection accuracy, and obtain the defect detection model from data collection to model training and evaluation.

[0028] Further, in step 2, the design process for the defect image reasoning stage is as follows;

[0029] S2.1, small picture wafer defect detection; first, the A*A image is segmented into several a*a small pictures, and the yolov8n-seg defect detection model is inferred for each small picture independently. Each small picture is inferred by the model to identify the objects in the small picture; then, the image is decomposed into manageable parts;

[0030] S2.2, model reasoning acceleration; using TensorRT for acceleration;

[0031] Among them, the entire process of TensorRT model reasoning acceleration is divided into three steps, namely model parsing, Engine optimization and execution;

[0032] S2.3, result merging: merging the detection results of all slices back into the original image, which involves locating and combining the information from each slice to reconstruct the complete detection result;

[0033] S2.4, post-processing, finally, a complete inference process is performed to obtain the final detection result.

[0034] A nanoimprint defect detection system based on deep learning technology, the system builds and executes the above method.

[0035] Further, the system includes a computer, as a data processing, installed with visual image analysis software;

[0036] A light source controller, used to control the brightness of the light source;

[0037] A camera for collecting wafer images with defect inspection sample data;

[0038] Capture card, which transfers wafer images to a computer;

[0039] The computer is electrically connected to the light source controller, the camera and the acquisition card.

[0040] Furthermore, the visual image analysis software pre-processes the wafer image; the pre-processing includes gray-scaling, noise removal, and data image enhancement.

[0041] The present invention solves the problem of poor hardware imaging quality. In existing real projects, visual imaging is weak and some subtle defects are difficult to capture and identify clearly. Therefore, in order to improve the visual imaging quality, detect subtle defects, and improve the quality of image acquisition and processing, the present invention uses a high-resolution camera and a point light source lighting system, combined with image white balance FFC enhancement technology, to improve image clarity and detail expression.

[0042] The present invention overcomes the problem of few defective samples. In the existing real production environment, the wafer product yield is high and it is difficult to collect defective samples. The present invention can still effectively collect and annotate defective samples under the condition of high production yield, and train and verify the detection model based on the TensorRT reasoning acceleration engine. The present invention introduces active learning and semi-supervised learning technology, and gradually optimizes the performance of the model using a small amount of labeled data and a large amount of unlabeled data. At the same time, data enhancement generation technology is used to generate diversified defective samples and enrich the training data set.

[0043] The present invention solves the problem of low model detection accuracy. The existing technology has few types of wafer defect detection, and the detection is rough, which cannot achieve the detection accuracy of small target defects. Due to the uneven distribution of wafer defect types in actual production, there may be extremely uneven distribution phenomena. The existing technology has low detection accuracy for intensive defects. The present invention adopts the deep learning model yolov8n-seg network, combined with data enhancement and fine-grained target segmentation detection algorithm, to ensure that the model can handle multiple types of defects, especially in categories with less data volume, it can also achieve good results.

[0044] The present invention solves the problems of missed detection and high false detection rate, improves the model's detection capability for unknown defects, and reduces missed detection and false detection. There are four main types of defects, namely black spots of different sizes, halos, tails, scratches, etc., and halos are greatly affected by light. For the main defect types such as black spots, halos, tails, scratches, etc., the present invention optimizes the detection algorithm to improve the detection accuracy. For each defect type, the present invention designs a special detection strategy and feature extraction method. Combined with the traditional defect detection model, unknown defects are identified and classified, a dynamic update mechanism is established, new defect samples are continuously collected, and the model is updated.

[0045] The present invention meets the strict speed requirements of the stamping process, while ensuring the detection accuracy, improving the detection speed, and meeting the strict stamping speed requirements on the production line. The present invention optimizes the model structure and reasoning process, reduces the amount of calculation and delay, and uses hardware acceleration technology GPU and TensorRT to accelerate reasoning technology to achieve real-time and rapid detection. At the same time, it optimizes data transmission and processing procedures to reduce unnecessary computing overhead.

[0046] The present invention has good quality control and defect detection; in the production process of nanoimprint templates, any tiny defects may affect the performance of the final product. Using a high-resolution camera to capture images of the template surface, and using a network model algorithm to detect defects in the nanoimprint template or finished product in real time, such as scratches, stains, halos, etc., can monitor product quality in real time on the production line, and promptly discover and correct problems.

[0047] The data analysis and optimization of the present invention are good. YOLOv8n-seg can generate a large amount of detection data, which can be used for further analysis and optimization. The model can detect defects such as scratches and particle contamination in real time and generate defect reports. Through data analysis, common problems and bottlenecks in the production process can be found, thereby optimizing process parameters and improving production yield and efficiency.

[0048] The automated production process of the present invention is advanced. In the nanoimprint production line, the YOLOv8n-seg algorithm can be integrated for automated detection and control. Through real-time detection and control, problems in the production process can be discovered and solved in a timely manner, and process generation parameters can be automatically adjusted and optimized to achieve intelligent production and quality management. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the overall working process of the wafer defect detection system of the present invention.

[0050] Figure 2 It is a schematic diagram of the slicing and cutting method of the present invention.

[0051] Figure 3 It is a schematic diagram of the image segmentation process of the present invention.

[0052] Figure 4 It is a schematic diagram of the detection process of the present invention.

[0053] Figure 5 It is a schematic diagram of the TensorRT model flow of the present invention.

[0054] Figure 6 It is a partial schematic diagram A of the overall workflow of the present invention.

[0055] Figure 7 It is a partial schematic diagram B of the overall workflow of the present invention. DETAILED DESCRIPTION

[0056] like Figure 1-7The system controls the brightness of the light source through the light source controller, uses a high-resolution camera to collect wafer defect detection sample data images, and then transfers the images to the computer through the acquisition card. The visual image analysis software preprocesses the wafer image, including grayscale, noise removal, data image enhancement, etc. to improve the image quality. For the original defect image of 4512*4512 size, the algorithm code is used to segment it into model training images of 640*640 size each to ensure the refinement and accuracy of the detection. The small image sample uses the YOLOv8n-seg target segmentation algorithm to detect defects in each image, and Labelme is used for data annotation. The annotation results are converted from JSON format to txt format for model training. The sample data set is randomly divided into training set and test set, and the performance of the model is trained and evaluated.

[0057] After instance segmentation and post-processing, the detection results are finally output for each slice. All segmented image results are merged, and complete large-image post-processing reasoning is performed to obtain 4512*4512 complete defect detection results for each original image. All original images are analyzed and statistically analyzed for defects, and the output data analysis results are stored. The process operation flow is intelligently corrected and guided based on the defect statistical data. The entire process covers all aspects from image acquisition, processing, analysis to model training and result output, ensuring efficient defect detection and high-quality production control. Figure 1 Demonstrates the overall workflow of a wafer defect inspection system.

[0058] Considering the memory requirements, computing costs, and workload of dataset construction, the present invention adopts a 640*640 size for dataset training. Before training, the original image is sliced ​​into 4512*4512 pixels, and the length and width of the slice are set to 640. Each original image corresponds to 64 640*640 images as the dataset for model training, thereby improving the detection accuracy of small objects.

[0059] The image is divided into multiple fixed-size slices covering the entire image, and then detection reasoning is performed on each slice separately, and the results are merged to generate a detection list for the entire image. The last row and the last column allow overlap between slices to ensure that the object can be detected in at least one slice, such as Figure 2 As shown, the last slice of the first row is cropped.

[0060] The 4512*4512 high-resolution image captured by the camera is divided into small images of a fixed size of 640*640 using an algorithm. Target segmentation detection is performed on each small image separately, and the detection results of the 64 small images are spliced ​​and merged. The detection results of each slice are merged, the objects of repeated detection are removed, and the final detection list of the entire image is generated.

[0061] The following are the detailed steps for the design process of the target segmentation training detection system:

[0062] (1) Preparation: Build the GPU operating environment required by the Yolov8 algorithm, install dependent libraries, and convert wafer defect data images into png format.

[0063] (2) Data preparation: Select different types of images including black spots, halos, smears, scratches, etc. from the collected images. Create labels for the training set, annotate the dataset with labelme, manually annotate the selected defect images, mark the type and location of the defects, and generate a json file. Since yolo only supports txt file input format, run the json format to convert it into txt format.

[0064] (3) Data partitioning: Randomly divide the training set and test set: The labeled data set is randomly divided into two parts, one for training the model and the other for verifying the effect of the model.

[0065] (4) Training model: Build and improve the segmentation network, and modify the YOLOv8n-seg network architecture according to actual needs to make it more suitable for crack detection tasks. Set training parameters. Before training the model, you need to set some key parameters, such as learning rate, batch size, number of iterations, etc. Use the labeled training set to train the improved YOLOv8n-seg network. During the training process, record the changes in the loss value and draw the loss function attenuation curve to observe the model convergence.

[0066] (5) Adjust parameters and test training results: After the model training is completed, use the test set to test the performance of the model. If the loss value no longer decreases significantly or reaches the predetermined convergence condition, the model is considered to have converged; otherwise, return and continue to adjust the training parameters and retrain.

[0067] (6) Evaluate the performance of the wafer defect detection model: Count the defect detection results in the test set and calculate the detection accuracy to evaluate the wafer defect detection accuracy. From data collection to model training and evaluation, an efficient and accurate defect detection model is obtained.

[0068] This invention is suitable for handling small object detection problems in high-resolution images because it can provide better contextual information, thereby improving the accuracy and efficiency of detection. It improves the accuracy and efficiency of detection by segmenting the image and processing the slices one by one. The following are the detailed steps for the design process of the defect image reasoning stage:

[0069] (1) Small image wafer defect detection: The 4512*4512 high-resolution image is segmented into 64 640*640 small images. The yolov8n-seg defect detection model is inferred for each small image independently. Each small image is inferred by the model to identify the objects in it. By breaking down the image into manageable parts, the model can focus on local details, thereby improving the detection effect of small objects.

[0070] (2) Model inference acceleration: Using TensorRT for acceleration can significantly improve the speed and efficiency of defect detection. TensorRT is a high-performance deep learning inference optimizer developed by NVIDIA that can optimize trained models and generate efficient inference engines. In defect detection tasks, the model optimized by TensorRT can batch process multiple sets of data at one time, making full use of the parallel computing power of the GPU and significantly reducing the inference time. This not only improves the detection speed, but also enables faster response in real-time application scenarios, ensuring efficient quality control.

[0071] (3) Result merging: Merging the detection results of all slices back into the original image involves locating and combining the information from each slice to reconstruct the complete detection result.

[0072] (4) Post-processing: Finally, a complete inference process is performed again and post-processing is performed to obtain the final detection results.

[0073] The entire process of TensorRT model reasoning acceleration can be divided into three steps, namely, model parsing (Parser),

[0074] Engine optimization and execution. Model reasoning accelerates the design process:

[0075] Model parsing (Parser): During the model parsing phase, TensorRT reads and parses pre-trained models from different frameworks, converts them into internal representations, and ensures that the model structure and parameters meet the requirements.

[0076] Engine optimization: In the Engine optimization stage, TensorRT performs layer fusion, precision optimization, memory optimization, and parallel computing on the parsed model to generate an efficient inference engine and maximize performance.

[0077] Execution: In the execution phase, the optimized Engine is loaded, input data is prepared, forward propagation calculations are performed, inference results are obtained and processed, and the actual inference task is completed.

[0078] First, the input is a pre-trained FP32 model and network. The model is input into TensorRT through parsers and other methods. TensorRT can generate a serialization, that is, stream the input to memory or files to form an optimized engine. The optimized engine can be serialized to memory (buffer) or files (file). When reading, it needs to be deserialized and turned into an engine for use. Then, when executing, a context is created, which is mainly to allocate pre-resources. The engine plus context can do inference.

[0079] The present invention is based on the nanoimprint defect detection system accelerated by tensorrt inference GPU, which improves the wafer defect detection speed without losing accuracy. The present invention adopts data enhancement method to enhance the model's ability to handle complex problems, reduce the computational cost without losing accuracy, monitor product quality in real time and intelligently guide the process flow, and can cope with more diverse scenarios and tasks. The present invention trains a wafer defect model based on the YOLOv8n-seg target segmentation detection model, uses a large amount of original defect data and enhanced data images, and finely divides the defect types to improve the model detection accuracy and fine granularity. The present invention slices the original image 4512*4512, uses a 640*640 size for data set training, and uses an 8*8 image cropping method to obtain 64 640*640 images corresponding to each original image as a data set for model training, and reconstructs the original image by combining small defect detection images, which is conducive to improving the detection accuracy of small objects.

[0080] The present invention is fully described for a clearer disclosure, and the prior art is not listed one by one.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents; it is obvious for those skilled in the art to combine multiple technical solutions of the present invention. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention. The technical contents not described in detail in the present invention are all known technologies.

Claims

1. A nanoimprint defect detection method based on deep learning technology, characterized in that: The method comprises the following steps: Step 1: First, control the brightness of the light source through the light source controller, and use the camera to collect wafer images with defect detection sample data; then, the acquisition card transmits the wafer image to the computer; build the design process of the target segmentation training detection system; Step 2: Visual image analysis software to pre-process the wafer image; Among them, during image analysis, first, for the wafer image of the original defect with a set size of A*A, defined as the original image, the algorithm code is used to segment it into each model training image of the set size a*a as a small image sample (whether it is correct); Step 3: Use the YOLOv8n-seg target segmentation algorithm to detect defects in each small image sample, use Labelme to annotate the data, and convert the annotation results from JSON format to txt format as a sample data set for model training; Step 4: randomly divide the sample data set into training set and test set, train and evaluate the performance of the model; the detection results are instance segmented and post-processed, and finally the defect detection results of each slice are output; Step 5: merge all the segmented slice image results to form a complete large image, that is, the original image; process and reason on the large image to obtain the complete defect detection results of each original image; Step six: Analyze and count the defects of all original images, store the output data analysis results, and make targeted corrections and guide the process operation flow based on the defect statistics.

2. The nanoimprint defect detection method based on deep learning technology according to claim 1, characterized in that: The process operation flow includes image acquisition, image processing, image analysis, model training and result output.

3. The nanoimprint defect detection method based on deep learning technology according to claim 1, characterized in that: In step 5, when generating the detection list for the entire original image, overlap the slices in the last row and the slices in the last column to ensure that the object can be detected in at least one slice; When merging the detection results of slices, remove the objects with duplicate detections.

4. The nanoimprint defect detection method based on deep learning technology according to claim 1, characterized in that: In step 1, the design process of the detection system for object segmentation training is as follows; S1.1, prepare; build the GPU operating environment required by the Yolov8 algorithm, install dependent libraries, and convert wafer defect data images into png format; S1.2, data preparation; select defect images from the collected original images, create labels for the training set, annotate the data set with labelme, manually annotate the selected defect images, mark the type and location of the defects, generate a json file, and run the json format conversion to txt format; Defects include black spots, halos, smears, and scratches; S1.3, data division; random division of training set and test set: randomly divide the labeled data set into two parts, one for training the model and the other for verifying the effect of the model; S1.4, training model; first, build and improve the segmentation network, modify the YOLOv8n-seg network architecture according to actual needs, so that the YOLOv8n-seg network architecture is suitable for crack detection tasks; then, set the training parameters, and set the key parameters before training the model; secondly, use the labeled training set to train the improved YOLOv8n-seg network. During the training process, record the changes in the loss value and draw the loss function attenuation curve to observe the model convergence; Key parameters include learning rate, batch size, and number of iterations; S1.5, adjust parameters and test training results: After the model training is completed, use the test set to test the performance of the model; if the loss value stops decreasing or reaches the predetermined convergence condition, the model is considered to have converged; otherwise, return to S1.1, continue to adjust the training parameters and retrain; S1.6, evaluate the performance of the wafer defect detection model; collect statistics on the defect detection results in the test set of S1.3, calculate the detection accuracy, evaluate the wafer defect detection accuracy, and obtain the defect detection model from data collection to model training and evaluation.

5. The nanoimprint defect detection method based on deep learning technology according to claim 1, characterized in that: In step 2, the design process for the defect image reasoning stage is as follows; S2.1, small picture wafer defect detection; first, the A*A image is segmented into several a*a small pictures, and the yolov8n-seg defect detection model is inferred for each small picture independently. Each small picture is inferred by the model to identify the objects in the small picture; then, the image is decomposed into manageable parts; S2.2, model reasoning acceleration; Use TensorRT for acceleration; Among them, the entire process of TensorRT model reasoning acceleration is divided into three steps, namely model parsing, Engine optimization and execution; S2.3, result merging: merging the detection results of all slices back into the original image, which involves locating and combining the information from each slice to reconstruct the complete detection result; S2.4, post-processing, finally, a complete inference process is performed to obtain the final detection result.

6. A nanoimprint defect detection system based on deep learning technology, characterized in that: The system constructs and executes the method described in claim 1.

7. The nanoimprint defect detection system based on deep learning technology according to claim 6, characterized in that: The system includes a computer, as data processing, installed with visual image analysis software; A light source controller, used to control the brightness of the light source; A camera for collecting wafer images with defect inspection sample data; Capture card, which transfers wafer images to a computer; The computer is electrically connected to the light source controller, the camera and the acquisition card.

8. The nanoimprint defect detection system based on deep learning technology according to claim 6, characterized in that: Visual image analysis software to pre-process wafer images; pre-processing includes grayscale, noise removal, and data image enhancement.

Images