Small target defect detection method and device, medium and product

Abstract

The embodiment of the invention relates to the field of semiconductor defect detection, and discloses a small target defect detection method and device, a medium and a product. The method comprises the following steps: collecting a picture data set with defect; performing coarse-grained and fine-grained classification on the defect of the picture data set; based on the picture data set, a training picture is segmented according to a preset size; cutting the training picture into a plurality of training sub-pictures; constructing a deep neural network target detection model, and training the deep neural network target detection model through the training sub-graph; and detecting the to-be-detected sub-image through the trained deep neural network target detection model, and splicing and mapping a detection result back to the to-be-detected image. The problem that the small target defect detection effect of the photoetching mask plate is poor can be at least solved.

Description

Technical Field

[0001] The present application relates to the field of semiconductor defect detection, and in particular to a small target defect detection method, equipment, medium and product. Background Art

[0002] Photolithography masks play a vital role in integrated circuit manufacturing and microfabrication. A mask is a high-precision mask used to transfer circuit patterns onto semiconductor wafers. The process of making photolithography masks is complex and precise, and usually includes the following steps: substrate preparation, photoresist coating, exposure, development, etching, photoresist removal, inspection and repair, cleaning and packaging, etc. During these complex processes, the photolithography mask will produce various defects.

[0003] At present, in the above-mentioned processing steps, the detection of these defects usually adopts the methods of manual naked eye judgment and traditional computer vision detection. Manual naked eye judgment is easy to fatigue and inefficient, while traditional computer vision detection methods are not effective in detecting defects on photolithography masks, especially for small-scale defects, the detection accuracy is poor. Traditional computer vision methods mainly rely on predefined rules and thresholds, such as edge detection, morphological operations, etc. These algorithms work well when processing simple and regular patterns, but when processing complex and irregular patterns and tiny or atypical defects, the detection accuracy is obviously insufficient, and it is easy to miss or misdetect.

[0004] With the continuous development of integrated circuit manufacturing technology, higher requirements are placed on the accuracy and efficiency of photolithography mask defect detection. Traditional methods can no longer meet these requirements, so a new detection method is urgently needed to improve the accuracy and efficiency of defect detection. Summary of the invention

[0005] One purpose of the present application is to provide a small target defect detection method, device, medium and product, at least to solve the problem of poor small target defect detection effect of photolithography mask.

[0006] To achieve the above objectives, some embodiments of the present application provide the following aspects: In the first aspect, some embodiments of the present application also provide a small target defect detection method. The method includes collecting a data set of images with defects; classifying the defects of the image data set in a coarse-grained and fine-grained manner, and marking the category and location information of the defects; based on the image data set, segmenting the training images according to a predetermined size; cutting the training images into multiple training sub-images; constructing a deep neural network target detection model, and training the deep neural network target detection model through the training sub-images; segmenting the image to be detected according to a predetermined size, and cutting the image to be detected into multiple sub-images to be detected; detecting the sub-image to be detected through the trained deep neural network target detection model, and splicing and mapping the detection results back to the image to be detected.

[0007] In a second aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0008] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0009] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the method described above when executed by a processor.

[0010] Compared with the related art, the solution provided in the embodiment of the present application adopts a data processing method based on a cut-image puzzle, optimized deep neural network model training, and continuous model iteration optimization, which significantly improves the accuracy and efficiency of photolithography mask defect detection, especially the detection effect of small target defects. This method not only improves the accuracy and robustness of detection, reduces manual intervention, but also improves the overall efficiency of the mask factory process and significantly reduces production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0012] Figure 1 A schematic diagram of a process for detecting small target defects according to an embodiment of the present application; Figure 2A schematic diagram of a process of another small target defect detection method provided according to an embodiment of the present application; Figure 3 A schematic diagram of the effect of a defective training picture provided according to an embodiment of the present application; Figure 4 A schematic diagram of the effect of a labeled training picture provided according to an embodiment of the present application; Figure 5 A schematic diagram of the effect of splitting a picture provided according to an embodiment of the present application; Figure 6 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] First embodiment The first embodiment of the present application relates to a small target defect detection method. Figure 1 As shown, the method may include the following steps: S101, collect image datasets with defects; perform coarse-grained and fine-grained classification on the defects in the image datasets, and mark the category and location information of the defects. By collecting a large number of image datasets with defects, the diversity and richness of the training data are ensured, which helps to build a deep neural network model with good generalization ability and improve the model's ability to recognize different types of defects. Detailed classification and labeling of defects in the image dataset not only provides rich training data, but also helps the model better learn and distinguish defects of different types and sizes, and enhances the detection precision and accuracy of the model.

[0015] S102, based on the image data set, the training image is segmented according to a predetermined size; the training image is segmented into multiple training sub-images. By segmenting the training image into multiple sub-images, the number of training samples can be increased and the training effect of the model can be improved. At the same time, by ensuring the overlapping parts between the sub-images, the integrity of the defect is retained, avoiding information loss caused by segmentation, and improving the detection effect of the model on small target defects.

[0016] S103, construct a deep neural network target detection model, and train the deep neural network target detection model through the training subgraph. Construct and train the deep neural network target detection model so that it can learn and extract features from a large number of training subgraphs, improve the detection capability and accuracy of defects. Further improve the model performance by optimizing the hyperparameters (such as learning rate and batch size) during the training process.

[0017] S104, divide the image to be detected according to a predetermined size, and cut the image to be detected into multiple sub-images to be detected; detect the sub-images to be detected through the trained deep neural network target detection model, and splice and map the detection results back to the image to be detected. Dividing the image to be detected allows the model to process the sub-images one by one, so as to accurately identify the defects in each sub-image, avoiding the problems of information loss and reduced detection accuracy that may be caused by direct detection of the entire image. The sub-images to be detected are detected by the trained model to ensure the efficiency and accuracy of the detection. The detection results are spliced ​​back to the original image to intuitively show the position and category of the defect in the original image, which is convenient for subsequent processing and analysis.

[0018] It is not difficult to find that compared with the related art, the solution provided in the embodiment of the present application adopts a deep neural network model for the defect features of the lithography mask, and is trained on a large-scale data set. It has strong generalization ability and can accurately identify complex and tiny defects. The deep neural network model performs multiple downsampling on the image to be detected, resulting in missing information on small target defects and poor detection effect on small target defects. A small target defect detection method based on cutting puzzles is adopted to improve the detection effect of small target defects. The accuracy of the model is improved by fine-grained division of the target.

[0019] Second embodiment The second embodiment of the present application relates to a small target defect detection method. The second implementation is an improvement based on the first embodiment. Figure 2 As shown, the specific improvements are: Training data collection and annotation: collect a large number of image datasets with defects in the photolithography mask production process, such as Figure 3 These image datasets cover defects of various types and sizes, ensuring the diversity and comprehensiveness of the training data.

[0020] Data annotation,First, the collected defect image dataset is coarsely classified into three categories: defect1, defect2, and defect3. Then each defect is finely classified and further subdivided into defect1_1, defect1_2, defect2_1, defect2_2, etc. The labeled defect images are as follows Figure 4 As shown. Using a semi-supervised learning method, combined with manually labeled data and unlabeled data, the defects in each image are annotated in detail, including the subcategory of the defect and specific location information. The major categories of defects include edge defects, surface particle defects, pattern distortion defects, etc.; the minor categories of defects with fine-grained classification include cracks in edge defects, dents in edge defects, large particles in surface particle defects, small particles in surface particle defects, line skewness in pattern distortion defects, and position offset in pattern distortion defects.

[0021] Image segmentation: The training images are segmented according to the predetermined size. Each image can be cut into multiple sub-images, such as Figure 5 As shown. To ensure the integrity of the defect, any two sub-graphs have overlapping parts of a certain size in the horizontal and vertical directions. By setting the overlapping parts in the horizontal and vertical directions, it can be ensured that even if the defect is located on the cutting boundary, it can still be fully captured in at least one sub-graph, thereby avoiding the loss of defect information caused by cutting. The increase in the number of sub-graphs after segmentation provides more training samples, which helps to improve the training effect and generalization ability of the deep neural network model. The segmented sub-graphs contain defects of different positions and sizes, which helps the model learn more representative features and improve the detection ability of various types of defects.

[0022] Build a deep neural network target detection model, use subgraphs for model training, perform hyperparameter search for parameters such as learning rate and batchsize, train them one by one, and finally select the parameters corresponding to the model with the highest accuracy as the search result, and save the model with the highest accuracy. Select a deep neural network model architecture suitable for defect detection tasks, such as YOLO (YouOnly Look Once), Faster R-CNN (Region Convolutional Neural Network), SSD (SingleShot MultiBox Detector), etc.

[0023] First, initialize the model parameters. You can use the pre-trained model parameters for initialization. This way, you can use the features learned by the existing model on a large data set to speed up the training convergence and improve the detection accuracy. Use the segmented sub-graphs and their corresponding labels (defect categories and location information) as training data, and perform data enhancement, such as random cropping, horizontal flipping, color jittering, etc., to further increase the diversity of training data and enhance the robustness of the model. Input the sub-graph data into the deep neural network model and perform forward propagation to calculate the prediction results. Calculate the loss function. Commonly used loss functions include classification loss (such as cross entropy loss) and positioning loss (such as mean square error loss). Select and optimize them based on the actual situation, perform back propagation, calculate the gradient and update the model parameters. Determine the range of hyperparameters that need to be optimized, including learning rate, batch size, weight decay, optimizer type, etc.; hyperparameter search can be performed using methods such as grid search, random search, or Bayesian optimization, and random sampling can be performed from a predefined hyperparameter range. Multiple training and validation can be performed to find the best hyperparameter combination; multiple trainings are performed, each with a different hyperparameter combination. After each training, the model performance is evaluated on the validation set, and the detection accuracy on the validation set is recorded; the validation results corresponding to all hyperparameter combinations are compared, and the hyperparameter combination with the highest accuracy on the validation set is selected as the final result; the model is retrained using the optimal hyperparameter combination found, and the model parameters with the highest accuracy on the validation set are saved during the training process. The trained model parameters and hyperparameter settings are saved for subsequent detection tasks.

[0024] Evaluate model performance on validation sets and test sets. Common evaluation indicators include average precision, recall rate, precision, etc. Compare the changes in indicators before and after training to evaluate the effect of hyperparameter optimization. After training is completed and the verification effect is good, deploy the model to the actual detection system for real-time defect detection.

[0025] For the predicted images, the same method as the training images is used to cut the images to obtain sub-images. Each sub-image calls the trained neural network target detection model and outputs the defect label category. The final result is spliced ​​and mapped to the original image to obtain the detection image, which includes the category and location information of the defect. When splicing, it is determined whether there is a defect. If there is a defect, the defect category of the image is predicted; if there is no defect, the image is predicted to be free of defects. Based on the newly added images with defects, the model is regularly iterated and optimized to continuously improve the accuracy of the model.

[0026] Collect newly added detection image data with defects. The detection images do not need to be labeled and can be directly input into the neural network target detection model as training images. Retrain the deep neural network target detection model based on the updated image dataset. Perform hyperparameter search on the new dataset to optimize the model parameters and ensure the detection accuracy of the model on the new data. Incremental learning or online learning can be used to gradually update the model parameters to avoid the high computational cost of full training. Regularly perform iterative model training and verification to ensure that the model always maintains high detection accuracy and robustness, and evaluate the performance of the new model on the validation set and test set. Common evaluation indicators include mean average precision, recall rate, precision rate, etc. If the new model is better than the old model in all evaluation indicators, the new model will be deployed in the actual detection system.

[0027] It is not difficult to find that compared with the related art, in the solution provided in the embodiment of the present application, by dividing the picture to be predicted into multiple sub-pictures and detecting them one by one using the trained deep neural network model, high efficiency and accuracy are ensured; by splicing the sub-picture detection results back into the original picture, intuitive defect location and category information is provided, which is convenient for subsequent processing and analysis; by regularly iterating and optimizing the model, the detection accuracy is continuously improved to ensure the reliability and adaptability of the model in practical applications.

[0028] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0029] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 6 An exemplary structural diagram of the electronic device is disclosed. Figure 6As shown, the electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0030] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0031] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0032] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0033] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0034] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0035] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0037] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0038] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0039] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. For example, an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device may be used to implement the embodiments. In some embodiments, the software program of the present application may be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) may be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application may be implemented by hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0040] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center to another website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0041] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0042] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0043] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A small target defect detection method, characterized in that: The method comprises: Collect a dataset of images with defects; classify the defects in the dataset in a coarse-grained and fine-grained manner, and mark the category and location information of the defects; Based on the image data set, the training image is segmented according to a predetermined size; the training image is segmented into a plurality of training sub-images; Constructing a deep neural network target detection model, and training the deep neural network target detection model through the training subgraph; The image to be detected is divided into a predetermined size, and the image to be detected is cut into a plurality of sub-images to be detected; the sub-images to be detected are detected by the trained deep neural network target detection model, and the detection results are spliced ​​and mapped back to the image to be detected; The step of cutting the training image into multiple training sub-images and the step of cutting the image to be detected into multiple sub-images to be detected include making any two of the training sub-images overlap to a certain extent in the horizontal and vertical directions; and making any two of the sub-images to be detected overlap to a certain extent in the horizontal and vertical directions.

2. The method according to claim 1, characterized in that The coarse-grained and fine-grained classification of defects of the image dataset includes: Defects are divided into multiple coarse-grained categories, and each coarse-grained category is subdivided into multiple fine-grained subcategories.

3. The method according to claim 2, characterized in that The category and location information of the marked defect include: The image dataset is labeled by using a semi-supervised learning method, combining manually labeled data and unlabeled data.

4. The method according to claim 1, characterized in that: The training of the deep neural network target detection model by the training subgraph includes: A hyperparameter search is performed on the learning rate and batch size parameters in the model training, and the model parameters with the highest accuracy are selected as the operating parameters of the deep neural network target detection model.

5. The method according to any one of claims 1 to 4, characterized in that: Mapping the detection results back to the image to be detected includes: The detection results of each of the sub-images to be detected are concatenated and mapped to obtain a detection image, where the detection image includes the category and location information of the defect.

6. The method according to claim 5, characterized in that The method further comprises: Based on the detection image, the deep neural network target detection model is iterated and updated.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 6.

8. A computer readable medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images