Defect detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a defect detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a transmission image and a reflection image; respectively inputting the transmission image and the reflection image into an image contour extraction model to obtain contour images of the transmission image and the reflection image; wherein the image contour extraction model is trained by using a plurality of transmission images, reflection images and contour diagrams of the transmission images and the reflection images in advance; matching the generated contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points; matching the transmission image and the reflection image based on the contour matching coordinate points; pixel values of all pixel points on the transmission image and the reflection image are calculated and added to obtain a composite image; and comparing the pixel value of each pixel point on the synthesized image with a threshold value to determine a defect pixel point and a position coordinate thereof.

Description

Technical Field

[0001] The present application relates to the technical field of semiconductor detection, and in particular to a defect detection method and device, an electronic device, and a storage medium. Background Art

[0002] A photomask is an indispensable component in the lithography process, and it carries a designed pattern. With the cooperation of a lithography machine and photoresist, the designed pattern on the photomask will be transferred to the photoresist on the substrate through processes such as exposure and development for image replication, thereby realizing the mass production of chips. During the production process of the photomask, pollutants such as particles, metal ions, chemical substances, and static electricity in the air and equipment will fall on the surface of the photomask, resulting in defects and the like, thus affecting the chip yield, performance, and reliability. Therefore, the defect detection link of the photomask is crucial.

[0003] Since the light intensity in the defect area becomes smaller, it will show a lower gray value in the image, and there is a complementary characteristic between the information of the transmission image and the reflection image. Therefore, one current method for defect detection of the photomask is mainly to collect the transmission light image and the reflection light image at the same position of the photomask, and then align the transmission light image and the reflection light image to complement the information of the same pixel points on the transmission light image and the reflection light image, and screen out the defects appearing on the photomask based on the complementary information.

[0004] However, due to the small feature size and the influence of light scattering, the resolution of the features on the image is low, so the current technology cannot ensure the accurate alignment of the transmission light image and the reflection light image, and thus cannot ensure the accuracy of defect detection. Summary of the Invention

[0005] Based on the above deficiencies of the prior art, the present application provides a defect detection method and device, an electronic device, and a storage medium to solve the problem that the prior art cannot ensure the accuracy of defect detection.

[0006] To achieve the above object, the present application provides the following technical solutions:

[0007] The first aspect of the present application provides a defect detection method, including:

[0008] Obtain a transmission image and a reflection image;

[0009] Input the transmission image and the reflection image into a pre-trained image contour extraction model respectively to obtain a contour image of the transmission image and a contour image of the reflection image; wherein, the image contour extraction model is pre-trained with multiple transmission images and their contour maps and multiple reflection images and their contour maps;

[0010] Match the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points;

[0011] Based on the contour matching coordinate points, match the transmission image and the reflection image;

[0012] Calculate the sum of the pixel values of each pixel point on the transmission image and the pixel values of the matching pixel points on the reflection image to obtain a composite image;

[0013] By comparing the pixel values of each pixel point on the composite image with a threshold, determine the defective pixel points and their position coordinates.

[0014] Optionally, in the above defect detection method, the obtaining of the transmission image and the reflection image includes:

[0015] Use a time delay integration camera to obtain the transmission image and the reflection image.

[0016] Optionally, in the above defect detection method, the training method of the image contour extraction model includes:

[0017] Obtain multiple transmission images and multiple reflection images as original images;

[0018] Preprocess each of the original images to obtain multiple sample images;

[0019] Form the sample images into a sample image sequence;

[0020] Respectively perform contour annotation on each of the sample images in the sample image sequence to obtain the annotated contour images corresponding to each of the sample images;

[0021] Use the sample image sequence and the annotated contour images to iteratively train the image contour extraction model until the loss value of the image contour extraction model converges to obtain the trained image contour extraction model.

[0022] Optionally, in the above defect detection method, the preprocessing of each of the original images to obtain multiple sample images includes:

[0023] Use a preset data augmentation method to perform data augmentation processing on each of the original images to obtain multiple augmented images, and use each of the augmented images and each of the original images as the sample images.

[0024] Optionally, in the above defect detection method, the respectively performing contour annotation on each of the sample images in the sample image sequence to obtain the annotated contour images corresponding to each of the sample images includes:

[0025] Use an image annotation tool to mark the contours of each of the sample images in the sample image sequence and set annotation names, and generate a json file containing the marked contour images with the same name as the sample images.

[0026] Optionally, in the above defect detection method, the image contour extraction model includes a generator for generating contour maps of the input transmission image and reflection image, and a discriminator for discriminating the accuracy of the contour maps generated by the generator.

[0027] Optionally, in the above defect detection method, the generator includes six downsampling layers and six upsampling layers, and the last downsampling layer is connected to the first upsampling layer. There is a skip connection chain between the first five downsampling layers and the last five upsampling layers. The last upsampling layer is connected to a fully connected layer and outputs a contour map; among them, each sampling layer performs feature extraction through three convolutions, and there is a skip connection chain between the second and third convolutions of the first and third layers;

[0028] The discriminator consists of eight convolutional layers; among them, the mean of the matrix output by the last convolutional layer is the discrimination result.

[0029] Optionally, in the above defect detection method, the iterative training of the image contour extraction model using the sample image sequence and the annotated contour images includes:

[0030] Use the sample image sequence and the annotated contour images to perform one round of alternating training on the generator and the discriminator in the currently optimal image contour extraction model determined by the previous round of alternating training;

[0031] Use the total loss function of the image contour extraction model to calculate the loss value of this round of alternating training;

[0032] Judge whether the loss value of this round of alternating training converges;

[0033] If it is judged that the loss value of this round of alternating training converges, then use the image contour extraction model obtained from this round of alternating training as the trained image contour extraction model;

[0034] If it is judged that the loss value of this round of alternating training does not converge, then compare whether the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training; among them, the similarity corresponding to one round of alternating training is the similarity between the contour map generated by the image contour extraction model obtained after training and the annotated contour map;

[0035] If the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the previous round of alternating training, then use the image contour extraction model obtained from the current round of alternating training as the current optimal image contour extraction model;

[0036] If the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the previous round of alternating training, then use the image contour extraction model obtained from the previous round of alternating training as the current optimal image contour extraction model;

[0037] After determining the current optimal image contour extraction model, return to execute using each of the sample images and their corresponding annotated contour images to perform one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined by the previous round of alternating training.

[0038] Optionally, in the above defect detection method, the step of using the sample image sequence and the annotated contour image to perform one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined by the previous round of alternating training includes:

[0039] According to a preset training order, sequentially use the sample image sequence and the annotated contour image to perform alternating training on the generator and the discriminator in the current optimal image contour extraction model determined by the previous round of alternating training; wherein, the preset training order is to train the discriminator N times and then train the generator once;

[0040] For each training, calculate the current loss value of the current training network through the loss function corresponding to the current training network; wherein, the current training network is the generator or the discriminator of this training;

[0041] Judge whether the current total number of training times is an integer multiple of the preset number of times;

[0042] If the current total number of training times is an integer multiple of the preset number of times, then calculate the current texture loss value through the texture error function; wherein, the texture error function represents the texture error between the annotated contour image corresponding to the sample image and the contour map generated by the generator;

[0043] Add the current loss value of the current training network and the current texture loss value to obtain the current total loss value;

[0044] If the current total number of training times is not an integer multiple of the preset number of times, then determine the current loss value of the current training network as the current total loss value;

[0045] Adjust the parameters of the current training network based on the current total loss value.

[0046] The second aspect of the present application provides a defect detection device, including:

[0047] An image acquisition unit for acquiring a transmission image and a reflection image;

[0048] A contour extraction unit for respectively inputting the transmission image and the reflection image into a pre-trained image contour extraction model to obtain a contour image of the transmission image and a contour image of the reflection image; wherein, the image contour extraction model is pre-trained using multiple transmission images and their contour maps and multiple reflection images and their contour maps;

[0049] A contour matching unit for matching the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points;

[0050] An image alignment unit for aligning the transmission image and the reflection image based on the contour matching coordinate points;

[0051] An image combination unit for calculating the sum of the pixel values of each pixel point on the transmission image and the pixel values of the matching pixel points on the reflection image to obtain a composite image;

[0052] A defect determination unit for determining defect pixel points and their position coordinates by comparing the pixel values of each pixel point on the composite image with a threshold.

[0053] Optionally, in the above-mentioned defect detection device, the image acquisition unit includes:

[0054] An image acquisition subunit for acquiring a transmission image and a reflection image using a time delay integration camera.

[0055] Optionally, in the above-mentioned defect detection device, it further includes:

[0056] A sample acquisition unit for acquiring multiple transmission images and multiple reflection images as original images;

[0057] A preprocessing unit for preprocessing each of the original images to obtain multiple sample images;

[0058] An image combination unit for forming the sample images into a sample image sequence;

[0059] A labeling unit for respectively contour-labeling each of the sample images in the sample image sequence to obtain a labeled contour image corresponding to each of the sample images;

[0060] A training unit for iteratively training the image contour extraction model by using the sample image sequence and the labeled contour images until the loss value of the image contour extraction model converges, so as to obtain the trained image contour extraction model.

[0061] Optionally, in the above defect detection device, the preprocessing unit includes:

[0062] A preprocessing subunit for performing data augmentation processing on each of the original images by using a preset data augmentation method to obtain multiple augmented images, and using each of the augmented images and each of the original images as the sample images.

[0063] Optionally, in the above defect detection device, the labeling unit includes:

[0064] A labeling subunit for respectively labeling the contours of each of the sample images in the sample image sequence through an image labeling tool and setting a label name, and generating a json file containing the labeled contour images and having the same name as the sample images.

[0065] Optionally, in the above defect detection device, the image contour extraction model includes a generator for generating contour maps of the input transmission image and reflection image, and a discriminator for discriminating the accuracy of the contour maps generated by the generator.

[0066] Optionally, in the above defect detection device, the generator includes six downsampling layers and six upsampling layers, and the last downsampling layer is connected to the first upsampling layer. There is a skip connection chain between the first five downsampling layers and the last five upsampling layers. The last upsampling layer is connected to a fully connected layer and outputs a contour map; wherein, each sampling layer performs feature extraction through three convolutions, and there is a skip connection chain between the second convolution and the third convolution of the first layer and the third layer;

[0067] The discriminator is composed of eight convolutional layers; wherein, the mean value of the matrix output by the last convolutional layer is the discrimination result.

[0068] Optionally, in the above defect detection device, the training unit includes:

[0069] An alternating training unit for performing one round of alternating training on the generator and the discriminator in the currently optimal image contour extraction model determined in the previous round of alternating training by using the sample image sequence and the labeled contour images;

[0070] A total loss calculation unit for calculating the loss value of this round of alternating training by using the total loss function of the image contour extraction model.

[0071] The first judgment unit is used to judge whether the loss value of the current round of alternating training converges;

[0072] The training end unit is used to, when it is judged that the loss value of the current round of alternating training converges, take the image contour extraction model obtained by the current round of alternating training as the trained image contour extraction model;

[0073] The comparison unit is used to, when it is judged that the loss value of the current round of alternating training does not converge, compare whether the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the previous round of alternating training; wherein, the similarity corresponding to one round of alternating training is the similarity between the contour map generated by the image contour extraction model obtained after training and the labeled contour map;

[0074] The first determination unit is used to, when the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the previous round of alternating training, take the image contour extraction model obtained by the current round of alternating training as the currently optimal image contour extraction model;

[0075] The second determination unit is used to, when the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the previous round of alternating training, take the image contour extraction model obtained by the previous round of alternating training as the currently optimal image contour extraction model; wherein, after determining the currently optimal image contour extraction model, return to the alternating training unit.

[0076] Optionally, in the above defect detection device, the alternating training unit includes:

[0077] The sequential training unit is used to sequentially perform alternating training on the generator and the discriminator in the currently optimal image contour extraction model determined by the previous round of alternating training by using the sample image sequence and the labeled contour image according to a preset training order; wherein, the preset training order is to train the discriminator N times and then train the generator once;

[0078] The network loss calculation unit is used to, every time a training is performed, calculate the current loss value of the current training network through the loss function corresponding to the current training network; wherein, the current training network is the generator or the discriminator of this training;

[0079] The second judgment unit is used to judge whether the current total number of training times is an integer multiple of the preset number of times;

[0080] The texture loss calculation unit is used to, when the current total number of training times is an integer multiple of the preset number of times, calculate the current texture loss value through the texture error function; wherein, the texture error function represents the texture error between the labeled contour image corresponding to the sample image and the contour map generated by the generator;

[0081] A current loss calculation unit, configured to add the current loss value of the current training network to the current texture loss value to obtain a current total loss value;

[0082] A loss determination unit, configured to determine the current loss value of the current training network as the current total loss value when the current total number of training times is not an integer multiple of a preset number of times;

[0083] Adjust the parameters of the current training network based on the current total loss value.

[0084] A third aspect of the present application provides an electronic device, including:

[0085] A memory and a processor;

[0086] Wherein, the memory is used to store a program;

[0087] The processor is configured to execute the program, and when the program is executed, it is specifically configured to implement the defect detection method described in any one of the above.

[0088] A fourth aspect of the present application provides a computer storage medium, configured to store a computer program, and when the computer program is executed by a processor, it is used to implement the defect detection method described in any one of the above.

[0089] The present application provides a defect detection method. First, a plurality of transmission images and their contour maps, as well as a plurality of reflection images and their contour maps are used to train a well-trained image contour extraction model, which can generate accurate contour maps of reflection images and transmission images. Therefore, when defect detection is required, first obtain the transmission image and the reflection image, and then input the transmission image and the reflection image into the pre-trained image contour extraction model respectively to obtain the contour image of the transmission image and the contour image of the reflection image, so as to accurately extract the contour images of the transmission image and the reflection image through the pre-trained model. Then, match the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points, so that the transmission image and the reflection image can be matched based on the contour matching coordinate points. Since the contour maps are accurately generated, the transmission image and the reflection image can be accurately aligned. And because the information of the emission image and the emission image is complementary, and the light intensity of the defect area is relatively small, by calculating the sum of the pixel values of each pixel point on the transmission image and the pixel values of the matching pixel points on the reflection image to obtain a composite image, and by comparing the pixel values of each pixel point on the composite image with a threshold, the defect pixel points and their position coordinates are determined, so that the defects can be accurately detected. Description of the Drawings

[0090] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0091] Figure 1 It is a flowchart of a defect detection method provided by an embodiment of the present application;

[0092] Figure 2 It is a flowchart of a training method for an image contour extraction model provided by an embodiment of the present application;

[0093] Figure 3 It is a flowchart of a method for iteratively training an image contour extraction model using a sample image sequence and an annotated contour image provided by an embodiment of the present application;

[0094] Figure 4 It is a flowchart of a method for alternately training an image contour extraction model provided by an embodiment of the present application;

[0095] Figure 5 It is a schematic diagram of the architecture of a defect detection device provided by an embodiment of the present application;

[0096] Figure 6 It is a schematic diagram of the architecture of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0097] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0098] In this application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0099] An embodiment of this application provides a defect detection method to solve the problem that the prior art cannot guarantee the accuracy of defect detection.

[0100] First of all, it should be noted that the following embodiments of this application are mainly described by taking the defect detection of a reticle as an example. However, the defect detection method provided by the embodiments of this application is not limited to the defect detection of a reticle, and can also be applied to the detection of other objects with the same characteristics as the reticle.

[0101] An embodiment of this application provides a defect detection method, as Figure 1 shown, specifically including the following steps:

[0102] S101. Obtain a transmission image and a reflection image.

[0103] Specifically, when it is necessary to perform defect detection on a certain object, obtain the transmission image and the reflection image at the same position thereof. Therefore, if defect detection is performed on a reticle, obtain the transmission image and the reflection image at the same position of the reticle.

[0104] Optionally, the transmission image and the reflection image at the same position can be specifically acquired by time delay integration.

[0105] Moreover, in order to improve the quality of the image and further improve the quality of contour extraction, the transmission image and the reflection image are preprocessed first.

[0106] S102. Input the transmission image and the reflection image into a pre-trained image contour extraction model respectively to obtain the contour image of the transmission image and the contour image of the reflection image.

[0107] It should be noted that, in order to accurately align the transmission image and the reflection image, in the embodiments of the present application, the contour matching method is adopted for alignment. Therefore, it is necessary to extract the contours of the transmission image and the reflection image, and then align the transmission image and the reflection image through the contours. Therefore, it is necessary to first extract the contours of the transmission image and the reflection image. Although the existing algorithms such as Canny, Sobel, and Laplace operators can also be used for contour extraction. However, due to the small feature size and the influence of light scattering, the resolution of the image contour is low. Therefore, these methods cannot accurately extract the contours of the transmission image and the reflection image. Therefore, in order to accurately extract the contours of the transmission image and the reflection image, in the embodiments of the present application, an image contour extraction model is pre-trained using multiple transmission images and their contour maps and multiple reflection images and their contour maps, so that it can generate the contour maps of the input transmission image and reflection image. Therefore, when defect detection is required, the currently acquired transmission image and reflection image are respectively input into the pre-trained image contour extraction model, so as to extract the contours of the transmission image and the reflection image through the image contour extraction model.

[0108] Optionally, another embodiment of the present application provides a training method for an image contour extraction model, as Figure 2 shown, including the following steps:

[0109] S201. Obtain multiple transmission images and multiple reflection images as original images.

[0110] Specifically, the transmission images and reflection images at multiple positions of multiple masks can be obtained through a time delay integration camera.

[0111] S202. Preprocess each original image to obtain multiple sample images.

[0112] In order to improve the quality of the original images to facilitate the improvement of the accuracy of contour extraction, and also to expand the sample data, in the embodiments of the present application, after each original image is obtained, it is preprocessed relative to it, and each image obtained after preprocessing is used as a sample image.

[0113] Optionally, a specific implementation manner of step S202 includes:

[0114] Use a preset data augmentation method to perform data augmentation processing on each original image to obtain multiple augmented images, and use each augmented image and each original image as the sample images.

[0115] Among them, the preset data augmentation method may include, but is not limited to, horizontal flipping, vertical flipping, random cropping, rotation, adding noise, etc. Thus, a large amount of training data can be obtained through data augmentation to train the model, and further, the accuracy of the model can be guaranteed.

[0116] S203. Combine each sample image into a sample image sequence.

[0117] S204. Perform contour annotation on each sample image in the sample image sequence respectively to obtain an annotated contour image corresponding to each sample image.

[0118] In order to facilitate subsequent analysis of whether the contour map generated by the generator is accurate, it is necessary to first perform contour annotation on each sample image in the sample image sequence to obtain an accurate contour image of each sample image.

[0119] Optionally, in another embodiment of the present application, a specific implementation manner of step S204 includes:

[0120] Annotate the contour of each sample image in the sample image sequence respectively through an image annotation tool and set an annotation name, and generate a json file that contains the annotated contour image and has the same name as the sample image.

[0121] S205. Use the sample image sequence and the annotated contour image to iteratively train the image contour extraction model until the loss value of the image contour extraction model converges, and obtain a trained image contour extraction model.

[0122] Optionally, in another embodiment of the present application, the image contour extraction model includes a generator for generating a contour map of an input transmission image and a reflection image, and a discriminator for discriminating the accuracy of the contour map generated by the generator.

[0123] Specifically, during the training process, the generator is continuously optimized so that the generated contour map is as consistent as possible with the real annotated contour map, achieving the purpose of "deceiving" the discriminator and making it as difficult as possible for the discriminator to distinguish between the generated contour map and the real annotated contour map. And the discriminator will also be continuously optimized to make its discrimination result more accurate, that is, it can accurately distinguish between the generated contour map and the real annotated contour map. Therefore, through continuous optimization of the generator and the discriminator, finally, the generator can generate an accurate contour map for the input transmission image or reflection image.

[0124] Specifically, during training, the sample image sequence is input into the generator in the image contour extraction model, and the contour images corresponding to each sample image in the sample image sequence are generated by the generator. Then, according to the discrimination result of the discriminator on the contour image generated by the generator and the annotated contour image, the parameters of the generator are updated. The contour image generated by the generator and the annotated contour map are input into the discriminator, and the parameters of the discriminator are continuously updated according to the accuracy of its discrimination result until the loss value of the image contour extraction model converges.

[0125] Optionally, the generator can specifically adopt an improved UNet network, that is, the basic network of the generator is the UNet network. The UNet network includes 5 downsampling layers and 5 upsampling layers. The last downsampling layer is connected to the first upsampling layer, and there is a skip connection chain between the first 4 downsampling layers and the last 4 upsampling layers. The last upsampling layer is connected to the fully connected layer to output the contour extraction result. However, in mask template detection, the types of mask templates are diverse, the shapes are different, most images are dense, and the clarity and pixel size of the images captured at different magnifications are different. Therefore, in order to ensure the generalization and robustness of the model, an additional downsampling layer and upsampling layer are added to the generator module of the GAN neural network in this application. So the generator includes six downsampling layers and six upsampling layers. And when extracting features in each layer, the original two convolutions are changed to three, and when extracting features in the 1st and 3rd layers, skip connection chains are added in the 2nd and 3rd convolutions. After the above process, richer features will be extracted to cope with the relatively complex detection scenarios of mask templates.

[0126] The network of the discriminator consists of 8 convolutional layers. The last convolutional layer outputs an n*n matrix, and the mean value of the output matrix is taken as the output of the discrimination result.

[0127] Optionally, in another embodiment of this application, a specific implementation manner of step S205 is as Figure 3 shown, including the following steps:

[0128] S301. Use each sample image and its corresponding annotated contour image to perform an alternating training round on the generator and discriminator in the currently optimal image contour extraction model determined by the previous round of alternating training.

[0129] It should be noted that since the training goal of the generator is to make the generated contour map as similar as possible to the annotated contour Figure 1To prevent the discriminator from recognizing the contour map it generates. The training objective of the discriminator is to accurately recognize the contour map generated by the generator and the annotated contour map. Therefore, the objectives of the two are opposite. To obtain a more accurate model faster, the generator and the discriminator are trained alternately, that is, the discriminator is trained a certain number of times first, and then the generator is trained a certain number of times. And one such alternating training is one round of alternating training.

[0130] Since in each round of alternating training, the generator and the discriminator in the image contour extraction model are updated, and the updated image contour extraction model is not necessarily better than the image contour extraction model in the previous round of alternating training. To improve the training efficiency, in the embodiments of the present application, the better image contour extraction model is selected as the current optimal image contour extraction model for the next round of training. Therefore, when performing this round of alternating training, each sample image and its corresponding annotated contour image are used to perform one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the previous round of alternating training.

[0131] Optionally, in another embodiment of the present application, a specific implementation manner of step S301 is as Figure 4 shown, and includes the following steps:

[0132] S401. According to the preset training order, each sample image and its corresponding annotated contour image are used to alternately train the generator and the discriminator in the current optimal image contour extraction model determined in the previous round of alternating training.

[0133] Among them, the preset training order is to train the discriminator N times and then train the generator once. Currently, other training orders can also be adopted.

[0134] S402. Each time training is performed, the current loss value of the current training network is calculated through the loss function corresponding to the current training network.

[0135] Among them, the current training network is the generator or the discriminator for this training.

[0136] Optionally, the loss function corresponding to the generator can represent the error between the contour map generated by it recognized by the discriminator and the annotated contour map and / or the actual error between its generated contour map and the annotated contour map. And the loss function corresponding to the discriminator represents the error when it discriminates the contour map generated by the generator and the annotated contour map.

[0137] S403. Determine whether the current total number of training times is an integer multiple of the preset number of times.

[0138] Due to the high density of partial images on the reticle, a large proportion of small structures, and the captured images are prone to blurring, misalignment, etc. at the edges. In order to make the contour map predicted by the network as complete as possible and retain detailed information, the embodiments of this application add the calculation of image texture loss on the basis of the original loss function. Since the error of the texture is small, in the embodiments of this application, after every preset number of training reductions, the calculation of the texture loss is performed once. At this time, the cumulative texture loss is relatively large and can be more easily manifested. Therefore, if the current total number of training times is an integer multiple of the preset number of times, step S404 is executed. If the current total number of training times is not an integer multiple of the preset number of times, step S406 is executed.

[0139] S404. Calculate the current texture loss value through the texture error function.

[0140] Among them, the texture error function represents the texture error between the labeled contour image corresponding to the sample image and the contour map generated by the generator. Specifically, the texture error function can be specifically expressed as:

[0141]

[0142] Among them, m is the number of sample images; And for the i-th labeled contour image E i After normalization, it is the result of Gaussian blur; Is the result after normalizing the contour map of the noise contour Z i Generated by the generator.

[0143] S405. Add the current loss value of the current training network to the current texture loss value to obtain the current total loss value.

[0144] S406. Determine the current loss value of the current training network as the current total loss value.

[0145] S407. Adjust the parameters of the current training network based on the current total loss value.

[0146] Specifically, if the current training network is a discriminator, the gradient information can be updated in the following manner:

[0147]

[0148] Among them, Is the update coefficient of the discriminator. "D(E i )" is the result of the discriminator discriminating the labeled contour image E i ; D(G(Z i )) is the result G(Z i Generated by the generator discriminated by the discriminator for the noise contour Z i) The result of discrimination.

[0149] When the current training network is the generator, the gradient information can be updated in the following way:

[0150]

[0151] Among them, is the update coefficient of the generator.

[0152] S302. Calculate the loss value of this round of alternating training using the total loss function of the image contour extraction model.

[0153] It should be noted that the training objectives of the image contour extraction model include two: First, the generator should generate images as similar as possible to the real contours. Second, the discriminator should identify whether the contour is a real annotation contour or a contour generated by the generator as accurately as possible. Therefore, considering these two objectives comprehensively, the overall loss function is set as:

[0154]

[0155] S303. Determine whether the loss value of this round of alternating training converges.

[0156] Among them, if it is determined that the loss value of this round of alternating training converges, then step S304 is executed. If it is determined that the loss value of this round of alternating training does not converge, then step S305 is executed.

[0157] S304. Take the image contour extraction model obtained from this round of alternating training as the trained image contour extraction model.

[0158] S305. Compare whether the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training.

[0159] Among them, the similarity corresponding to one round of alternating training is the similarity between the contour map generated by the image contour extraction model after training and the annotated contour map.

[0160] Among them, if the similarity corresponding to this round of alternating training is not less than the similarity corresponding to the previous round of alternating training, then step S306 is executed. If the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training, then step S307 is executed.

[0161] S306. Take the image contour extraction model obtained from this round of alternating training as the current optimal image contour extraction model.

[0162] It should be noted that after determining the current optimal image contour extraction model in step S306, return to execute step S301 to perform the next round of alternating training.

[0163] S307. Use the image contour extraction model obtained from the previous round of alternating training as the current optimal image contour extraction model.

[0164] Similarly, after determining the current optimal image contour extraction model in step S307, return to step S301 to perform the next round of alternating training.

[0165] S103. Match the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points.

[0166] Specifically, the contour image of the transmission image and the contour image of the reflection image can be matched by the image matching normalized cross-correlation algorithm to obtain the matching coordinate points, that is, the contour matching coordinate points.

[0167] S104. Based on the contour matching coordinate points, match the transmission image and the reflection image.

[0168] Specifically, based on the contour matching coordinate points, the transmission image and the reflection image can be matched and aligned through various methods such as affine transformation.

[0169] S105. Calculate the sum of the pixel values of each pixel point on the transmission image and the pixel values of the matching pixel points on the reflection image to obtain a composite image.

[0170] Since the information of the transmission image and the reflection image is complementary, in the embodiments of the present application, the pixel values of each pixel point on the aligned transmission image are weighted with the pixel values of the matching pixel points on the reflection image, so that the information of the image is more complete, and it is easier to image the defective pixel points, and then the defective pixel points can be accurately detected.

[0171] S106. By comparing the pixel values of each pixel point on the composite image with a threshold, determine the defective pixel points and their position coordinates.

[0172] Specifically, a fixed threshold or an adaptive threshold can be used to compare with the pixel values of each pixel point on the composite image, that is, to compare the sum of the pixel values on the transmission image and the reflection image, so as to determine the pixel points that meet the threshold. Since the light intensity in the defective area becomes smaller, these pixel points are the defective pixel points, and the position coordinates of these defective pixel points are obtained.

[0173] An embodiment of the present application provides a defect detection method. First, a plurality of transmission images and their contour maps, as well as a plurality of reflection images and their contour maps, are used to train an image contour extraction model in advance, which can generate accurate contour maps of reflection images and transmission images. Therefore, when defect detection is required, first obtain the transmission image and the reflection image, and then input the transmission image and the reflection image into the pre-trained image contour extraction model respectively to obtain the contour image of the transmission image and the contour image of the reflection image, so as to accurately extract the contour images of the transmission image and the reflection image through the pre-trained model. Then, the contour image of the transmission image is matched with the contour image of the reflection image to obtain contour matching coordinate points, so that the transmission image and the reflection image can be matched based on the contour matching coordinate points. Since the contour maps are accurately generated, the transmission image and the reflection image can be accurately aligned. And because the information of the emission image and the emission image is complementary, and the light intensity of the defect area is relatively small, by calculating the sum of the pixel values of each pixel point on the transmission image and the pixel values of the matching pixel points on the reflection image, a composite image is obtained, and by comparing the pixel values of each pixel point on the composite image with a threshold value, the defect pixel points and their position coordinates are determined, so that the defects can be accurately detected.

[0174] Another embodiment of the present application provides a defect detection device, as Figure 5 shown, including the following units:

[0175] An image acquisition unit 501, configured to acquire a transmission image and a reflection image.

[0176] A contour extraction unit 502, configured to input the transmission image and the reflection image into a pre-trained image contour extraction model respectively to obtain a contour image of the transmission image and a contour image of the reflection image.

[0177] Among them, the image contour extraction model is pre-trained by using a plurality of transmission images and their contour maps, as well as a plurality of reflection images and their contour maps.

[0178] A contour matching unit 503, configured to match the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points.

[0179] An image alignment unit 504, configured to match the transmission image and the reflection image based on the contour matching coordinate points.

[0180] An image combination unit 505, configured to calculate the sum of the pixel values of each pixel point on the transmission image and the pixel values of the matching pixel points on the reflection image to obtain a composite image.

[0181] A defect determination unit 506, configured to determine defect pixel points and their position coordinates by comparing the pixel values of each pixel point on the synthesized image with a threshold value.

[0182] Optionally, in the defect detection device provided in another embodiment of the present application, the image acquisition unit includes:

[0183] An image acquisition subunit, configured to acquire a transmission image and a reflection image by using a time delay integration camera.

[0184] Optionally, in the defect detection device provided in another embodiment of the present application, it further includes:

[0185] A sample acquisition unit, configured to acquire multiple transmission images and multiple reflection images as original images.

[0186] A preprocessing unit, configured to preprocess each original image to obtain multiple sample images.

[0187] An image combination unit, configured to form the sample images into a sample image sequence.

[0188] A labeling unit, configured to respectively perform contour labeling on each sample image in the sample image sequence to obtain a labeled contour image corresponding to each sample image.

[0189] A training unit, configured to iteratively train an image contour extraction model by using the sample image sequence and the labeled contour images until the loss value of the image contour extraction model converges, to obtain a trained image contour extraction model.

[0190] Optionally, in the defect detection device provided in another embodiment of the present application, the preprocessing unit includes:

[0191] A preprocessing subunit, configured to perform data augmentation processing on each original image by using a preset data augmentation method to obtain multiple augmented images, and use each augmented image and each original image as sample images.

[0192] Optionally, in the defect detection device provided in another embodiment of the present application, the labeling unit includes:

[0193] A labeling subunit, configured to respectively label the contour of each sample image in the sample image sequence through a picture labeling tool and set a label name, and generate a json file including the labeled contour image and having the same name as the sample image.

[0194] Optionally, in the defect detection device provided in another embodiment of the present application, the image contour extraction model includes a generator for generating contour maps of the input transmission image and reflection image, and a discriminator for discriminating the accuracy of the contour maps generated by the generator.

[0195] Optionally, in the defect detection device provided in another embodiment of the present application, the generator includes six downsampling layers and six upsampling layers, and the last downsampling layer is connected to the first upsampling layer. There is a skip connection chain between the first five downsampling layers and the last five upsampling layers. The last upsampling layer is connected to the fully connected layer and outputs a contour map. Among them, each sampling layer performs feature extraction through three convolutions, and there is a skip connection chain between the second and third convolutions of the first and third layers.

[0196] The discriminator consists of eight convolutional layers. Among them, the mean value of the matrix output by the last convolutional layer is the discrimination result.

[0197] Optionally, in the defect detection device provided in another embodiment of the present application, the training unit includes:

[0198] An alternating training unit for performing one round of alternating training on the generator and the discriminator in the currently optimal image contour extraction model determined in the previous round of alternating training by using the sample image sequence and the annotated contour image.

[0199] A total loss calculation unit for calculating the loss value of this round of alternating training by using the total loss function of the image contour extraction model.

[0200] A first judgment unit for judging whether the loss value of this round of alternating training converges.

[0201] A training end unit for, when it is judged that the loss value of this round of alternating training converges, taking the image contour extraction model obtained in this round of alternating training as the trained image contour extraction model.

[0202] A comparison unit for, when it is judged that the loss value of this round of alternating training does not converge, comparing whether the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training. Among them, the similarity corresponding to one round of alternating training is the similarity between the contour map generated by the image contour extraction model obtained after training and the annotated contour map.

[0203] A first determination unit for, when the similarity corresponding to this round of alternating training is not less than the similarity corresponding to the previous round of alternating training, taking the image contour extraction model obtained in this round of alternating training as the currently optimal image contour extraction model.

[0204] A second determination unit for, when the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training, taking the image contour extraction model obtained in the previous round of alternating training as the currently optimal image contour extraction model. Among them, after determining the currently optimal image contour extraction model, return to the alternating training unit.

[0205] Optionally, in the defect detection device provided in another embodiment of the present application, the alternating training unit includes:

[0206] A sequential training unit for sequentially training the generator and the discriminator in the currently optimal image contour extraction model determined by the previous round of alternating training by using a sample image sequence and an annotated contour image in a preset training order. The preset training order is to train the discriminator N times and then train the generator once.

[0207] A network loss calculation unit for calculating the current loss value of the current training network through the loss function corresponding to the current training network every time a training is performed. The current training network is the generator or the discriminator for this training.

[0208] A second judgment unit for judging whether the current total number of training times is an integer multiple of the preset number of times.

[0209] A texture loss calculation unit for calculating the current texture loss value through a texture error function when the current total number of training times is an integer multiple of the preset number of times. The texture error function represents the texture error between the annotated contour image corresponding to the sample image and the contour image generated by the generator.

[0210] A current loss calculation unit for adding the current loss value of the current training network and the current texture loss value to obtain the current total loss value.

[0211] A loss determination unit for determining the current loss value of the current training network as the current total loss value when the current total number of training times is not an integer multiple of the preset number of times.

[0212] Adjust the parameters of the current training network based on the current total loss value.

[0213] It should be noted that for the specific working processes of the various units provided in the above embodiments of the present application, reference can be made to the implementation processes of the corresponding steps in the above method embodiments, which will not be elaborated here.

[0214] Another embodiment of the present application provides an electronic device, as Figure 6 shown, including:

[0215] A memory 601 and a processor 602.

[0216] Among them, the memory 601 is used to store programs.

[0217] The processor 602 is used to execute the program stored in the memory 601. When the program is executed, it is specifically used to implement the defect detection method provided in any of the above embodiments.

[0218] Another embodiment of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the defect detection method provided in any of the above embodiments.

[0219] Computer storage media include both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media such as modulated data signals and carrier waves.

[0220] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0221] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A defect detection method, characterized in that: include: Acquire a transmission image and a reflection image; The transmission image and the reflection image are respectively input into a pre-trained image contour extraction model to obtain a contour image of the transmission image and a contour image of the reflection image; wherein the image contour extraction model is pre-trained using a plurality of transmission images and their contour images and a plurality of reflection images and their contour images; Matching the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points; Matching the transmission image and the reflection image based on the contour matching coordinate points; Calculating the pixel value of each pixel point on the transmission image and adding the pixel value of each matching pixel point on the reflection image to obtain a composite image; By comparing the pixel value of each pixel point on the synthetic image with a threshold value, the defective pixel point and its position coordinates are determined.

2. The method according to claim 1, characterized in that The step of acquiring the transmission image and the reflection image comprises: A time-delay integration camera is used to acquire transmission and reflection images.

3. The method according to claim 1, characterized in that The training method of the image contour extraction model comprises: Acquire multiple transmission images and multiple reflection images as original images; Preprocessing each of the original images to obtain a plurality of sample images; Combining the sample images into a sample image sequence; Respectively annotate the contours of the sample images in the sample image sequence to obtain an annotated contour image corresponding to each sample image; The image contour extraction model is iteratively trained using the sample image sequence and the annotated contour image until the loss value of the image contour extraction model converges, thereby obtaining the trained image contour extraction model.

4. The method according to claim 3, characterized in that The preprocessing of each of the original images to obtain a plurality of sample images includes: A preset data expansion method is used to perform data expansion processing on each of the original images to obtain a plurality of expanded images, and each of the expanded images and each of the original images is used as the sample image.

5. The method according to claim 3, characterized in that: The step of respectively labeling the contours of each sample image in the sample image sequence to obtain a labeled contour image corresponding to each sample image includes: The contour of each sample image in the sample image sequence is respectively annotated by a picture annotation tool and an annotation name is set, so as to generate a json file containing the annotated contour image and having a name consistent with the name of the sample image.

6. The method according to claim 1, characterized in that The image contour extraction model includes a generator for generating a contour map of the input transmission image and reflection image, and a discriminator for judging the accuracy of the contour map generated by the generator.

7. The method according to claim 6, characterized in that The generator includes six downsampling layers and six upsampling layers, and the last downsampling layer is linked to the first upsampling layer, there is a jump connection chain between the first five downsampling layers and the last five upsampling layers, the last upsampling layer is connected to the fully connected layer, and outputs a contour map; wherein each sampling layer performs feature extraction through three convolutions, and there is a jump connection chain between the second convolution and the third convolution of the first layer and the third layer; The discriminator consists of eight convolutional layers, wherein the mean of the matrix output by the last convolutional layer is the discrimination result.

8. The method according to claim 6, characterized in that The iterative training of the image contour extraction model using the sample image sequence and the annotated contour image comprises: Using the sample image sequence and the annotated contour image, a round of alternating training is performed on the generator and the discriminator in the currently optimal image contour extraction model determined in the previous round of alternating training; Calculating the loss value of this round of alternating training using the total loss function of the image contour extraction model; Determine whether the loss value of this round of alternating training has converged; If it is determined that the loss value of this round of alternating training converges, the image contour extraction model obtained by this round of alternating training is used as the trained image contour extraction model; If it is determined that the loss value of this round of alternating training has not converged, then compare whether the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training; wherein the similarity corresponding to a round of alternating training is the similarity between the contour map generated by the image contour extraction model obtained after training and the labeled contour map; If the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the previous round of alternating training, the image contour extraction model obtained by the current round of alternating training is used as the current optimal image contour extraction model; If the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the previous round of alternating training, the image contour extraction model obtained by the previous round of alternating training is used as the current optimal image contour extraction model; After determining the currently optimal image contour extraction model, return to executing the method of using each of the sample images and their corresponding annotated contour images to perform a round of alternating training on the generator and the discriminator in the currently optimal image contour extraction model determined in the previous round of alternating training.

9. The method according to claim 8, characterized in that The method of using the sample image sequence and the annotated contour image to perform a round of alternating training on the generator and the discriminator in the currently optimal image contour extraction model determined by the previous round of alternating training includes: According to a preset training sequence, the sample image sequence and the annotated contour image are used in turn to alternately train the generator and the discriminator in the currently optimal image contour extraction model determined by the previous round of alternating training; wherein the preset training sequence is to train the discriminator N times and then train the generator once; Each time training is performed, the current loss value of the current training network is calculated by using the loss function corresponding to the current training network; wherein the current training network is the generator or the discriminator of this training; Determine whether the current total number of training times is an integer multiple of the preset number of times; If the current total number of training times is an integer multiple of the preset number of times, the current texture loss value is calculated by a texture error function; wherein the texture error function represents the texture error between the annotated contour image corresponding to the sample image and the contour image generated by the generator; Adding the current loss value of the current training network to the current texture loss value to obtain a current total loss value; If the current total number of training times is not an integer multiple of the preset number of times, the current loss value of the current training network is determined as the current total loss value; The parameters of the current training network are adjusted based on the current total loss value.

10. A defect detection device, characterized in that: include: An image acquisition unit, used for acquiring a transmission image and a reflection image; A contour extraction unit, used to input the transmission image and the reflection image into a pre-trained image contour extraction model, respectively, to obtain a contour image of the transmission image and a contour image of the reflection image; wherein the image contour extraction model is pre-trained using a plurality of transmission images and their contour maps and a plurality of reflection images and their contour maps; the image contour extraction model includes a generator for generating a contour map of an input image, and a discriminator for judging the accuracy of the contour map generated by the generator; A contour matching unit, used to match the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points; An image alignment unit, configured to match the transmission image and the reflection image based on the contour matching coordinate points; An image combining unit, used for calculating the pixel value of each pixel point on the transmission image and adding the pixel value of each matching pixel point on the reflection image to obtain a composite image; The defect determination unit is used to determine the defective pixel point and its position coordinates by comparing the pixel value of each pixel point on the synthetic image with a threshold value.

11. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the defect detection method according to any one of claims 1 to 9.

12. A computer storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, is used to implement the defect detection method according to any one of claims 1 to 9.

Images