Defect detection method and device, electronic device, and storage medium

Abstract

The application discloses a defect detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a transmission image and a reflection image; inputting the transmission image and the reflection image into an image contour extraction model respectively to obtain contour images of the transmission image and the reflection image; wherein the image contour extraction model is trained in advance by using multiple transmission images, reflection images and contour images of the transmission images and the reflection images; matching the generated contour images of the transmission image and the reflection image to obtain contour matching coordinate points; matching the transmission image and the reflection image based on the contour matching coordinate points; adding pixel values of all pixel points on the transmission image and the reflection image to obtain a synthesis image; and comparing the pixel values of all pixel points on the synthesis image with a threshold value to determine defect pixel points and position coordinates of the defect pixel points.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semiconductor detection, and in particular to a defect detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] A mask is an indispensable component in the photolithography process, and a design pattern is carried on the mask. In cooperation with a photolithography machine and a photoresist, the designed pattern on the mask is transferred to the photoresist on a substrate through exposure and development processes, so as to realize mass production of chips. However, during the production of the mask, pollutants such as particles, metal ions, chemicals and static electricity existing in the air and equipment will fall on the surface of the mask, causing defects and the like of the mask, thereby affecting the yield, performance and reliability of the chip. Therefore, the defect detection link of the mask is very important.

[0003] Since the light intensity of the defect area is small, a low gray value will be shown in the image, and the information of the transmission image and the reflection image has complementary characteristics. Therefore, one way of the current defect detection of the mask is to collect the transmission light image and the reflection light image of the same position of the mask, and then align the transmission light image and the reflection light image, so as to complement the information of the same pixel points on the transmission light image and the reflection light image, and screen out the defects in the mask based on the complementary information.

[0004] However, due to the small feature size and the influence of light scattering, the resolution of the features on the image is low, so the current technology cannot guarantee accurate alignment of the transmission light image and the reflection light image, thereby cannot guarantee the accuracy of defect detection. SUMMARY

[0005] Based on the deficiencies of the prior art, the present application provides a defect detection method and device, electronic equipment and storage medium to solve the problem that the prior art cannot guarantee the accuracy of defect detection.

[0006] In order to achieve the above purpose, the present application provides the following technical solutions:

[0007] The first aspect of the present application provides a defect detection method, comprising:

[0008] obtaining a transmission image and a reflection image;

[0009] respectively inputting the transmission image and the reflection image into a pre-trained image contour extraction model to obtain a contour image of the transmission image and a contour image of the reflection image; wherein the image contour extraction model is pre-trained by using multiple transmission images and contour images thereof and multiple reflection images and contour images thereof;

[0010] matching the contour image of the transmission image with the contour image of the reflection image to obtain a contour matching coordinate point;

[0011] matching the transmission image and the reflection image based on the contour matching coordinate point;

[0012] adding pixel values of each pixel point on the transmission image and pixel values of each matching pixel point on the reflection image to obtain a composite image;

[0013] determining a defect pixel point and a position coordinate of the defect pixel point by comparing pixel values of each pixel point on the composite image with a threshold value.

[0014] Optionally, in the defect detection method described above, the obtaining of the transmission image and the reflection image comprises:

[0015] obtaining the transmission image and the reflection image by using a time delay integration camera.

[0016] Optionally, in the defect detection method described above, the training method of the image contour extraction model comprises:

[0017] obtaining multiple transmission images and multiple reflection images as original images;

[0018] preprocessing each of the original images to obtain multiple sample images;

[0019] grouping each of the sample images into a sample image sequence;

[0020] performing contour labeling on each of the sample images in the sample image sequence to obtain a labeled contour image corresponding to each of the sample images;

[0021] iteratively training the image contour extraction model by using the sample image sequence and the labeled contour image until a loss value of the image contour extraction model converges, to obtain the trained image contour extraction model.

[0022] Optionally, in the defect detection method described above, the preprocessing of each of the original images to obtain multiple sample images comprises:

[0023] performing data augmentation processing on each of the original images by using a preset data augmentation method to obtain multiple augmented images, and using each of the augmented images and each of the original images as the sample image.

[0024] Optionally, in the defect detection method described above, the performing of contour labeling on each of the sample images in the sample image sequence to obtain a labeled contour image corresponding to each of the sample images comprises:

[0025] annotating the contour of each of the sample images in the sample image sequence through a picture annotation tool and setting an annotation name, to generate a json file containing the annotated contour image and having a name consistent with that of the sample image.

[0026] Optionally, in the defect detection method described above, the image contour extraction model comprises a generator for generating contour images of the input transmission image and reflection image, and a discriminator for judging the accuracy of the contour images generated by the generator.

[0027] Optionally, in the defect detection method described above, the generator comprises six down-sampling layers and six up-sampling layers, the last down-sampling layer and the first up-sampling layer are linked, there is a skip connection chain between the first five down-sampling layers and the last five up-sampling layers, the last up-sampling layer is connected to a fully connected layer, and the contour image is output; wherein each sampling layer performs feature extraction through three convolution operations, and the second and third convolution operations of the first and third layers have a skip connection chain.

[0028] The discriminator is composed of eight convolution layers; wherein the mean value of the matrix output by the last convolution layer is the discrimination result.

[0029] Optionally, in the defect detection method described above, the iterative training of the image contour extraction model using the sample image sequence and the annotated contour image comprises:

[0030] performing one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training using the sample image sequence and the annotated contour image;

[0031] calculating the loss value of the current round of alternating training using the total loss function of the image contour extraction model;

[0032] judging whether the loss value of the current round of alternating training converges;

[0033] if it is judged that the loss value of the current round of alternating training converges, the image contour extraction model obtained in the current round of alternating training is taken as the trained image contour extraction model;

[0034] if it is judged that the loss value of the current round of alternating training does not converge, the similarity corresponding to the current round of alternating training is compared with the similarity corresponding to the last round of alternating training; wherein the similarity corresponding to a round of alternating training is the similarity between the contour image generated by the image contour extraction model after training and the annotated contour image;

[0035] If the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the last round of alternating training, the image contour extraction model obtained in the current round of alternating training is taken as the current optimal image contour extraction model;

[0036] If the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training, the image contour extraction model obtained in the last round of alternating training is taken as the current optimal image contour extraction model;

[0037] After determining the current optimal image contour extraction model, returning to performing one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training by using each of the sample images and the corresponding labeled contour image.

[0038] Optionally, in the defect detection method described above, the one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training by using the sample image sequence and the labeled contour image comprises:

[0039] According to a preset training sequence, the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training are sequentially trained by using the sample image sequence and the labeled contour image; wherein the preset training sequence is that the discriminator is trained for N times and then the generator is trained once;

[0040] Each time the training is performed, a current loss value of the current training network is calculated by using a loss function corresponding to the current training network; wherein the current training network is the generator or the discriminator in the current training;

[0041] It is determined whether the current total training number is an integer multiple of a preset number;

[0042] If the current total training number is an integer multiple of the preset number, a current texture loss value is calculated by using a texture error function; wherein the texture error function represents a texture error between the labeled contour image corresponding to the sample image and the contour image generated by the generator;

[0043] The current loss value of the current training network and the current texture loss value are added to obtain a current total loss value;

[0044] If the current total training number is not an integer multiple of the preset number, the current loss value of the current training network is determined as the current total loss value;

[0045] The parameters of the current training network are adjusted based on the current total loss value.

[0046] The second aspect of the present application provides a defect detection device, comprising:

[0047] an image acquisition unit, configured to acquire a transmission image and a reflection image;

[0048] a contour extraction unit, configured to input the transmission image and the reflection image into a pre-trained image contour extraction model respectively, to obtain a contour image of the transmission image and a contour image of the reflection image; wherein the image contour extraction model is pre-trained by using a plurality of transmission images and contour images thereof, and a plurality of reflection images and contour images thereof;

[0049] a contour matching unit, configured to match the contour image of the transmission image with the contour image of the reflection image, to obtain contour matching coordinate points;

[0050] an image alignment unit, configured to match the transmission image and the reflection image based on the contour matching coordinate points;

[0051] an image combination unit, configured to add pixel values of each pixel point on the transmission image and pixel values of each matching pixel point on the reflection image, to obtain a composite image;

[0052] a defect determination unit, configured to determine a defect pixel point and its position coordinate by comparing pixel values of each pixel point on the composite image with a threshold value.

[0053] Optionally, in the defect detection device described above, the image acquisition unit comprises:

[0054] an image acquisition subunit, configured to acquire the transmission image and the reflection image by using a time delay integration camera.

[0055] Optionally, in the defect detection device described above, further comprising:

[0056] a sample acquisition unit, configured to acquire a plurality of transmission images and a plurality of reflection images as original images;

[0057] a preprocessing unit, configured to pre-process each of the original images, to obtain a plurality of sample images;

[0058] an image combination unit, configured to combine each of the sample images to form a sample image sequence;

[0059] a labeling unit, configured to label each of the sample images in the sample image sequence with a contour, to obtain a labeled contour image corresponding to each of the sample images;

[0060] a training unit configured to iteratively train the image contour extraction model using the sequence of sample images and the labeled contour images until a loss value of the image contour extraction model converges, and obtain a trained image contour extraction model.

[0061] Optionally, in the defect detection device, the preprocessing unit comprises:

[0062] a preprocessing subunit configured to perform data augmentation processing on each of the original images using a preset data augmentation method to obtain a plurality of augmented images, and use each of the augmented images and each of the original images as the sample images.

[0063] Optionally, in the defect detection device, the labeling unit comprises:

[0064] a labeling subunit configured to label the contour of each of the sample images in the sequence of sample images and set a label name by using a picture labeling tool, and generate a json file containing the labeled contour image and having a name consistent with the name of the sample image.

[0065] Optionally, in the defect detection device, the image contour extraction model comprises a generator configured to generate a contour map of the input transmission image and reflection image, and a discriminator configured to determine the accuracy of the contour map generated by the generator.

[0066] Optionally, in the defect detection device, the generator comprises six down-sampling layers and six up-sampling layers, the last down-sampling layer is linked to the first up-sampling layer, there is a skip connection chain between the first five down-sampling layers and the last five up-sampling layers, the last up-sampling layer is connected to a full connection layer, and the contour map is output; wherein each sampling layer extracts features through three convolution operations, and the first layer and the second and third convolution operations of the third layer have a skip connection chain.

[0067] The discriminator comprises eight convolution layers; wherein the mean value of the matrix output by the last convolution layer is the discrimination result.

[0068] Optionally, in the defect detection device, the training unit comprises:

[0069] an alternating training unit configured to perform one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training using the sequence of sample images and the labeled contour images.

[0070] a total loss calculation unit configured to calculate the loss value of the current round of alternating training using a total loss function of the image contour extraction model.

[0071] The first determining unit is configured to determine whether the loss value of the current round of alternating training converges.

[0072] The training ending unit is configured to, when it is determined that the loss value of the current round of alternating training converges, take the image contour extraction model obtained in the current round of alternating training as the trained image contour extraction model.

[0073] The comparison unit is configured to, when it is determined that the loss value of the current round of alternating training does not converge, compare whether the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training, wherein the similarity corresponding to a round of alternating training is the similarity between the contour map generated by the image contour extraction model obtained after training and the labeled contour map.

[0074] The first determining unit is configured to, when the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the last round of alternating training, take the image contour extraction model obtained in the current round of alternating training as the current optimal image contour extraction model.

[0075] The second determining unit is configured to, when the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training, take the image contour extraction model obtained in the last round of alternating training as the current optimal image contour extraction model, and return to the alternating training unit after determining the current optimal image contour extraction model.

[0076] Optionally, in the defect detection device, the alternating training unit comprises:

[0077] The sequential training unit is configured to sequentially perform alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training by using the sample image sequence and the labeled contour image according to a preset training sequence, wherein the preset training sequence is that the discriminator is trained for N times and then the generator is trained once.

[0078] The network loss calculation unit is configured to calculate a current loss value of the current training network by using a loss function corresponding to the current training network each time the training is performed, wherein the current training network is the generator or the discriminator in the current training.

[0079] The second determining unit is configured to determine whether the current total training number is an integer multiple of a preset number.

[0080] The texture loss calculation unit is configured to, when the current total training number is an integer multiple of the preset number, calculate a current texture loss value by using a texture error function, wherein the texture error function represents a texture error between the labeled contour image corresponding to the sample image and the contour map generated by the generator.

[0081] a current loss calculation unit, configured to add the current loss value of the current training network and the current texture loss value to obtain a current total loss value;

[0082] a loss determination unit, configured to determine the current loss value of the current training network as the current total loss value when the current total training number is not an integer multiple of the preset number of times;

[0083] adjust the parameters of the current training network based on the current total loss value.

[0084] The third aspect of the present application provides an electronic device, comprising:

[0085] a memory and a processor;

[0086] The memory is configured to store a program.

[0087] The processor is configured to execute the program, and the program, when executed, is specifically configured to implement the defect detection method according to any one of the preceding aspects.

[0088] The fourth aspect of the present application provides a computer storage medium for storing a computer program, wherein the computer program, when executed by a processor, is configured to implement the defect detection method according to any one of the preceding aspects.

[0089] The present application provides a defect detection method, which pre-trains an image contour extraction model using multiple transmission images and their contour maps and multiple reflection images and their contour maps, so as to generate accurate contour maps of the reflection images and the transmission images. Therefore, when defect detection is needed, the transmission images and the reflection images are first obtained, and then the transmission images and the reflection images are respectively input into the pre-trained image contour extraction model to obtain the contour images of the transmission images and the reflection images, so that the contour images of the transmission images and the reflection images are accurately extracted by the pre-trained model. Then, the contour images of the transmission images and the contour images of the reflection images are matched to obtain contour matching coordinate points, so that the transmission images and the reflection images can be matched based on the contour matching coordinate points. Since the contour maps are accurately generated, the transmission images and the reflection images can be accurately aligned. Since the information of the transmission images and the reflection images is complementary, and the light intensity of the defect area is relatively small, the pixel values of each pixel point on the transmission images and the pixel values of each matching pixel point on the reflection images are added to obtain a composite image, and the pixel values of each pixel point on the composite image are compared with a threshold value to determine the defect pixel points and their position coordinates, so that the defects can be accurately detected. BRIEF DESCRIPTION OF DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on the accompanying drawings are within the scope of protection of the present application.

[0091] Figure 1 A flowchart of a defect detection method provided for the embodiments of the present application;

[0092] Figure 2 A flowchart of a training method of an image contour extraction model provided for the embodiments of the present application;

[0093] Figure 3 A flowchart of a method for iteratively training an image contour extraction model using a sample image sequence and a labeled contour image provided for the embodiments of the present application;

[0094] Figure 4 A flowchart of a method for alternately training an image contour extraction model provided for the embodiments of the present application;

[0095] Figure 5 An architectural schematic diagram of a defect detection device provided for the embodiments of the present application;

[0096] Figure 6 An architectural schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0097] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0098] In this application, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0099] The embodiment of the present application provides a defect detection method to solve the problem that the prior art cannot guarantee the accuracy of defect detection.

[0100] First of all, it should be pointed out that the following embodiments of the present application are mainly described by taking the defect detection of a mask plate as an example. However, the defect detection method provided by the embodiments of the present application is not limited to be applied to the defect detection of the mask plate, and can also be applied to the detection of other objects having the same characteristics as the mask plate.

[0101] The embodiment of the present application provides a defect detection method, as shown in Figure 1 The embodiment of the present application provides a defect detection method, as shown in

[0102] S101, acquire a transmission image and a reflection image.

[0103] Specifically, when it is necessary to detect defects of an object, the transmission image and the reflection image of the same position of the object are acquired. Therefore, if the mask plate is detected, the transmission image and the reflection image of the same position of the mask plate are acquired.

[0104] Optionally, the transmission image and the reflection image of the same position can be acquired by using time delay integration.

[0105] In addition, in order to improve the quality of the image and further improve the quality of the contour extraction, the transmission image and the reflection image are preprocessed.

[0106] S102, respectively input the transmission image and the reflection image into a pre-trained image contour extraction model to obtain a contour image of the transmission image and a contour image of the reflection image.

[0107] It should be noted that in order to accurately align the transmission image and the reflection image, in the embodiment of the present application, the alignment is performed in a contour matching manner, so the contour of the transmission image and the contour of the reflection image need to be extracted, and then the transmission image and the reflection image are aligned through the contours. Therefore, the contours of the transmission image and the reflection image need to be extracted first. Although contour extraction can be performed through existing Canny, Sobel, Laplace operator and the like. However, due to the small feature size and the influence of light scattering, the resolution of the image contour is low, so these methods cannot accurately extract the contours of the transmission image and the reflection image. Therefore, in order to accurately extract the contours of the transmission image and the reflection image, in the embodiment of the present application, the image contour extraction model is trained in advance using multiple transmission images and their contour maps and multiple reflection images and their contour maps, so that the image contour extraction model can generate the contour map of the input transmission image and reflection image. Therefore, when defect detection is needed, the currently collected transmission image and reflection image are respectively input into the pre-trained image contour extraction model, so as to extract the contours of the transmission image and the reflection image through the image contour extraction model.

[0108] Optionally, another embodiment of the present application provides a training method of an image contour extraction model, as shown in Figure 2 The method comprises the following steps:

[0109] S201, acquire multiple transmission images and multiple reflection images as original images.

[0110] Specifically, the transmission images and the reflection images of multiple positions of multiple mask plates can be acquired through a time delay integration camera.

[0111] S202, pre-process each original image to obtain multiple sample images.

[0112] In order to improve the quality of the original image, so as to improve the accuracy of contour extraction, and also to expand the sample data, in the embodiment of the present application, after obtaining each original image, the original image is pre-processed, and each image obtained after pre-processing is used as a sample image.

[0113] Optionally, a specific implementation of step S202 comprises:

[0114] Each original image is processed by a preset data expansion method to obtain multiple expansion images, and each expansion image and each original image is used as the sample image.

[0115] The preset data augmentation method can include, but is not limited to, horizontal flipping, vertical flipping, random cropping, rotation, noise adding, and the like. Thus, a large amount of training data can be obtained through data augmentation to train the model, and the accuracy of the model can be ensured.

[0116] S203, group each sample image into a sample image sequence.

[0117] S204, respectively, outline the sample images in the sample image sequence, and obtain the labeled contour images corresponding to each sample image.

[0118] In order to facilitate subsequent analysis of whether the contour image generated by the generator is accurate, it is necessary to first outline each sample image in the sample image sequence to obtain the accurate contour image of each sample image.

[0119] Optionally, in another embodiment of the present application, a specific implementation of step S204 includes:

[0120] Each sample image in the sample image sequence is labeled by a picture labeling tool, and the contour of each sample image is set as a label name, and a json file containing the labeled contour image is generated, and the name of the json file is consistent with the name of the sample image.

[0121] S205, using the sample image sequence and the labeled contour image, iteratively training the image contour extraction model until the loss value of the image contour extraction model converges, and obtaining the trained image contour extraction model.

[0122] Optionally, in another embodiment of the present application, the image contour extraction model includes a generator for generating an outline of an input transmission image and a reflection image, and a discriminator for judging the accuracy of the contour image generated by the generator.

[0123] Specifically, in the training process, the generator is continuously optimized so that the generated contour image and the real labeled contour image are as consistent as possible, achieving the purpose of "cheating" the discriminator, and making the discriminator as difficult as possible to distinguish the generated contour image from the real labeled contour image. The discriminator is also continuously optimized to make its discrimination result more accurate, that is, it can accurately distinguish the generated contour image from the real labeled contour image. Therefore, through continuous optimization of the generator and the discriminator, the generator can finally generate an accurate contour image for the input transmission image or reflection image.

[0124] Therefore, in the training, the sample image sequence is input into the generator in the image contour extraction model, and the contour image corresponding to each sample image in the sample image sequence is generated by the generator. Then, according to the discrimination result of the discriminator on the contour image generated by the generator and the labeled contour image, the parameters of the generator are updated. The contour image generated by the generator and the labeled contour image are input into the discriminator, and the parameters of the discriminator are constantly updated according to the accuracy of the discrimination result, until the loss value of the image contour extraction model converges.

[0125] Optionally, the generator can specifically adopt an improved UNet network, that is, the basic network of the generator is a UNet network. The UNet network includes 5 down-sampling layers and 5 up-sampling layers, the last down-sampling layer and the first up-sampling layer are linked, and there is a skip connection chain between the first four down-sampling layers and the last four up-sampling layers. The last up-sampling layer is connected to a fully connected layer to output the contour extraction result. However, in the mask detection, the types of masks are various, the shapes are different, and most of the images are relatively dense. In addition, the clarity and pixel size of the images captured at different magnifications are different. Therefore, in order to ensure the generalization and robustness of the model, the application additionally adds a down-sampling layer and an up-sampling layer in the generator module of the GAN neural network, so that the generator includes six down-sampling layers and six up-sampling layers. In addition, the original two times of convolution are changed to three times of convolution when extracting features at each layer, and a skip connection chain is added at the second convolution and the third convolution when extracting features at the first layer and the third layer. After the above process, more rich features will be extracted to cope with the relatively complex detection scene of the mask.

[0126] The network of the discriminator is composed of 8 convolutional layers, and the last convolutional layer outputs an n*n matrix, and the mean value of the output matrix is taken as the output of the discrimination result.

[0127] Optionally, in another embodiment of the application, a specific implementation of step S205 includes the following steps as shown in Figure 3

[0128] S301, using each sample image and its corresponding labeled contour image, the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training are alternately trained.

[0129] It should be noted that, since the goal of the generator training is to make the generated contour as close as possible to the labeled contour Figure 1 ​The generator is not allowed to identify the generated contour map. The training target of the discriminator is to accurately identify the contour map generated by the generator and the labeled contour map. Therefore, the targets of the two are opposite, and therefore, in order to obtain a more accurate model faster, the generator and the discriminator are trained alternately, that is, the discriminator is trained for a certain number of times, and then the generator is trained for a certain number of times. One such alternating training is one round of alternating training.

[0130] Since each round of alternating training updates the generator and the discriminator in the image contour extraction model, the updated image contour extraction model is not necessarily better than the image contour extraction model of the last round of alternating training. In order to speed up the training efficiency, in the embodiment of the present application, the better image contour extraction model is selected as the current optimal image contour extraction model for the next round of training. Therefore, when performing the current round of alternating training, the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training are alternately trained using each sample image and its corresponding labeled contour image.

[0131] Alternatively, in another embodiment of the present application, a specific implementation of step S301 includes the following steps as shown in Figure 4

[0132] S401. According to a predetermined training sequence, alternately train the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training using each sample image and its corresponding labeled contour image.

[0133] The predetermined training sequence is to train the discriminator for N times and then train the generator once. Alternatively, other training sequences can be used.

[0134] S402. Calculate the current loss value of the current training network by the loss function corresponding to the current training network after each training.

[0135] The current training network is the generator or the discriminator in the current training.

[0136] Alternatively, the loss function corresponding to the generator can represent the error between the contour map generated by the discriminator and the labeled contour map and / or the actual error between the contour map generated by the generator and the labeled contour map. The loss function corresponding to the discriminator represents the error when the discriminator discriminates the contour map generated by the generator and the labeled contour map.

[0137] S403. Determine whether the current total training number is an integer multiple of the predetermined number.

[0138] ​Since the mask part image has high density, small structure accounts for a large proportion, and the image captured is prone to blur and misalignment at the edge, in order to make the contour map predicted by the network as complete as possible and retain the detail information, the embodiment of the application adds the calculation of the image texture loss on the basis of the original loss function. Since the texture error is small, in the embodiment of the application, the calculation of the texture loss is performed only once after the training is reduced by a preset number of times, at this time, the texture loss is accumulated and can be more easily realized. Therefore, if the current total training number is an integer multiple of the preset number, step S404 is performed. If the current total training number is not an integer multiple of the preset number, step S406 is performed.

[0139] S404, calculate the current texture loss value by a texture error function.

[0140] The texture error function represents the texture error of the sample image corresponding to the labeled contour image and the contour map generated by the generator. Specifically, the texture error function can be specifically represented as:

[0141]

[0142] Wherein, m is the number of sample images; And after normalizing the i-th labeled contour image E i , the result of Gaussian blur is obtained; For the contour map of the generator generated in accordance with the noise contour Z i , the result after normalization is obtained.

[0143] S405, add the current loss value of the current training network to the current texture loss value to obtain the current total loss value.

[0144] S406, determine the current loss value of the current training network as the current total loss value.

[0145] S407, adjust the parameters of the current training network based on the current total loss value.

[0146] Specifically, if the current training network is a discriminator, the gradient information can be updated in the following manner:

[0147]

[0148] Wherein, is the update coefficient of the discriminator. "D(E i )" is the result of discriminating the labeled contour image E i by the discriminator; D(G(Z i )) is the result of the discriminator on the generator generated in accordance with the noise contour Z i G(Z ithe result of the discrimination.

[0149] When the current training network is the generator, the gradient information can be updated in the following manner:

[0150]

[0151] wherein, is the update coefficient of the generator.

[0152] S302, calculate the loss value of the current round of alternating training by using the total loss function of the image contour extraction model.

[0153] It should be noted that the training target of the image contour extraction model includes two: first, the generator should generate images as close to the real contour as possible. Second, the discriminator should identify whether the contour is a real contour or a contour generated by the generator. Therefore, considering the two targets comprehensively, the overall loss function is set as:

[0154]

[0155] S303, judge whether the loss value of the current round of alternating training converges.

[0156] If it is judged that the loss value of the current round of alternating training converges, step S304 is executed. If it is judged that the loss value of the current round of alternating training does not converge, step S305 is executed.

[0157] S304, the image contour extraction model obtained by the current round of alternating training is taken as the trained image contour extraction model.

[0158] S305, compare whether the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training.

[0159] The similarity corresponding to a round of alternating training is the similarity between the contour map generated by the image contour extraction model after training and the labeled contour map.

[0160] If the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the last round of alternating training, step S306 is executed. If the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training, step S307 is executed.

[0161] S306, the image contour extraction model obtained by the current round of alternating training is taken as the current optimal image contour extraction model.

[0162] It should be noted that after the current optimal image contour extraction model is determined in step S306, step S301 is returned to perform the next round of alternating training.

[0163] S307, taking the image contour extraction model obtained in the last round of alternating training as the current optimal image contour extraction model.

[0164] Likewise, after performing step S307 to determine the current optimal image contour extraction model, return to performing step S301 to conduct the next round of alternating training.

[0165] S103, matching the contour image of the transmission image with the contour image of the reflection image to obtain contour matching coordinate points.

[0166] Specifically, the contour image of the transmission image and the contour image of the reflection image can be matched by using an image matching normalized cross-correlation algorithm, so as to obtain matching coordinate points, i.e., contour matching coordinate points.

[0167] S104, matching the transmission image and the reflection image based on the contour matching coordinate points.

[0168] Specifically, the transmission image and the reflection image can be matched and aligned by using various methods such as affine transformation based on the contour matching coordinate points.

[0169] S105, adding the pixel values of each pixel point on the transmission image and the pixel values of each matching pixel point on the reflection image to obtain a composite image.

[0170] Since the information of the transmission image and the reflection image is complementary, in the embodiments of the present application, the pixel values of each pixel point on the aligned transmission image and the pixel values of each matching pixel point on the reflection image are weighted, so that the information of the image is more complete, and the pixel points with defects in the image can be more easily found, and then the defective pixel points can be accurately detected.

[0171] S106, determining the defective pixel points and their position coordinates by comparing the pixel values of each pixel point on the composite image with a threshold value.

[0172] Specifically, a fixed threshold value or an adaptive threshold value can be used to compare with the pixel values of each pixel point on the composite image, i.e., the sum of the pixel values on the transmission image and the reflection image is compared, so that the pixel points satisfying the threshold value can be determined. Since the light intensity of the defect area is small, these pixel points are the defective pixel points, and the position coordinates of these defective pixel points are obtained.

[0173] The embodiment of the present application provides a defect detection method, an image contour extraction model is trained in advance by using multiple transmission images and contour maps thereof and multiple reflection images and contour maps thereof, and the image contour extraction model can generate accurate contour maps of the transmission images and the reflection images. Therefore, when defect detection is needed, a transmission image and a reflection image are acquired first, then the transmission image and the reflection image are input into the pre-trained image contour extraction model respectively to obtain a contour image of the transmission image and a contour image of the reflection image, so that the contour images of the transmission image and the reflection image are accurately extracted by the pre-trained model. Then, the contour image of the transmission image and the contour image of the reflection image are matched to obtain contour matching coordinate points, so that the transmission image and the reflection image can be matched based on the contour matching coordinate points. Since the contour maps are accurately generated, the transmission image and the reflection image can be accurately aligned. Since the information of the transmission image and the reflection image is complementary, and the light intensity of a defect area is relatively small, the pixel values of each pixel point on the transmission image and the pixel values of each matching pixel point on the reflection image are added to obtain a synthesis image, and the pixel values of each pixel point on the synthesis image are compared with a threshold value to determine a defect pixel point and a position coordinate of the defect pixel point, so that the defect can be accurately detected.

[0174] Another embodiment of the present application provides a defect detection device, as shown in the accompanying drawings, comprising the following units: Figure 5

[0175] The image acquisition unit 501 is used for acquiring a transmission image and a reflection image.

[0176] The contour extraction unit 502 is used for inputting the transmission image and the reflection image into a pre-trained image contour extraction model respectively to obtain a contour image of the transmission image and a contour image of the reflection image.

[0177] The image contour extraction model is trained in advance by using multiple transmission images and contour maps thereof and multiple reflection images and contour maps thereof.

[0178] The contour matching unit 503 is used for matching the contour image of the transmission image and the contour image of the reflection image to obtain contour matching coordinate points.

[0179] The image alignment unit 504 is used for matching the transmission image and the reflection image based on the contour matching coordinate points.

[0180] The image combination unit 505 is used for adding the pixel values of each pixel point on the transmission image and the pixel values of each matching pixel point on the reflection image to obtain a synthesis image.

[0181] ​The defect determination unit 506 is configured to determine the defect pixel and the position coordinate of the defect pixel by comparing the pixel value of each pixel on the synthetic image with a threshold value.

[0182] Optionally, in the defect detection device provided in another embodiment of the present application, the image acquisition unit comprises:

[0183] The image acquisition subunit is configured to acquire the transmission image and the reflection image by using the time delay integration camera.

[0184] Optionally, in the defect detection device provided in another embodiment of the present application, the image acquisition unit comprises:

[0185] The sample acquisition unit is configured to acquire a plurality of transmission images and a plurality of reflection images as the original images.

[0186] The preprocessing unit is configured to pre-process each original image to obtain a plurality of sample images.

[0187] The image combination unit is configured to combine each sample image to form a sample image sequence.

[0188] The labeling unit is configured to label the contour of each sample image in the sample image sequence to obtain a labeled contour image corresponding to each sample image.

[0189] The training unit is configured to iteratively train the image contour extraction model by using the sample image sequence and the labeled contour image until the loss value of the image contour extraction model converges, to obtain the trained image contour extraction model.

[0190] Optionally, in the defect detection device provided in another embodiment of the present application, the preprocessing unit comprises:

[0191] The preprocessing subunit is configured to perform data augmentation processing on each original image by using a preset data augmentation method to obtain a plurality of augmented images, and each augmented image and each original image as a sample image.

[0192] Optionally, in the defect detection device provided in another embodiment of the present application, the labeling unit comprises:

[0193] The labeling subunit is configured to label the contour of each sample image in the sample image sequence by using a picture labeling tool and set a label name to generate a json file containing the labeled contour image, and the name of the json file is consistent with the name of the sample image.

[0194] Optionally, in the defect detection device provided in another embodiment of the present application, the image contour extraction model comprises a generator configured to generate a contour map of the input transmission image and reflection image, and a discriminator configured to determine the accuracy of the contour map generated by the generator.

[0195] Optionally, in the defect detection device provided in another embodiment of the present application, the generator comprises six down-sampling layers and six up-sampling layers, and the last down-sampling layer and the first up-sampling layer are linked, and there is a skip connection chain between the first five down-sampling layers and the last five up-sampling layers, and the last up-sampling layer is connected to a full connection layer and outputs the contour map. Wherein, each sampling layer extracts features through three times of convolution, and the second and third convolutions of the first and third layers have a skip connection chain.

[0196] The discriminator is composed of eight convolution layers. Wherein, the mean of the matrix output by the last convolution layer is the discrimination result.

[0197] Optionally, in the defect detection device provided in another embodiment of the present application, the training unit comprises:

[0198] The alternating training unit is configured to perform one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the last round of alternating training by using the sample image sequence and the labeled contour image.

[0199] The total loss calculation unit is configured to calculate the loss value of the current round of alternating training by using the total loss function of the image contour extraction model.

[0200] The first judgment unit is configured to judge whether the loss value of the current round of alternating training converges.

[0201] The training end unit is configured to take the image contour extraction model obtained in the current round of alternating training as the trained image contour extraction model when it is judged that the loss value of the current round of alternating training converges.

[0202] The comparison unit is configured to compare whether the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training when it is judged that the loss value of the current round of alternating training does not converge. Wherein, the similarity corresponding to a round of alternating training is the similarity between the contour map generated by the image contour extraction model obtained after training and the labeled contour image.

[0203] The first determination unit is configured to take the image contour extraction model obtained in the current round of alternating training as the current optimal image contour extraction model when the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the last round of alternating training.

[0204] The second determination unit is configured to take the image contour extraction model obtained in the last round of alternating training as the current optimal image contour extraction model when the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the last round of alternating training. Wherein, after determining the current optimal image contour extraction model, the alternating training unit is returned.

[0205] Optionally, in the defect detection device provided in another embodiment of the present application, the alternately training unit comprises:

[0206] The sequentially training unit is configured to sequentially train the generator and the discriminator in the current optimal image contour extraction model determined by the previous alternately training by using the sample image sequence and the labeled contour image according to a preset training sequence. The preset training sequence is that the discriminator is trained for N times and then the generator is trained once.

[0207] The network loss calculation unit is configured to calculate a current loss value of the current training network by using a loss function corresponding to the current training network each time the training is performed. The current training network is the generator or the discriminator in the current training.

[0208] The second judging unit is configured to judge whether the current total training number is an integer multiple of the preset number.

[0209] The texture loss calculation unit is configured to calculate a current texture loss value by using a texture error function when the current total training number is an integer multiple of the preset number. The texture error function represents a texture error between the labeled contour image corresponding to the sample image and the contour image generated by the generator.

[0210] The current loss calculation unit is configured to add the current loss value of the current training network and the current texture loss value to obtain a current total loss value.

[0211] The loss determination unit is configured to determine the current loss value of the current training network as the current total loss value when the current total training number is not an integer multiple of the preset number.

[0212] The parameters of the current training network are adjusted based on the current total loss value.

[0213] It should be noted that the specific working processes of the various units provided in the above embodiments of the present application can be correspondingly referred to the implementation processes of the corresponding steps in the above method embodiments, which will not be described here.

[0214] Another embodiment of the present application provides an electronic device, as shown in the figure, comprising: Figure 6

[0215] The memory 601 and the processor 602.

[0216] The memory 601 is configured to store a program.

[0217] The processor 602 is configured to execute the program stored in the memory 601, and the program is executed to implement the defect detection method provided in any one of the above embodiments.

[0218] ​Another embodiment of the present application provides a computer storage medium, configured to store a computer program, wherein the computer program is executed by a processor to implement the defect detection method according to any one of the above embodiments.

[0219] Computer storage media includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory computer readable media, such as modulated data signals and carrier waves.

[0220] The skilled person can further realize that the units and algorithm steps of various examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of various examples have been described in general terms above as being performed in a specific manner. Whether the functions are performed in hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can use different methods to implement the described functions for each particular application, but such implementation should not be considered beyond the scope of the present application.

[0221] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A defect detection method, characterized in that, include: Acquire transmission and reflection images; The transmitted image and the reflected image are respectively input into a pre-trained image contour extraction model to obtain the contour images of the transmitted image and the reflected image; wherein, the image contour extraction model is pre-trained using multiple transmitted images and their contour maps, as well as multiple reflected images and their contour maps. The contour image of the transmitted image is matched with the contour image of the reflected image to obtain the contour matching coordinate points; Based on the contour matching coordinate points, the transmission image and the reflection image are matched; The pixel values ​​of each pixel in the transmitted image are calculated and added to the pixel values ​​of the matching pixels in the reflected image to obtain a composite image; By comparing the pixel values ​​of each pixel in the synthesized image with a threshold, the defective pixels and their location coordinates are determined.

2. The method according to claim 1, characterized in that, The acquisition of the transmission image and the reflection image includes: Transmission and reflection images are acquired using a time-delay integral camera.

3. The method according to claim 1, characterized in that, The training method for the image contour extraction model includes: Multiple transmission images and multiple reflection images are acquired as the original image; Each of the original images is preprocessed to obtain multiple sample images; The sample images are combined into a sample image sequence; Contour annotation is performed on each of the sample images in the sample image sequence to obtain the annotated contour image corresponding to each sample image; Using the sample image sequence and the labeled contour image, the image contour extraction model is iteratively trained until the loss value of the image contour extraction model converges, thus obtaining the trained image contour extraction model.

4. The method according to claim 3, characterized in that, The preprocessing of each of the original images to obtain multiple sample images includes: The original images are augmented using a preset data augmentation method to obtain multiple augmented images, and each augmented image and each original image are used as the sample images.

5. The method according to claim 3, characterized in that, The step of performing contour annotation on each of the sample images in the sample image sequence to obtain an annotated contour image corresponding to each sample image includes: The outline of each sample image in the sample image sequence is marked using an image annotation tool, and a annotation name is set. A JSON file containing the annotated outline images and whose name is consistent with the name of the sample image is generated.

6. The method according to claim 3, characterized in that, The image contour extraction model includes a generator for generating contour maps of the input transmission image and reflection image, and a discriminator for judging the accuracy of the contour maps generated by the generator.

7. The method according to claim 6, characterized in that, The generator includes six downsampling layers and six upsampling layers, with the last downsampling layer linked to the first upsampling layer. There are skip connection chains between the first five downsampling layers and the last five upsampling layers. The last upsampling layer is connected to a fully connected layer and outputs a contour map. Each sampling layer extracts features through three convolutions, and there are skip connection chains between the second and third convolutions of the first and third layers. The discriminator consists of eight convolutional layers; the mean of the matrix output by the last convolutional layer is the discrimination result.

8. The method according to claim 6, characterized in that, The iterative training of the image contour extraction model using the sample image sequence and the labeled contour image includes: Using the sample image sequence and the labeled contour image, perform one round of alternating training on the generator and the discriminator in the current optimal image contour extraction model determined in the previous round of alternating training; The loss value of this round of alternating training is calculated using the total loss function of the image contour extraction model. Determine whether the loss value of this round of alternating training has converged; If it is determined that the loss value of this round of alternating training has converged, then the image contour extraction model obtained in this round of alternating training is taken as the trained image contour extraction model. If it is determined that the loss value of this round of alternating training has not converged, then compare whether the similarity corresponding to this round of alternating training is less than the similarity corresponding to the previous round of alternating training; wherein, the similarity corresponding to one round of alternating training is the similarity between the contour map generated by the image contour extraction model after training and the labeled contour map. If the similarity corresponding to the current round of alternating training is not less than the similarity corresponding to the previous round of alternating training, then the image contour extraction model obtained in the current round of alternating training will be taken as the current optimal image contour extraction model. If the similarity corresponding to the current round of alternating training is less than the similarity corresponding to the previous round of alternating training, then the image contour extraction model obtained in the previous round of alternating training will be taken as the current optimal image contour extraction model. After determining the current optimal image contour extraction model, the process returns to perform an alternating training of the generator and discriminator in the current optimal image contour extraction model determined in the previous round of alternating training, using each of the sample images and their corresponding labeled contour images.

9. The method according to claim 8, characterized in that, The step of using the sample image sequence and the labeled contour image to perform an alternating training of the generator and the discriminator in the currently optimal image contour extraction model determined in the previous round of alternating training includes: According to the preset training order, the generator and the discriminator in the current optimal image contour extraction model determined in the previous round of alternating training are trained alternately using the sample image sequence and the labeled contour image; wherein, the preset training order is to train the discriminator N times and then train the generator once. For each training iteration, the current loss value of the current training network is calculated using the loss function corresponding to the current training network; wherein, the current training network is the generator or the discriminator used in this training iteration. Determine if the current total number of training iterations is an integer multiple of the preset number of iterations; If the current total number of training iterations is an integer multiple of the preset number of iterations, then the current texture loss value is calculated using the texture error function; wherein, the texture error function represents the texture error between the labeled contour image corresponding to the sample image and the contour image generated by the generator; The current loss value of the currently trained network is added to the current texture loss value to obtain the current total loss value; If the current total number of training iterations is not an integer multiple of the preset number of iterations, then the current loss value of the currently trained network is determined as the current total loss value; The parameters of the currently trained network are adjusted based on the current total loss value.

10. A defect detection device, characterized in that, include: The image acquisition unit is used to acquire transmission and reflection images; A contour extraction unit is used to input the transmission image and the reflection image into a pre-trained image contour extraction model to obtain contour images of the transmission image and the reflection image, respectively; wherein, the image contour extraction model is pre-trained using multiple transmission images and their contour maps, and multiple reflection images and their contour maps; the image contour extraction model includes a generator for generating contour maps of the input image, and a discriminator for judging the accuracy of the contour maps generated by the generator; A contour matching unit is used to match the contour image of the transmitted image with the contour image of the reflected image to obtain contour matching coordinate points; An image alignment unit is used to match the transmitted image and the reflected image based on the contour matching coordinate points; An image combining unit is used to calculate and add the pixel values ​​of each pixel in the transmitted image to the pixel values ​​of the matching pixels in the reflected image to obtain a composite image; The defect determination unit is used to determine the defective pixel and its position coordinates by comparing the pixel value of each pixel on the synthesized image with a threshold.

11. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the defect detection method as described in any one of claims 1 to 9.

12. A computer storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, is used to implement the defect detection method as described in any one of claims 1 to 9.

Images