Image difference reduction preprocessing method and device

Abstract

Disclosed are a source image preprocessing method and apparatus, comprising: outputting a feature map by applying at least one kernel to a source image; generating a reconstructed source image by applying at least one mask to a feature map corresponding to the source image; and updating the at least one kernel based on a value of a loss function between the reconstructed source image and the target image.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus with image-difference reduction preprocessing. Background Art

[0002] For example, inspection of defects of electronic devices is an important process for reducing device failures and process accidents in various fields. In order to detect defects from image data of electronic devices, defect signals may be extracted therefrom.

[0003] A defect signal may be extracted from a difference image of the difference between a reference image and a specific image of the electronic device. At least compared to a fine defect signal, the defect signal may not be properly extracted due to high background noise (low SNR). As a result, the performance of the defect inspection system may be suboptimal. Summary of the invention

[0004] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a method, performed by one or more processors, is used to preprocess a source image, and the method includes: generating a feature map by applying at least one kernel to the source image; generating a first reconstructed source image by applying at least one mask to the feature map, the feature map corresponding to the source image; and updating the kernel based on a value of a loss function between the first reconstructed source image and a target image.

[0006] The at least one kernel may include a foreground kernel and a background kernel, and the generating of the feature map may include: applying the foreground kernel to the source image; and applying the background kernel to the source image.

[0007] The feature map may include a first feature map generated by applying a foreground kernel to the source image and a second feature map generated by applying a background kernel to the source image, and the generation of the first reconstructed source image may include: applying a foreground mask to the first feature map; and applying a background mask to the second feature map.

[0008] The generating of the first reconstructed source image may also include: forming a first image by applying a foreground mask to the first feature map; forming a second image by applying a background mask to the second feature map; and combining the first image with the second image.

[0009] The method may also include: generating a second reconstructed source image by applying an updated version of the kernel and a mask to the source image, the updated version of the kernel being generated based on the kernel being updated according to the value of the loss function; and determining whether to end updating the kernel based on the second reconstructed source image being generated.

[0010] The determination of whether to end updating the kernel may include updating the updated kernel based on a value of a loss function between the second reconstructed source image and the target image, or ending training through updating of the kernel.

[0011] The determination of whether to end the updating of the kernel may include ending the training by the updating of the kernel based on the source image being reconstructed a predetermined number of times.

[0012] The method may further include determining, when an end of the kernel-based updating is determined, by preprocessing the source image using the trained background kernel.

[0013] In another general aspect, a method for detecting defects in a defective image having a corresponding reference image without defects is performed by one or more processors and includes: applying a trained background kernel to the reference image to generate a preprocessed reference image, the trained background kernel being trained based on applying the background kernel and a mask to the reference image; and detecting defects in the defective image based on a comparison between the preprocessed reference image and the defective image.

[0014] The method may also include: outputting a feature map by applying a background kernel to a reference image, applying a mask to the feature map to generate a reconstructed reference image, the reconstructed reference image being a reconstruction of the reference image; and updating the background kernel based on the reconstructed reference image.

[0015] The output of the feature map may include: applying a foreground kernel to a reference image; and applying a background kernel to the reference image.

[0016] Applying the foreground kernel to the reference image can generate a first feature map of the feature map, applying the background kernel to the reference image can generate a second feature map of the feature map, and reconstruction of the reference image may include: applying a foreground mask of the mask to the first feature map to generate a feature map of the first mask; applying the background mask of the mask to the second feature map to generate a feature map of the second mask; and reconstructing the reference image based on a combination of the feature map of the first mask and the feature map of the second mask.

[0017] In another general aspect, a device for preprocessing a reference image includes: one or more processors; a memory storing instructions, the instructions being configured to cause the one or more processors to perform a process, the process including: generating a feature map by applying a kernel to a reference image; generating a first reconstructed reference image by applying a mask to the feature map, the feature map corresponding to the reference image; and updating the kernel based on a value of a loss function between the first reconstructed reference image and a defect image.

[0018] The kernel may include a foreground kernel and a background kernel, and generating the feature map may include applying the foreground kernel to the reference image and applying the background kernel to the reference image.

[0019] The mask may include a foreground mask and a background mask, and generating a first reconstructed reference image may include forming an image of a first mask by applying the foreground mask to a feature map output by a foreground kernel, and forming an image of a second mask by applying the background mask to the feature map output by the background kernel.

[0020] A first reconstructed reference image may be generated based on a combination of the image of the first mask and the image of the second mask.

[0021] The mask may include a background mask corresponding to the foreground-background segmentation of the reference image.

[0022] The foreground mask can be trained based on the value of the loss function.

[0023] The foreground mask may be generated by applying a foreground-background segmentation model or rule to a reference image.

[0024] The kernel may include a background kernel, and the first reconstructed reference image has reduced background noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A pre-processing apparatus according to one or more embodiments is shown.

[0026] Figure 2 A training method for preprocessing images according to one or more embodiments is shown.

[0027] Figure 3 A method for training a preprocessing system according to one or more embodiments is shown.

[0028] Figure 4 An inference method performed by a trained kernel according to one or more embodiments is shown.

[0029] Figure 5 Training of a preprocessing system in accordance with one or more embodiments is shown.

[0030] Figure 6Training of a preprocessing system according to some embodiments is shown.

[0031] Figure 7 An inference method performed by a trained kernel according to one or more embodiments is shown.

[0032] Fig. 8A A defect inspection system for an electronic device according to one or more embodiments is shown.

[0033] Figure 8B Defects distinguished in an image by a defect inspection system are shown in accordance with one or more embodiments.

[0034] Fig. 9 A neural network is shown in accordance with one or more embodiments.

[0035] Fig.10 A pre-processing apparatus according to one or more embodiments is shown.

[0036] Throughout the drawings and detailed description, unless otherwise specified or provided, the same or similar reference numerals will be understood to refer to the same or similar elements, features and structures. The drawings may not be drawn to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0037] The following detailed description is provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be apparent. For example, the sequence of operations described herein is merely an example and is not limited to those set forth herein, but may be changed, as will be apparent after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, in order to increase clarity and conciseness, the description of features known after understanding the disclosure of the present application may be omitted.

[0038] The features described herein may be embodied in different forms and should not be construed as being limited to the examples described herein. However, the examples described herein are provided only to illustrate some of the many possible ways to implement the methods, devices and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0039] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "a", "an" and "the" are also intended to include plural forms. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As non-limiting examples, the terms "comprise" or "comprises", "include" or "includes" and "have" or "has" specify the presence of the features, numbers, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, numbers, operations, components, elements and / or combinations thereof.

[0040] Throughout the specification, when a component or element is described as being "connected to," "coupled to," or "engaged to" another component or element, it may be directly "connected to," "coupled to," or "engaged to" the other component or element, or one or more other components or elements may be reasonably interposed therebetween. When a component or element is described as being "directly connected to," "directly coupled to," or "directly engaged to" another component or element, there may not be other elements therebetween. Similarly, expressions such as "between" and "directly between," and "adjacent to" and "directly adjacent to" may also be interpreted as described above.

[0041] Although various components, components, regions, layers or parts may be described herein using terms such as "first", "second" and "third" or A, B, (a), (b), etc., these components, components, regions, layers or parts are not limited by these terms. Each of these terms is not used to define, for example, the essence, order or sequence of the corresponding component, component, region, layer or part, but is only used to distinguish the corresponding component, component, region, layer or part from other components, components, regions, layers or parts. Therefore, without departing from the teachings of the examples described herein, the first component, component, region, layer or part mentioned in the examples may also be referred to as the second component, component, region, layer or part.

[0042] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs and based on the understanding of the disclosure of the present application. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the disclosure of the relevant art and the present application, and should not be interpreted in an idealized or overly formal sense unless explicitly defined as such in this article. The use of the term "may" in relation to an example or embodiment herein (e.g., what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0043] Figure 1 A pre-processing system according to one or more embodiments is shown.

[0044] The source image and the target image (training pair) may each include a foreground and background of the same scene or object. The foreground may correspond to a region of interest (ROI), and the background may be other parts except the ROI. With respect to the source image and the target image, typically, the difference between the backgrounds of the corresponding source image and the target image is small (the backgrounds may be close to the same), however, the difference between the foregrounds (ROIs) of the corresponding images may be larger, that is, the foregrounds may not be close to the same. For example, when the target image is a defect image (including defects of an electronic device or other workpiece), the source image may serve as a reference image (ground truth image) that lacks defects and can be compared with the defect image when detecting defects in the target image.

[0045] In an embodiment, the preprocessing system 100 may preprocess the source image to minimize the difference between the background of the source image and the target image. For example, the preprocessing system 100 may denoise (reduce) the noise in the difference between the preprocessed source image and the target image, thereby increasing the signal-to-noise ratio (SNR) of the difference between the source image and the target image.

[0046] For this purpose, given an image training pair of a source image and a target image, the preprocessing system 100 can change the source image so that the source image is closer to the target image. When an area of ​​the target image that is different from the source image is determined to correspond to a defect, the defect inspection device can detect the defect in the target image.

[0047] refer to Figure 1 , the pre-processing system 100 may include a kernel generator 110 and a region designator 120. In an embodiment, the kernel generator 110 and / or the region designator 120 may be provided by a neural network (eg, a convolutional neural network).

[0048] The kernel generator 110 may generate a kernel (e.g., a foreground kernel or a background kernel) that is subsequently used to process an image. The neural network may update the kernel according to the operation result of the operation of the kernel on the image. In an embodiment, the kernel generator 110 may generate a foreground kernel and a background kernel, and the same image (source image) may be input to the foreground kernel and the background kernel to train the neural network (see Figure 6 ). The background kernel (as updated by learning of the neural network) can be used by the neural network in the non-training inference stage to generate pre-processed images (for defect detection in production / non-training images). However, the trained foreground kernel may not be used in the inference stage.

[0049] Further with respect to training, the region designator 120 may generate a mask for distinguishing / separating a background region and a foreground region in an image, and the mask may be updated based on the result of the neural network applying the mask to the image. In an embodiment, the region designator 120 may generate a foreground mask and a background mask, and the foreground mask and the background mask may be applied to an image (e.g., a source image) to train the neural network. The foreground kernel may be used to train the foreground mask, and the background kernel may be used to train the background mask. That is, the kernel (mentioned above) applied to the image may be trained based on the foreground region and the background region determined by the region designator 120.

[0050] The region designator 120 may generate a mask of a training image (e.g., a source image) according to a binary method, by which pixels of the mask (which respectively correspond to pixels of the image) initially have binary values ​​0 or 1. The pixel values ​​of the mask may then be updated to have numerical values ​​ranging from 0 to 1.

[0051] The region designator 120 may also generate a second mask (e.g., foreground) from a first mask (e.g., background) by generating a first mask (e.g., background) and then generating a copy of the first mask and inverting the copied mask to form a second mask (e.g., foreground). For example, the region designator 120 may generate a foreground mask by first generating a background mask and then subtracting the corresponding pixel values ​​of the background mask from 1 (in the copy of the background mask). For example, when the pixel value of a particular pixel of the background mask is 0.95, the pixel value of the corresponding pixel of the foreground mask may be 0.05. Alternatively, the region designator 120 may generate a background mask by generating a foreground mask and subtracting the corresponding pixel values ​​of the generated foreground mask from 1.

[0052] Figure 2 Training of preprocessing images through a neural network is shown in accordance with one or more embodiments. Figure 3 A method of training a neural network according to one or more embodiments is shown. Figure 4A method of performing inference with a trained kernel according to one or more embodiments is shown.

[0053] In an embodiment, for a training pair (source image and target image), the preprocessing system 100 can generate a reconstructed source image (i.e., a reconstructed version of the source image) by applying a filter (e.g., a kernel) to the source image, and can update the weights of the filter to minimize the value of the loss function calculated based on the reconstructed version of the source image (compared to the target image). The source image can be repeatedly reconstructed by the corresponding updated filter until a predetermined criterion (e.g., a maximum number of iterations or a minimum degree of loss) is met. For example, the weights can be updated by back propagation.

[0054] In an embodiment, the loss function may be expressed as a function indicating the difference between the reconstructed source image and the target image.The filter may include a kernel and a mask (multiple kernels and masks are possible).

[0055] refer to Figure 2 and Figure 3 In summary, a kernel and a mask can be applied to a source image to generate a reconstructed version of the source image. A loss function can be calculated by the difference between the reconstructed source image and the target image, and then the weights of the kernel and / or the mask can be updated according to the value of the loss function.

[0056] To further summarize, upon completion of the kernel and / or mask update, the pre-processing system 100 may generate a pre-processed image from the source image in an inference phase that uses the updated filter. Figure 2 and Figure 4 , when the trained kernel is applied to the source image in the inference phase, a source image preprocessed in this way can be generated. Since the difference between the background of the preprocessed source image and the background of the target image is minimized by preprocessing using the trained kernel, the region of interest can be more accurately determined in the target image based on the preprocessed source image (due to improved SNR).

[0057] For details, refer to Figure 2 , when an image pair of a source image and a target image is generated ( S110 ), the pre-processing system 100 may output a feature map (or characteristic map) of the source image by applying the kernel to the source image ( S120 ).

[0058] In an embodiment, the convolution operation of the kernel image may be performed on the source image (by Figure 3 When the kernel is smaller than the source image, the kernel may perform an operation on the corresponding pixel values ​​of the image according to a stride having a predetermined stride size. That is, the kernel may be convolved on the image. The preprocessing system 100 may adjust the padding and the stride size attached to the source image so that the feature map may be output with the same size as the source image.

[0059] In an embodiment, the kernel applied to the source image may be an n×n kernel generated by a neural network or a Gaussian kernel.

[0060] refer to Figure 2 , the pre-processing system 100 may generate a reconstructed version of the source image by applying the mask to the feature map output by the kernel (S130). Figure 3 , a dot product operation can be performed on the feature map and the mask (in Figure 3 For example, pixels in the feature map may be multiplied by their respective counterpart pixels in the mask.

[0061] refer to Figure 2 , the preprocessing system 100 may update the weight of the kernel based on the value of the loss function calculated according to the reconstructed source image (S140). In an embodiment, the mask may be generated by a neural network or a predetermined rule. When the mask is generated by a neural network, the preprocessing system 100 may update the weight of the mask based on the value of the loss function calculated according to the reconstructed source image. When the mask is generated according to a predetermined rule, the mask may not be updated. Equation 1 represents the reconstructed source image (S Rec ) and the target image (T).

[0062] Equation 1

[0063] Minimize L1 Loss(|S Rec -T|)

[0064] refer to Figure 2 , the preprocessing system 100 can determine whether the latest iteration has caused the value of the loss function (calculated based on the reconstructed source image) to be within a predetermined range, and thus can determine whether to re-execute training (S150). For example, when the value of the loss function calculated based on the reconstructed source image is not within the predetermined range, the preprocessing system 100 can start the next epoch of training, or when the value of the loss function calculated based on the reconstructed source image is within the predetermined range, the preprocessing system 100 can end the training. Alternatively, when a predetermined number of epochs have been executed (for example, for the purpose of fast processing), the preprocessing system 100 can terminate the training, and the core becomes available to perform the inference phase.

[0065] In the inference phase, the preprocessing system 100 may generate a preprocessed source image by applying the trained kernel to the source image and output the thus preprocessed source image and the target image as a new image pair ( S160 ).

[0066] Figure 5A method for training a neural network according to one or more embodiments is shown.

[0067] refer to Figure 5 , a reconstructed source image can be generated by applying the kernel and the mask to the same source image and then combining the corresponding feature maps obtained by applying the kernel and the mask to the source image.

[0068] The pre-processing system 100 may apply a foreground kernel and a foreground mask to the source image, and may apply a background kernel and a background mask to the source image.

[0069] In an embodiment, a foreground kernel may be applied to a source image, a first feature map of the source image may be output, and a foreground mask may be applied to the first feature map generated by the foreground kernel, thereby updating the first feature map. In addition, a background kernel may be applied to the source image, thereby forming a second feature map of the source image, and a background mask may be applied to the second feature map generated by the background kernel. In this way, the foreground mask and the background mask may allow the foreground kernel and the background kernel to learn the foreground and background of the source image, respectively.

[0070] The kernel generator 110 may apply a predetermined condition to generate a foreground kernel and a background kernel in a fixed manner. The predetermined condition of the kernel generator 110 may be as shown in Equation 2.

[0071] Equation 2

[0072]

[0073] Referring to Equation 2, μ is a mean value, S is a source image, and ∈ is a residual.

[0074] In an embodiment, for a set of source-target image pairs, when the difference between a source image and its corresponding target image generally falls within a predetermined image region (e.g., a region where defects generally occur), a rule-based mask may be used. For example, when the location of the difference between the source image and the target image is predetermined to be generally provided in a 7×7 region having a location close to the middle of the image, the foreground mask may only / primarily pass-through the portion of the first feature map located in the 7×7 region (i.e., in the middle of the image), and the background mask may only / primarily pass-through the portion of the second feature map that does not include the 7×7 region.

[0075] In another embodiment, a foreground mask may be applied before the foreground kernel is applied (in Figure 5In the example, the position of the kernel may be swapped with the position of the mask. That is, the source image is applied to the foreground mask, and then the result is applied to the foreground kernel. Similarly, the source image is applied to the background mask, and the result is applied to the background kernel. The preprocessing system 100 may combine the two images (feature maps) outputted respectively according to the foreground kernel and the background kernel, and may generate a reconstructed source image (combination).

[0076] The preprocessing system 100 may update the weights of the foreground kernel and the background kernel based on the value of the loss function (a loss function indicating the difference between the reconstructed source image and the corresponding target image). The preprocessing system 100 may update the foreground mask and / or the background mask based on the value of the loss function calculated from the reconstructed source image.

[0077] The preprocessing system 100 may update the foreground mask and / or the background mask based on the value of the loss function calculated from the reconstructed source image. In a non-limiting example, the foreground mask may be updated based on the value of the loss function, and then the background mask may be updated by inverting the foreground mask (e.g., subtracting the foreground mask from 1). For example, in the case where the foreground mask can be updated more easily than the background mask based on the value of the loss function, the foreground mask may be updated based on the value of the loss function, and then the background mask may be updated by inverting the foreground mask. Alternatively, in the case where the capacity of the foreground mask is less than the capacity of the background mask, the foreground mask may be updated based on the value of the loss function, and then the background mask (which will be used in the next period) may be the inverted foreground mask.

[0078] The preprocessing system 100 may determine whether to start training again based on a determination of whether the value of the loss function calculated based on the reconstructed source image is within a predetermined range (or satisfies a threshold). For example, when the value of the loss function is not within the predetermined range, the preprocessing system 100 may start the next epoch. However, when the value of the loss function is within the predetermined range, the preprocessing system 100 may end the training. Alternatively, when the epoch is executed a predetermined number of times, the preprocessing system 100 may end the training and may output the trained background kernel for performing the inference phase.

[0079] Figure 6 A method of training a neural network according to one or more embodiments is shown.

[0080] refer to Figure 6 , the mask and kernel are applied to the same source image, and the images / maps output by the mask and kernel are combined, so the source image is reconstructed.

[0081] The pre-processing system 100 may apply a foreground mask and a foreground kernel to the source image, and may also apply a background mask and a background kernel to the source image.

[0082] In an embodiment, a foreground mask may be applied to a source image, and the result thereof may be applied to a foreground kernel, so that a feature map is output from the foreground kernel. In addition, a background mask may be applied to a source image, and the result thereof may be applied to a background kernel, so that a feature map is output from the background kernel. Thus, the foreground kernel and the background kernel may learn the parts filtered by the foreground mask and the background mask.

[0083] The preprocessing system 100 can combine the two feature map images output by the foreground kernel and the background kernel, and can generate a reconstructed source image. Because the mask is applied before the kernel, additional elements can be included in the preprocessing system 100 for correcting distortions that may be generated in the corresponding regions (foreground and background) of the feature map images. For example, the preprocessing system 100 can also include a boundary processing device and / or a neural network (not shown) for processing the boundary of the foreground and background.

[0084] The preprocessing system 100 may update the weights of the foreground kernel and / or the background kernel based on the value of the loss function indicating the difference between the reconstructed source image and the target image. The preprocessing system 100 may update the foreground mask and / or the background mask based on the value of the loss function calculated from the reconstructed source image.

[0085] The preprocessing system 100 may update the foreground mask and / or the background mask based on the value of the loss function of the reconstructed source image. For example, the foreground mask may be updated based on the value of the loss function, and the background mask may be updated by inverting the foreground mask. When the capacity of the foreground mask is less than the capacity of the background mask, the foreground mask may be updated based on the value of the loss function, and the background mask (which will be used in the next period) may be an inverted foreground mask.

[0086] The preprocessing system 100 may determine whether to start training again based on a determination of whether the value of the loss function calculated based on the reconstructed source image is within a predetermined range or satisfies a threshold. For example, when the value of the loss function is not within a predetermined range, the preprocessing system 100 may start the next epoch. When the value of the loss function is within a predetermined range, the preprocessing system 100 may terminate the training. Alternatively, when a predetermined number of epochs are executed, the preprocessing system 100 may terminate the training and may output the trained background kernel for performing an inference phase.

[0087] Figure 7 An inference method through a trained kernel according to some embodiments is shown.

[0088] refer to Figure 7, the preprocessing system 100 can generate a preprocessed source image by applying the trained background kernel to the source image. The background kernel can be trained by any of the above examples or embodiments. In an embodiment, when reconstructing the source image, the preprocessing system 100 can determine whether to terminate the update of the kernel. The preprocessing system 100 can update the kernel again based on the value of the loss function between the reconstructed source image and the target image, or can end the training performed by the update of the kernel. Alternatively, when the source image is reconstructed a predetermined number of times, the preprocessing system 100 can terminate the training performed by the update of the kernel.

[0089] When the training is finished, the preprocessing system 100 can preprocess the source image in the inference phase by applying the trained background kernel to the source image. The preprocessed source image can be used to detect defects in the target image.

[0090] Fig. 8A A defect inspection system for an electronic device according to one or more embodiments is shown. Figure 8B Defects distinguished in an image by a defect inspection system are shown in accordance with one or more embodiments.

[0091] In an embodiment, the defect inspection system 200 may photograph an electronic device to obtain a defect image, and may determine defects in the defect image by using a reference image corresponding to the defect image. The reference image may be an image of an electronic device that does not include defects (not necessarily the same physical electronic device, but a copy). The defect inspection system 200 may be applied to the field of defect detection using defect data and reference data. For example, the defect inspection system 200 may be used in a semiconductor manufacturing process.

[0092] refer to Fig. 8A , the defect inspection system 200 may include an image pair generating device 210 , a pre-processing device 220 , and a defect inspection device 230 .

[0093] The image pair generating device 210 may obtain a defect image by photographing a suspected defective portion of the electronic device, and may generate an image pair of the defect image and a reference image, wherein the reference image is generated to correspond to the defect image.

[0094] Specifically, the preprocessing device 220 can convert a reference image (corresponding to the defect image) into a reconstructed reference image by using a filter, and can generate a preprocessed version of the reference image, wherein the noise of the background of the reference image is removed or reduced based on the difference between the reconstructed reference image and the defect image. The filter may include at least one kernel and at least one mask.

[0095] In an embodiment, the preprocessing device 220 may apply a background kernel and a foreground kernel to a reference image, and may apply a background mask and a foreground mask to a feature map output by the background kernel and the foreground kernel, respectively. The preprocessing device 220 may apply a background mask to a feature map output by the background kernel, and may apply a foreground mask to a feature map output by the foreground kernel. The reference image may be reconstructed by combining two images / maps masked by the background mask and the foreground mask. The preprocessing device 220 may update (train) the background kernel and the foreground kernel based on the value of the loss function of the difference between the reconstructed reference image and the defective image. Then, in order to continue training, the previously updated kernel may be applied again to the reference image to reconstruct the reconstructed reference image, and the previously updated kernel may be updated again based on the reconstructed reference image. When the update of the kernel is completed, in the inference phase, the preprocessing device 220 may apply the updated background kernel to the reference image to generate a so-preprocessed version of the reference image.

[0096] The defect inspection apparatus 230 may detect defects from the defect image by comparing the preprocessed reference image with the defect image.

[0097] refer to Figure 8B , it can be seen from the difference image (which is the difference between the pre-processed reference image and the defect image) that the difference in the background of the pre-processed reference image is reduced compared to the difference with the non-pre-processed reference image, as indicated by the prominence / brightness of the generally darker (no difference) areas and the areas identified as defects. The lower background difference between the pre-processed reference image and the defect image gives greater prominence (signal strength) to the difference of the defect.

[0098] Fig. 9 A neural network is shown in accordance with one or more embodiments.

[0099] refer to Fig. 9 , the neural network (NN) 900 may include an input layer 910, a hidden layer 920, and an output layer 930. The input layer 910, the hidden layer 920, and the output layer 930 may each include a corresponding set of nodes, and the strength of the connection between the nodes may correspond to a weight (connection weight). The nodes included in the input layer 910, the hidden layer 920, and the output layer 930 may be connected to each other using a fully connected type architecture. The number of parameters (weights and biases) may be equal to the number of connections in the neural network 900.

[0100] The input layer 910 may include input nodes (x 1 to x i ), and the input node (x 1 to x i) may correspond to the number of independent input variables. Based on inputting a training data set into the input layer 910 for the purpose of training the neural network 900 and inputting a test data set into the input layer 910 of the trained neural network 900, the output layer 930 of the trained neural network 900 may output an inference / reasoning result.

[0101] The hidden layer 920 may be located between the input layer 910 and the output layer 930, and may include at least one hidden layer (920 1 to 920 n The output layer 930 may include at least one output node (y 1 To j ). Activation functions may be used in the hidden layer 920 and the output layer 930 to determine node output / activation. In an embodiment, the neural network 900 may be learned by updating the weights of the hidden nodes included in the hidden layer 920.

[0102] Fig.10 A pre-processing apparatus according to one or more embodiments is shown.

[0103] The pre-processing device may be implemented as a computer system, for example, a computer readable medium (but not the signal itself). Fig.10 , the computer system 1000 may include at least one processor 1010 and at least one memory 1020. The memory 1020 may be connected to the processor 1010, and may store various information for driving the processor 1010 or at least one program executed by the processor 1010.

[0104] The processor 1010 may implement the functions, stages or methods proposed in the embodiments. The operation of the computer system 1000 may be implemented by the processor 1010. The at least one processor 1010 may include at least one of a CPU, a GPU and an NPU. In practice, the processor 1010 may be one or more processors of one or more types.

[0105] The memory 1020 may be provided inside and / or outside the processor and may be connected to the processor by various known means. The memory may represent various forms of volatile or nonvolatile storage media, and for example, the memory may include a read-only memory (ROM) and a random access memory (RAM).

[0106] This article is about Figure 1-Figure 9The computing devices, electronic devices, processors, memories, image sensors, displays, information output systems and hardware, storage devices, and other devices, devices, units, modules, and components described are implemented by hardware components or represent hardware components. Where appropriate, examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer can be implemented by one or more processing elements, such as a logic gate array, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result. In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or the processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors or a processor and a controller may implement a single hardware component or two or more hardware components.The hardware components may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instructions single-data (MISD) multiprocessing, and multiple-instructions multiple-data (MIMD) multiprocessing.

[0107] Perform the operations described in this application Figure 1-Figure 9 The method shown in is performed by computing hardware, for example, by one or more processors or computers, implementing instructions or software as described above to perform the operations described in the present application performed by the method. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0108] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by hardware components and methods as described above. In one example, the instructions or software include machine code directly executed by one or more processors or computers, such as machine code generated by a compiler. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flow charts shown in the accompanying drawings that disclose algorithms for performing operations performed by hardware components and methods as described above, and the corresponding descriptions of this article.

[0109] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), micro or card such as a multimedia card (e.g., secure digital (SD) or extreme digital (SD)) or a hard disk drive (HDD). The invention relates to a card-type memory, a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid-state disk, and any other device of a digital, XD) type, which is configured to store instructions or software and any associated data, data files, and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files, and data structures to one or more processors or computers so that one or more processors or computers can execute instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed on a network-coupled computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed manner.

[0110] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered illustrative only and not for purposes of limitation. The description of features or aspects in each example is considered to be applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents.

[0111] Therefore, in addition to the above disclosure, the scope of the present disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as being included in the present disclosure.

Claims

1. A method for preprocessing a source image performed by one or more processors, the method comprising: generating a feature map by applying at least one kernel to the source image; generating a first reconstructed source image by applying at least one mask to the feature map, the feature map corresponding to the source image; as well as The at least one kernel is updated based on a value of a loss function between the first reconstructed source image and a target image.

2. The method according to claim 1, wherein: The at least one kernel comprises a foreground kernel and a background kernel, and wherein the generating of the feature map comprises: applying the foreground kernel to the source image; and applying the background kernel to the source image, wherein the feature map comprises a first feature map generated by applying the foreground kernel to the source image and a second feature map generated by applying the background kernel to the source image, and wherein the generating of the first reconstructed source image comprises: applying a foreground mask to the first feature map; and applying a background mask to the second feature map, Wherein, the generating of the first reconstructed source image further comprises: forming a first image by applying the foreground mask to the first feature map; forming a second image by applying the background mask to the second feature map; and The first image is combined with the second image.

3. The method according to claim 1, further comprising: generating a second reconstructed source image by applying an updated version of the kernel and the mask to the source image, the updated version of the kernel being generated based on the kernel being updated according to the value of the loss function; as well as Based on the second reconstructed source image being generated, determining whether to end updating the core, wherein the determination of whether to end updating the kernel comprises updating the updated kernel based on a value of a loss function between the second reconstructed source image and the target image, or ending training of the update by the kernel, or Wherein, the determining whether to end the updating of the kernel comprises ending the training of the updating by the kernel based on the source image being reconstructed a predetermined number of times. 4 . The method of claim 3 , further comprising pre-processing the source image by using a trained background kernel based on the end of the updating of the kernel being determined.

5. A method for detecting a defect in a defect image, the defect image having a corresponding reference image without the defect, the method being performed by one or more processors and comprising: applying a trained background kernel to the reference image to generate a preprocessed reference image, the trained background kernel being trained based on applying a background kernel and a mask to the reference image; as well as The defect in the defect image is detected based on a comparison between the preprocessed reference image and the defect image.

6. The method according to claim 5, further comprising: outputting a feature map by applying the background kernel to the reference image, applying the mask to the feature map to generate a reconstructed reference image, the reconstructed reference image being a reconstruction of the reference image; as well as updating the background kernel based on the reconstructed reference image, Wherein, the output of the feature map includes: applying a foreground kernel to the reference image; and applying the background kernel to the reference image, wherein applying the foreground kernel to the reference image generates a first feature map of the feature map, wherein applying the background kernel to the reference image generates a second feature map of the feature map, and wherein the reconstruction of the reference image comprises: applying a foreground mask of the mask to the first feature map to generate a feature map of the first mask; applying a background mask of the mask to the second feature map to generate a feature map of a second mask; and The reference image is reconstructed based on a combination of the feature map of the first mask and the feature map of the second mask.

7. A device for preprocessing a reference image, the device comprising: one or more processors; A memory storing instructions, wherein the instructions are configured to cause the one or more processors to perform a process, the process comprising: generating a feature map by applying a kernel to the reference image; generating a first reconstructed reference image by applying a mask to the feature map, the feature map corresponding to the reference image; and The kernel is updated based on a value of a loss function between the first reconstructed reference image and the defect image.

8. The device according to claim 7, wherein: The kernel comprises a foreground kernel and a background kernel, wherein generating the feature map comprises applying the foreground kernel to the reference image and applying the background kernel to the reference image, wherein the mask comprises a foreground mask and a background mask, and wherein generating the first reconstructed reference image comprises forming a first masked image by applying the foreground mask to a feature map output by the foreground kernel, and forming a second masked image by applying the background mask to a feature map output by the background kernel, and The first reconstructed reference image is generated based on a combination of the first mask image and the second mask image.

9. The device according to claim 7, wherein: The mask comprises a background mask corresponding to a foreground-background segmentation of the reference image, wherein the foreground mask is trained based on the value of the loss function, and / or The foreground mask is generated by applying a foreground-background segmentation model or rule to the reference image.

10. The device according to claim 7, wherein: The kernel comprises a background kernel, and wherein the first reconstructed reference image has reduced background noise.

Images