Method for generating reconstructed images
Through memory-enhanced autoencoder and sparse addressing mechanism, the excessive generalization problem of autoencoder in defect detection is solved, and high-accurate defect detection is achieved in noise environments, which improves the robustness of defect detection.
Patent Information
- Application Number
- CN202011298764.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-19
- Publication Date
- 2025-08-12
- Estimated Expiration
- 2040-11-19
AI Technical Summary
The existing defect detection methods based on autoencoders have problems with overgeneralization, especially when the product surface has a large number of textures, the reconstruction image is unstable and is overly sensitive to noise, making it difficult to effectively detect unlabeled defects in production equipment.
The memory-enhanced autoencoder is used to combine sparse addressing and trust area memory update mechanisms to convert the input image into feature maps through the encoder, and the feature prototype and weight in memory are used to generate an approximate feature map, and the image is reconstructed through the decoder to avoid excessive generalization of memory slots and defective feature pollution.
It realizes effective detection of defects in a noise environment, avoids false positive detection, improves the accuracy and robustness of defect detection, and maintains a high accuracy rate under high noise ratio.
Smart Images
Figure CN114596375B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image-based product defect detection, and more particularly to a method for generating and reconstructing images applied to a defect detection front end. Background Art
[0002] For manufacturers, product appearance assessment is a fundamental step in quality assurance. Undetected defects such as scratches, impacts, and discoloration can increase product return repair costs and erode customer trust. Currently, most appearance inspections are performed by humans due to the difficulty of using traditional computer vision algorithms to characterize various defects in automated optical inspection (AOI) machines. However, managing human inspectors presents management challenges, as it's difficult to maintain consistent inspection standards across different product lines.
[0003] Object detector networks have been proposed to address this problem. However, these fully supervised models require datasets with clearly labeled bounding boxes. Collecting such datasets is labor-intensive and maintaining consistent labeling is challenging. Furthermore, these methods perform poorly on defects not present in the dataset. Gathering sufficient training data for a wide range of defect types is time-consuming, making them unsuitable for products with short lifecycles.
[0004] Rather than relying on correctly labeling defects, another approach to defect detection is to use a generative approach that learns the distribution of a standard sample and identifies areas that deviate significantly from the sample as defects. This approach allows the model to detect defects it has not previously seen. For example, an autoencoder trained using standard images can eliminate defects from the input image. However, in practice, the autoencoder may learn to reconstruct defects due to overgeneralization. In particular, when the product surface has a lot of texture, the reconstructed image may become unstable, resulting in many false positives in subsequent detections.
[0005] As mentioned above, this generative approach does not require carefully labeled images, but it does assume that the input images are defect-free. Therefore, when defective images are accidentally mixed into the training dataset, the algorithm becomes overly sensitive to noise. This situation often occurs in production equipment. Furthermore, many input images are imperfect. Excluding these imperfect images would significantly reduce the number of standard image samples available. Therefore, a training method based on deep learning that is tolerant to data noise is needed, which can train on defect-free image regions. Summary of the Invention
[0006] In view of the above-mentioned shortcomings of the prior art, the object of the present invention is to propose a method for generating a reconstructed image to solve the over-generalization problem commonly seen in existing autoencoder-based anomaly detection methods.
[0007] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present invention provides a method for generating a reconstructed image, which is applicable to an input image having a target object, and the method for generating a reconstructed image includes: converting the input image into a feature map having multiple feature vectors using an encoder; executing a training program based on multiple training images of multiple reference objects to generate multiple feature prototypes associated with the training images and storing the feature prototypes in a memory; selecting a portion of the feature prototypes from the memory based on multiple similarities between the feature prototypes and the feature vectors; generating an approximate feature map based on the portion of the feature prototypes and multiple weights, wherein the weights each represent the similarity between the feature prototypes and the feature vectors; and converting the approximate feature map into a reconstructed image using a decoder; wherein the encoder, the decoder and the memory form an autoencoder.
[0008] In an embodiment of the first aspect, selecting the portion of the feature prototypes from the memory based on the similarity between the feature prototype and the feature vector includes: selecting a portion of the weights from large to small from the weights; selecting the portion of the feature prototypes corresponding to the portion of the weights; calculating multiple sparse approximations based on the number of the portion of the weights; and using the sparse approximations as the weights for each.
[0009] In an embodiment of the first aspect, executing the training program based on the training image of the reference object includes: setting a default feature prototype; for each of the training images, dividing the training image into multiple blocks; using the encoder to convert the blocks into multiple block features respectively; calculating multiple distances between the block features and the default feature prototype; retaining the block features corresponding to one or more distances of the multiple distances that are less than a threshold; and updating the default feature prototype as one of the feature prototypes based on the retained block features.
[0010] In an embodiment of the first aspect, executing the training program based on the training image of the reference object includes: setting a default feature prototype; for each of the training images, converting the training image into a plurality of block features using the encoder; calculating a plurality of distances between the block features and the default feature prototype; retaining the block features corresponding to one or more distances of the plurality of distances that are less than a threshold; and updating the default feature prototype as one of the feature prototypes based on the retained block features.
[0011] In an embodiment of the first aspect, the weight is calculated using the following formula: where ω i is the i-th weight, z is one of the eigenvectors, M i is the i-th feature prototype in the memory, and M is the number of memory slots in the memory.
[0012] In an embodiment of the first aspect, the threshold is an average value of the multiple distances.
[0013] In an embodiment of the first aspect, a loss function of the autoencoder is associated with the following formula: trust =r(z,M (1) )||zM (1) ||2, where L trust is one of the loss functions, z is one of the feature vectors, M (1) is the one closest to z among the feature prototypes, and r(z,M (1) ) is used to indicate z and M (1) Is the distance less than the threshold?
[0014] In one embodiment of the first aspect, r(z,M (1) ) is calculated using the following formula: Wherein δ1 is the threshold value, and δ2 is another threshold value greater than δ1.
[0015] In summary, the method for generating reconstructed images proposed in the present invention has the following contributions or benefits: The defect classifier implemented using the method is robust to noise in the training dataset. The sparse addressing scheme proposed in the present invention prevents overgeneralization of the autoencoder's memory slots. The trust region memory update scheme proposed in the present invention prevents memory slot contamination by defective images during the training phase.
[0016] The above description of the content of the present invention and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide further explanation of the scope of the patent application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Shown is a structural diagram of a reconstruction model in one embodiment of the present invention;
[0018] Figure 2 Shown is a flow chart of a method for generating a reconstructed image according to an embodiment of the present invention;
[0019] Figure 3 The detailed flow chart shown is step S3;
[0020] Figure 4 Shown as a schematic diagram of the trust region;
[0021] Figure 5 and Figure 6 Detailed flow charts showing two implementations of step S1; and
[0022] Figure 7 Shown is a comparison diagram of the reconstruction model proposed in the present invention and other models in resisting noise data.
[0023] Component number description
[0024] 10 Memory-enhanced Autoencoders
[0025] E-encoder
[0026] D decoder
[0027] M Memory / Feature Prototype
[0028] Z feature map
[0029] z i Eigenvector
[0030] Approximate feature map
[0031] Approximate eigenvectors
[0032] TR Trust Zone
[0033] z Block characteristics
[0034] Steps S1 to S5
[0035] Steps S31 to S33
[0036] Steps S11 to S16, S13 DETAILED DESCRIPTION
[0037] The following detailed description of the features and characteristics of the present invention is sufficient to enable anyone skilled in the art to understand the technical content of the present invention and implement it accordingly. Based on the disclosure of this specification, the scope of the claims, and the drawings, anyone skilled in the art can easily understand the relevant concepts and features of the present invention. The following examples are intended to further illustrate the concepts of the present invention and are not intended to limit the scope of the present invention in any way.
[0038] The present invention provides a method for generating a reconstructed image using a reconstruction model provided in one embodiment of the present invention and an input image. The input image is an image of a target object, such as a circuit board or a laptop computer cover. The target object may have defects such as scratches, impacts, or discoloration. The reconstructed image can be considered a defect-free input image.
[0039] An unsupervised defect detection process can be summarized as follows: a reconstruction model generates a reconstructed image based on an input image. The reconstructed image is compared with the input image to determine if the difference is greater than a threshold. As can be seen from the above, the performance of a defect detector depends on the performance of the reconstruction model.
[0040] Figure 1 FIG2 is an architecture diagram of a reconstruction model according to an embodiment of the present invention. The reconstruction model mainly adopts a memory augmented auto encoder (MemAE) 10 and combines sparse memory addressing and trust region memory update mechanisms proposed by the present invention.
[0041] The memory-augmented autoencoder 10 stores prototypes of standard image features in memory slots and can reconstruct a standard version of the input image from memory. Sparse addressing provides a memory slot selection mechanism. A trust region memory update mechanism prevents the storage of unwanted defective features. Figure 2 FIG. 1 is a flow chart of a method for generating a reconstructed image according to an embodiment of the present invention. Figure 2Steps Figure 1 components of the .
[0042] Step S1 is "Executing a training process based on multiple training images to generate multiple feature prototypes and storing them in memory." This is the training phase of the reconstruction model. The multiple training images are images of multiple reference objects. These reference objects and the target object are all of the same type, for example, laptop computer covers. Compared to the target object, the reference objects are defect-free, so the training images are standard images without defects. Feature prototypes are associated with these training images.
[0043] Step S2 is “the encoder converts the input image into a plurality of feature vectors.” Step S2 is the inference stage of the reconstruction model.
[0044] The present invention utilizes an external memory M to store a set of standard feature prototypes to prevent defects from being reconstructed. These feature prototypes are used to generate the reconstructed image. During the inference phase, these feature prototypes are fixed, making it difficult for the autoencoder to reconstruct defects because memory M only contains standard features.
[0045] like Figure 1 As shown, the standard autoencoder follows the encoder-decoder structure. The encoder E takes the input image x∈R with a length of H and a width of W. H×W Projected to a low-dimensional latent space. The memory M is implemented as a tensor M∈R M×Z , where M represents the number of memory slots and Z represents the dimension of the latent vector z.
[0046] like Figure 2 As shown in step S2, the present invention first calculates the latent representation Z = E(x) of the input image x. It should be noted that in order to preserve spatial information, the present invention is designed to allow the encoder E to output a feature map Z∈R h×w×Z , and the dimension of the feature map is lower than that of the original image. The feature map is the result of mapping the feature vector to the latent space. For convenience, the present invention uses z i ∈R Z to represent the i-th element in Z. Each vector z in Z i Represents the features of a patch in the input image x, z i That is, the feature vector described in step S2.
[0047] Step S3 is “selecting multiple feature prototypes from the memory based on the similarity between the feature prototype and the feature vector.” The implementation details of step S3 will be described in detail later when describing sparse addressing.
[0048] Step S4 is “generating an approximate feature map based on these feature prototypes and multiple weights.” The approximate feature map is composed of multiple approximate feature vectors.
[0049] The present invention does not directly transmit the feature map Z to the decoder D, but uses the convex combination of feature prototypes stored in the memory M to calculate each z i Approximate characteristics of The above operation is defined as follows: where ω is a weight vector representing the similarity between z and each feature prototype stored in memory M.
[0050] Formula 1:
[0051] In step S3, the weight vector ω acts as a soft addressing mechanism to retrieve the closest feature prototype necessary for reconstruction from the memory. The present invention uses the negative Euclidean distance to measure the distance between the feature vector z and the memory item M. i The similarity between them is calculated and the weights are normalized by applying the normalization function softmax, as shown in the following method 2. Each memory item stores a feature prototype.
[0052] Formula 2:
[0053] Step S5 is “Decoder converts the approximate feature map into a reconstructed image”. Specifically, the decoder D only uses the approximate features derived from the memory items. Output reconstructed image
[0054] Please refer to Figure 3 , showing the detailed process of step S3. Step S31 is "selecting a number of weights and corresponding feature prototypes from large to small", step S32 is "calculating a number of sparse approximations based on the number of these weights", and step S33 is "using these sparse approximations as weights".
[0055] The implementation details of step S3 are described below. Enforcing sparsity in memory addressing forces the reconstruction model to use fewer, but more relevant, memory items to approximate the feature vector z. This approach effectively prevents the reconstruction model from unexpectedly merging multiple irrelevant memory items and resulting in reconstruction defects. Furthermore, this approach implicitly performs memory selection, saving computational effort by removing memory items that are never accessed when reconstructing the image.
[0056] As shown in step S31, assuming ω (i)Represents the ordering of the weight vector ω, where ω (1) >ω (2) >...>ω (M) As shown in step S32, the present invention calculates a sparse approximation of a weight vector ω This corresponds to obtaining the k closest memory items and then performing a re-normalization step, as shown in the following method 3 and step S33, where the l function returns 1 when the conditional expression behind it is met, and returns 0 otherwise.
[0057] Formula 3:
[0058] Since the present invention only uses a small number of selected memory items for reconstruction, it is necessary to avoid the reconstruction model from learning redundant memory items. Therefore, for the input potential vector z, the present invention selects the closest memory item M (1) and the second closest memory item M (2) Add a border, as shown in method 4 below.
[0059] Formula 4: L margin =[||zM (1) ||2-||zM (2) ||2+1] + .
[0060] The following describes the mechanism by which the trust zone updates memory.
[0061] Without the assumption that the training dataset only contains standard samples, the memory-enhanced autoencoder 10 will treat the defect samples as standard samples and learn to store defect features in memory, resulting in poor defect detection performance. The present invention utilizes two important concepts to prevent defect samples from contaminating the memory. (1) Defects are relatively rare, and defects do not all appear in the same location. This means that the proportion of defects at the block level is much lower than the proportion of defects at the image level. (2) The standard image (i.e., the training image described in step S1) has regularity in appearance, and the memory-enhanced autoencoder 10 is more likely to reconstruct the standard image than the defects in the initial training stage. The above implies that in the initial stage, the standard features are closer to the memory items than the defect features.
[0062] Figure 4 It is shown as a schematic diagram of the trust region TR. Figure 4 As shown in FIG, the trust region TR encircles the memory item M and 6 block features z, and the remaining 3 block features are outside the trust region TR. Based on the above concept, the present invention updates the memory item based on a specific trust region, which pulls the features in the region toward the closest memory item and pushes the features outside the region away from the memory item, as shown in FIG. Figure 4 In the direction of the arrow.
[0063] The “training procedure” described in step S1 is actually a process of optimizing each feature prototype. Figure 5 and 6 , which illustrate two implementations of step S1.
[0064] Step S11 is “setting the default feature prototype”, that is, starting from the first training image, each memory slot is initialized.
[0065] Step S12 is “segmenting the training image into a plurality of blocks”. For example, the training image of the laptop cover is divided into nine grids, where each grid represents a block.
[0066] Step S13 is "the encoder converts these blocks into multiple block features", the distribution of these block features is as follows Figure 4 shown.
[0067] Step S14 is “calculating multiple distances between these block features and the default feature prototypes”.
[0068] Step S15 is to “retain one or more block features corresponding to one or more distances less than a threshold value among these distances”. The threshold value is the average value of the multiple distances calculated in step S14 and is equal to Figure 4 The radius of the plotted trust region.
[0069] Step S16 is "Updating the default feature prototype based on the retained one or more block features." Specifically, the present invention divides the trust region TR with a specified radius, centered around the memory item M, to distinguish the feature space within the trust region TR from the feature space outside the trust region, as shown in Method 5 below. All items within δ1 are considered standard features and are pulled closer together. To prevent the model from pushing defective features to infinity, the present invention ignores items that exceed a preset trust threshold δ2. This corresponds to the process of steps S15-S16.
[0070] Formula 5:
[0071] The distance between the image blocks that are easy to reconstruct and the memory slot is usually smaller than the distance between the image blocks that are difficult to reconstruct and the memory slot, so δ1 must be suitable for these situations. The present invention first calculates multiple features z corresponding to all blocks of the current input image i With each memory item M iThe multiple distances between the two items are calculated, as described in step S14; δ1 is then set to the average of these distances, as described in step S15, to obtain the closest one or more memory items, and these memory items are updated, as described in step S16. Because standard features are abundant and similar to each other, they are mostly pulled close to the memory items, with only a few being pushed out. However, defective features will always be pushed away because the distance between defective features and memory items is often greater than the average distance. This method can prevent defective features from contaminating memory items.
[0072] Please refer to Figure 6 , shown as another implementation of step S1. Figure 6 and Figure 5 The difference is Figure 6 There is no step S12, and Figure 6 Step S13' and Figure 5 Step S13 is different. Figure 5 In the embodiment of the present invention, step S12 is to actually divide the training image into a plurality of blocks, such as blocks 1 to 9, and then in step S13, the features of each of blocks 1 to 9 are calculated. In other words, the encoder obtains block features at the block level. On the other hand, Figure 6 Step S13' is "The encoder converts the training image into a plurality of block features." In step S13', encoder E obtains the block features directly from the image level. For example, block feature 1 obtained in step S13' may be a linear combination of block features 2, 3, and 5 obtained in step S13. Block feature 2 obtained in step S13' may be a linear combination of block features 2, 6, 7, and 9 obtained in step S13.
[0073] The present invention implements the update of the trust interval as a loss function, as shown in the following method 6, where M (1) Represents the memory item in memory M that is closest to z.
[0074] Formula 6: L trust =r(z,M (1) )×||zM (1) ||2.
[0075] The reconstruction model of the present invention uses multiple loss functions in the training phase, including reconstruction loss, SSIM loss, VGG feature loss, GAN loss, and GAN feature loss. The details of these loss functions can be found in the literature listed below. The overall loss function is defined as follows: rec ,λ sm ,λ vgg ,λ GAN ,λ feat ,λ margin ,λ trust) are the hyperparameters that control each weight.
[0076] Formula 7:
[0077] Among them, the reconstruction loss L rec You can refer to the content recorded in "Semantic image synthesis with spatially-adaptive normalization" by Taesung Park et al. (In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019) to obtain it.
[0078] SSIM loss L sm You can refer to the content recorded in Paul Bergmann et al. "Improving unsupervised defects segmentation by applying structural similarity to autoencoders" (In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), pages 372–380, 2019).
[0079] VGG feature loss L vgg You can refer to the content recorded in "Perceptual losses for real-time style transfer and super-resolution" by Justin Johnson et al. (In European conference on computer vision, pages 694–711. Springer, 2016).
[0080] GAN loss L GANYou can refer to the content recorded in "Generative adversarial nets" by Ian Goodfellow et al. (In Advances in neural information processing systems, pages 2672–2680, 2014) for acquisition.
[0081] GAN feature loss L feat You can refer to the content recorded in "High-resolution images synthesis and semantic manipulation with conditional gans" by Ting-Chun Wang et al. (In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.4) and the content recorded in "Learning to superresolve blurry face and text images" by Xiangyu Xu et al. (In Proceedings of the IEEE International Conference on Computer Vision, pages 251–260, 2017).
[0082] In order to use the autoencoder to solve the problem of over-generalization, the present invention needs to effectively limit the latent space so that the autoencoder can still reconstruct the standard image area without reconstructing defects. To this end, the present invention designs a reconstruction model based on a memory-enhanced autoencoder. The present invention uses memory to store the latent space. In the memory update stage, the present invention increases the sparsity so that the updated information is focused on a small number of memory slots. In addition, the present invention proposes a mechanism for updating the memory with a trust region, which can exclude defective latent space samples and avoid noise samples from contaminating the memory slots. The reconstruction model of the present invention is resistant to noise and can achieve good performance even when the input image contains more than 40% defective images. As Figure 7 shown. Figure 7 The horizontal axis is the noise ratio of the input image, and the vertical axis uses the area under the curve (AUC) as an evaluation indicator, which can be regarded as the accuracy of the reconstructed image. Figure 7The proposed reconstruction model achieves nearly 90% accuracy even when the noise ratio approaches 40%. Compared to models using only autoencoders, or combining autoencoders with other loss functions, the proposed reconstruction model surpasses these models by over 10%.
[0083] Given a dataset containing both standard and defective images, a reconstruction model trained using the present invention can be used to distinguish between standard and defective images without requiring prior labeling of the two images. Furthermore, by treating small imperfect image patches as defects, the present invention can reduce the proportion of defect-free images in the dataset. Thus, the present invention utilizes good image patches in both standard and defective images to increase the number of images used to train the reconstruction model, demonstrating the robustness of the proposed reconstruction model against noise (defective images).
[0084] In summary, the method for generating reconstructed images proposed in the present invention has the following contributions or benefits: The defect classifier implemented using the method is robust to noise in the training dataset. The sparse addressing scheme proposed in the present invention prevents overgeneralization of the autoencoder's memory slots. The trust region memory update scheme proposed in the present invention prevents memory slot contamination by defective images during the training phase.
[0085] Although the present invention is disclosed above with reference to the aforementioned embodiments, they are not intended to limit the present invention. Any modifications and variations made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention. The scope of protection of the present invention is defined by the claims.
Claims
1. A method for generating a reconstructed image, characterized in that: Applicable to an input image having a target object, the method for generating a reconstructed image includes: Converting the input image into a feature map having a plurality of feature vectors using an encoder; executing a training process according to a plurality of training images of a plurality of reference objects to generate a plurality of feature prototypes associated with the training images and storing the feature prototypes in a memory; selecting a portion of the feature prototypes from the memory according to a plurality of similarities between the feature prototypes and the feature vectors; generating an approximate feature map based on the portion of the feature prototype and a plurality of weights, wherein the weights each represent a similarity between the feature prototype and the feature vector; and The approximate feature map is converted into a reconstructed image by a decoder; wherein The encoder, the decoder and the memory form an autoencoder; Executing the training program based on the training image of the reference object includes: setting a default feature prototype; for each of the training images, dividing the training image into multiple blocks; using the encoder to convert the blocks into multiple block features respectively; calculating multiple distances between the block features and the default feature prototype; retaining the block features corresponding to one or more distances less than a threshold among the multiple distances; and updating the default feature prototype as one of the feature prototypes based on the retained block features.
2. The method for generating a reconstructed image according to claim 1, wherein: Selecting the portion of the feature prototypes from the memory according to the similarity between the feature prototype and the feature vector includes: Selecting a portion of the weights from the weights in descending order; Selecting the portion of the feature prototypes corresponding to the portion of the weights; calculating a plurality of sparse approximations based on the number of the portion of the weights; and Each uses the sparse approximation as the weight.
3. The method for generating a reconstructed image according to claim 1, wherein: The weight is calculated using the following formula: where ω i is the i-th weight, z is one of the eigenvectors, M i is the i-th feature prototype in the memory, and M is the number of memory slots in the memory.
4. The method for generating a reconstructed image according to claim 1, wherein: The threshold is an average value of the multiple distances.
5. The method for generating a reconstructed image according to claim 1, wherein: A loss function of the autoencoder is associated with the following formula: trust =r(z,M (1) )||zM (1) ||2, where L trust is one of the loss functions, z is one of the feature vectors, M (1) is the one closest to z among the feature prototypes, and r(z,M (1) ) is used to indicate z and M (1) Is the distance less than the threshold? 6. The method for generating a reconstructed image according to claim 5, wherein: r(z,M (1) ) is calculated using the following formula: Wherein δ1 is the threshold value, and δ2 is another threshold value greater than δ1.