An image anomaly detection method and device based on repair reconstruction

Abstract

The application discloses an image anomaly detection method and device based on repair reconstruction, which comprises the following steps: step 1, collecting surface image data of normal products and performing a pretreatment operation on the data; step 2, performing a local masking operation on the normal image data; step 3, image repair reconstruction, and training an unsupervised anomaly detection model; and step 4, based on the trained unsupervised anomaly detection model, performing anomaly detection and positioning on product images to be detected. The application adopts an unsupervised algorithm, does not need data labeling, and can realize anomaly detection and positioning of products; adopts deformable convolution, adaptively learns the shape of a convolution window, can be closer to the shape and size of an anomaly during sampling, and is more robust; a dynamic pruning strategy is introduced, the model is encouraged to minimize the performance decline cost to maximize the reduction of the parameter quantity, the stability and accuracy of the model are better balanced, and anomaly positioning can be more accurately realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly detection, in particular to an image anomaly detection method and device based on repair reconstruction. BACKGROUND

[0002] Surface anomaly detection of products is a technical problem that has attracted much attention in the manufacturing industry in recent years. The key of the detection algorithm lies in extracting features from images that can identify anomalies, and then performing anomaly detection based on the features.

[0003] Existing image anomaly detection methods can be divided into two categories: traditional methods and deep learning-based methods. The anomaly detection technology based on traditional methods includes six categories: template matching-based, statistical model-based, image decomposition-based, frequency domain analysis-based, sparse coding reconstruction-based, and classification surface construction-based anomaly detection methods. The deep learning-based method includes four categories: distance metric-based, classification surface construction-based, image reconstruction-based, and combination of traditional methods-based anomaly detection methods.

[0004] The reconstruction-based method trains neural network architectures such as autoencoders, variational autoencoders, or generative adversarial networks, which only reconstruct normal training images. Therefore, abnormal images can be discovered because they are not well reconstructed. In order to locate anomalies, the reconstruction-based method usually uses pixel-level reconstruction error as an anomaly score. This method is very intuitive and has strong interpretability, but the performance is often not high enough, and the problem of excessive reconstruction error occurs, which makes it difficult to effectively distinguish between normal images and abnormal images. SUMMARY

[0005] The purpose of the present application is to provide an image anomaly detection method and device based on repair reconstruction to solve the above problems. The present application can eliminate the problem of excessive reconstruction error and achieve more accurate anomaly detection and positioning.

[0006] The present application achieves the above-mentioned purposes through the following technical solutions:

[0007] An image anomaly detection method and device based on repair reconstruction, comprising the following steps:

[0008] Step 1, collect surface image data of normal products and perform preprocessing operation on the data;

[0009] Step 2, perform local masking operation on normal image data;

[0010] Step 3, image repair reconstruction, train unsupervised anomaly detection model;

[0011] Step 4, based on the trained unsupervised anomaly detection model, perform anomaly detection and positioning on the product image to be detected.

[0012] Further, the data preprocessing method in step 1 includes but is not limited to image cleaning, image cutting, data enhancement, etc.

[0013] Further, the method of local masking in step 2 includes: inputting the original normal image, performing n times of local masking operation on it to obtain n masked "defective images", the intersection of the masked image blocks in the n "defective images" is empty, and the union is the original normal image.

[0014] Further, the method of image repair reconstruction in step 3 is: inputting the n "defective images" in step 2 into the image repair reconstruction network to obtain n "repaired images", extracting the repaired image blocks from the n "repaired images", and merging to obtain a reconstructed image with the same size as the original image.

[0015] Further, the image repair reconstruction network is improved based on the U 2 -Net network, using deformable convolution to replace all 3x3 convolutions in the network, and introducing a dynamic pruning strategy to reduce the model parameter amount.

[0016] Further, the deformable convolution learns an offset Δp n for each point on each convolution, and establishes an adaptive convolution structure for different feature layers, and the calculation formula of the feature map is as follows:

[0017]

[0018] Further, the dynamic pruning strategy is to introduce a Mask Layer to mask the channels of the feature map, and each feature map is connected with a mask vector, which samples the feature channels to reduce the feature dimension; the specific method is: for any feature map L after convolution, a vector e l is randomly initialized, and the length of the vector is the channel number of the feature map of this layer; e l is a learnable mask parameter, multiplied by a scaling coefficient s, and then passed through a sigmoid activation function to obtain a mask vector; in the training stage, s takes a random number between 1-s max , at this time the mask vector is a soft mask; in the inference stage, a very large value s max is directly assigned to s to binarize the mask vector, so as to obtain the pruned feature extraction network; the calculation formula is as follows:

[0019] f l ′=f l *m l

[0020] m l = sigmoid(se l )

[0021] s = rand(1, s max )

[0022] Wherein, f l is the original feature map, f l ' is the pruned feature map.

[0023] Further, the method of training the unsupervised anomaly detection model in step 3: based on the loss function optimization model, the loss function is composed of L2 distance loss, image similarity measure loss L SSIM And L GMSD , and sparse loss L S , as follows:

[0024] Loss = L2 + lambda1L SSIM + lambda2L GMSD + lambda3L S

[0025]

[0026] Wherein, L is the number of convolutional layers, K l is the kernel size of the lth layer convolution, c l is the number of channels of the lth layer convolution, and ||m l ||1 represents the 1-norm of the lth layer mask vector.

[0027] Further, the method of product image anomaly detection and positioning in step 4: input the product image to be detected into the trained anomaly detection model, output the corresponding repair reconstruction image; compare the reconstruction image and the original image, calculate the similarity to obtain the anomaly score image; smooth the anomaly score image and set the segmentation threshold to realize anomaly detection and anomaly positioning; the method of smoothing includes but is not limited to: filtering, image morphological processing.

[0028] Another aspect of the present application also provides an unsupervised anomaly detection device based on repair reconstruction, comprising a data acquisition module and an anomaly detection module.

[0029] The data acquisition module fixes the camera on the product production line for collecting the surface image data of the industrial product.

[0030] The anomaly detection module comprises a computer readable storage medium and a processor, and is connected with the camera through a data line.

[0031] The computer readable storage medium is used for storing a computer program, and the computer program is used for executing steps in the product image anomaly detection method based on the embedded feature vector by a processor, so that real-time surface anomaly detection is performed on products on an industrial production line.

[0032] The present application has the following advantages:

[0033] (1) An unsupervised algorithm is used, and data labeling is not required, so that product anomaly detection and positioning can be realized;

[0034] (2) A deformable convolution is used, and the shape of a convolution window is adaptively learned, so that the shape and size of the anomaly can be closer during sampling, and the robustness is higher;

[0035] (3) A dynamic pruning strategy is introduced, the model is encouraged to minimize the performance degradation cost to maximize the reduction of the parameter quantity, the stability and accuracy of the model are better balanced, and the anomaly positioning can be more accurately realized. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 The flowchart of the unsupervised anomaly detection method based on repair reconstruction in the present application;

[0038] Figure 2 The schematic diagram of the unsupervised anomaly detection method based on repair reconstruction in the present application;

[0039] Figure 3 The structural schematic diagram of the image repair reconstruction network in the present application;

[0040] Figure 4 The schematic diagram of the product image anomaly detection device based on repair reconstruction in the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present application more clear, the technical scheme of the present application will be described in detail. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0042] Embodiment one

[0043] Figure 1 and Figure 2 respectively show the flow chart and schematic diagram of the unsupervised anomaly detection method based on repair reconstruction in the application, which comprises the following steps:

[0044] Step 1, collect the surface image data of normal products, and perform pre-processing operation on the data.

[0045] In specific embodiments, the data pre-processing method includes but is not limited to image cleaning, image cutting, data enhancement, etc.

[0046] Step 2, perform local masking operation on the normal image data.

[0047] In specific embodiments, the local masking method: input the original normal image I, use a k×k grid to process it into N image blocks, each image block corresponds to a unique position coordinate; based on the non-replacement sampling method, randomly sample N / n position coordinates each time, mask the image blocks corresponding to these position coordinates (i.e. set the corresponding positions in the image matrix to 0); repeat n times to obtain n "defective images", since the sampling method is non-replacement sampling, the intersection of the image blocks masked in the n "defective images" is empty, and the union is the original normal image I, which can ensure that each image block is masked once and repaired and reconstructed in subsequent processing.

[0048] Step 3, image repair reconstruction, train an unsupervised anomaly detection model.

[0049] In specific embodiments, the image repair reconstruction method: input the n "defective images" in step 2 into the image repair reconstruction network to obtain n "repaired images", based on the position coordinates sampled n times in step 2, extract the repaired image blocks from the n "repaired images" respectively, and combine them to obtain a reconstructed image I of the same size as the original image. r .

[0050] In specific embodiments, the image repair reconstruction network is as shown in Figure 3 : based on the improvement of U 2 -Net network, replace all 3×3 convolutions in the network with deformable convolution, and the combination of deeper and more complex model structure and deformable convolution can effectively solve the problem of too large reconstruction error; at the same time, introduce dynamic pruning strategy to reduce the model parameter quantity.

[0051] In specific embodiments, the deformable convolution: considering the structure of the CNN convolution kernel is fixed, the modeling of the unknown shape transformation target is limited to a certain extent, because different positions may correspond to objects with different scales or different shapes. In order to better model, a method capable of automatically adjusting the scale or the receptive field is needed. Therefore, deformable convolution is adopted here, and an offset Δp is learned for each point on each convolution n An adaptive convolution structure is established for different feature layers. The calculation formula of the ordinary convolution feature map y Conv And the deformable convolution feature map y DConv The calculation formula is as follows:

[0052]

[0053]

[0054] In specific embodiments, the dynamic pruning strategy: introduce Mask Layer, mask the channels of the feature map, and each feature map is connected with a mask vector. The feature channels are sampled, so as to reduce the feature dimension. The specific method is as follows: for any feature map L after convolution, a vector e l is randomly initialized, and the length of the vector is the channel number of the feature map of this layer; e l is a learnable mask parameter, multiplied by a scaling coefficient s, and then passed through a sigmoid activation function to obtain a mask vector; in the training stage, s takes a random number between 1-s max At this time, the mask vector is a soft mask; in the inference stage, a very large value s max is directly assigned to s to binarize the mask vector, so as to obtain the pruned feature extraction network. The calculation formula is as follows:

[0055] f l ′=f l *m l

[0056] m l =sigmoid(se l )

[0057] s=rand(1,s max )

[0058] Where f l is the original feature map, and f l ′ is the pruned feature map.

[0059] In specific embodiments, the method for training the unsupervised anomaly detection model: based on loss function optimization model. The design of the loss function contains two parts: first, compare the original image and the reconstructed image, consider the L2 distance loss, and the image similarity measure loss L SSIM and L GMSD ; in addition, the sparse loss L S is introduced for the dynamic pruning part, which encourages the model to maximize the reduction of parameter quantity at the cost of minimal performance degradation. The loss function is as follows:

[0060] Loss=L2+λ1L SSIM +λ2L GMSD +λ3L S

[0061]

[0062] Where L is the number of convolutional layers, K l is the kernel size of the lth convolutional layer, c l is the number of channels of the lth convolutional layer, and ‖m l ‖1 represents the 1-norm of the lth mask vector.

[0063] Step 4, based on the trained unsupervised anomaly detection model, the product image is detected and positioned.

[0064] In specific embodiments, the method for product image anomaly detection and positioning: input the product image to be detected into the trained anomaly detection model, output the corresponding repair reconstruction image; compare the reconstruction image and the original image, calculate the similarity to obtain an anomaly score map; smooth the anomaly score map and set a segmentation threshold to realize anomaly detection and anomaly positioning.

[0065] In specific embodiments, the method for image smoothing includes but is not limited to filtering and image morphological processing.

[0066] Embodiment two

[0067] This embodiment discloses a product image anomaly detection device based on repair reconstruction, as shown in the figure. It includes a data acquisition module and an anomaly detection module Figure 4

[0068] The data acquisition module fixes the camera on the product production line to collect the surface image data of industrial products.

[0069] The anomaly detection module includes a computer readable storage medium and a processor, connected to the camera through a data line.

[0070] ​A computer-readable storage medium is used to store a computer program that executes the steps of the product image anomaly detection method based on embedded feature vectors through a processor, and performs real-time surface anomaly detection on products on an industrial production line.

[0071] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately. Furthermore, various different embodiments of the present invention can also be arbitrarily combined, as long as they do not violate the spirit of the present invention, they should also be considered as the content disclosed in the present invention.

Claims

1. An image anomaly detection method based on inpainting and reconstruction, characterized in that, Includes the following steps: Step 1: Collect surface image data of normal products and perform preprocessing operations on the data; Step 2: Perform a partial masking operation on the normal image data; Step 3: Image inpainting and reconstruction, and training of an unsupervised anomaly detection model; Step 4: Based on the trained unsupervised anomaly detection model, perform anomaly detection and localization on the product image to be inspected; The partial occlusion method in step 2 includes: inputting the original normal image, performing n partial occlusion operations on it to obtain n occluded "incomplete images". The intersection of the occluded image patches in these n "incomplete images" is empty, and the union is the original normal image. The image inpainting and reconstruction method in step 3 is as follows: input the n "damaged images" from step 2 into the image inpainting and reconstruction network to obtain n "repaired images". Extract the repaired image patches from the n "repaired images" and merge them to obtain a reconstructed image of the same size as the original image. The image inpainting and reconstruction network is based on U 2 - The Net network is improved by replacing all 3×3 convolutions in the network with deformable convolutions and introducing a dynamic pruning strategy to reduce the number of model parameters; The dynamic pruning strategy involves introducing a Mask Layer to mask the channels of the feature maps. Each feature map is followed by a mask vector, and the feature channels are sampled to reduce the feature dimensionality. Specifically, for any convolutional feature map L, a vector is randomly initialized. The length of this vector is the number of channels in the feature map of that layer; These are learnable mask parameters. Multiplying them by a scaling factor 's' and then passing them through a sigmoid activation function yields the mask vector. During the training phase, 's' is set to 1-... The random number between these values ​​is used; at this point, the mask vector is a soft mask. During the inference phase, a very large value is directly assigned to s. The mask vector is binarized to obtain the pruned feature extraction network; the calculation formula is as follows: ； ； ； in, This is the original feature map. This is a feature diagram after pruning.

2. The image anomaly detection method based on restoration and reconstruction as described in claim 1, characterized in that, The deformable convolution learns an offset for each point on each convolution. An adaptive convolutional structure is established for different feature layers, and the formula for calculating the feature map is as follows: 。 3. The image anomaly detection method based on restoration and reconstruction as described in claim 1, characterized in that, Step 3 involves training an unsupervised anomaly detection model using a loss function that optimizes the model. Distance loss, image similarity metric loss and and sparsity loss The composition is as follows: ； ； Where L is the number of convolutional layers, Let l be the kernel size of the l-th convolutional layer. It is the number of channels in the l-th convolutional layer. Let represent the 1-norm of the mask vector at the l-th layer.

4. The image anomaly detection method based on restoration and reconstruction as described in claim 1, characterized in that, The method for product image anomaly detection and localization in step 4 is as follows: input the product image to be inspected into the trained anomaly detection model and output the corresponding repaired and reconstructed image; compare the reconstructed image and the original image, calculate the similarity to obtain the anomaly score image; smooth the anomaly score image and set a segmentation threshold to achieve anomaly detection and anomaly localization.

5. The image anomaly detection method based on restoration and reconstruction as described in claim 4, characterized in that, The smoothing methods include: filtering and image morphological processing.

6. An unsupervised anomaly detection device based on repair and reconstruction, characterized in that, Includes a data acquisition module and an anomaly detection module; The data acquisition module fixes the camera on the product production line to collect surface image data of industrial products. An anomaly detection module, including a computer-readable storage medium and a processor, is connected to the camera via a data cable; A computer-readable storage medium is used to store a computer program that, through a processor, executes the steps of an image anomaly detection method based on repair and reconstruction as described in any one of claims 1-5, to perform real-time surface anomaly detection on products on an industrial production line.

Images