A product image defect detection method and device based on unsupervised feature combination

Abstract

The present invention discloses a method and device for detecting defects in product images based on unsupervised feature combination, the method comprising collecting a small number of defect-free product images as training samples, and dividing the images into small blocks of equal size; dividing each image into two parts, one part is the image block, and the other part is the masked image obtained by partially masking the corresponding image block; using the image block and the masked image to train a memory network and an image restoration network, combining the features of the memory network and the image restoration network, and sharing a decoder; using the trained memory network and the image restoration network to perform defect detection on the product image to be inspected. The present invention uses a memory network to extract the overall structural information of the image block, and uses an image restoration network to extract the local domain information of adjacent image blocks, and finally fuses the overall information and the local information, so that the details of the complex image can be reconstructed, the detection accuracy is high and the false detection rate is extremely low.

Description

Technical Field

[0001] The present invention relates to the technical field of product defect detection, and in particular to a product image defect detection method and device based on unsupervised feature combination. Background Art

[0002] Product image defect detection is one of the most important links to ensure product quality. In the past, product image defects were manually detected by trained workers, which was very time-consuming and inefficient. Later, the classical machine vision method replaced manual work in many cases. However, with the advent of the Industrial 4.0 era, production lines are moving towards universalization. The classical machine vision method with manually designed features has a long development cycle, non-universal features, and cannot ensure fast and flexible adaptation to new products. After the development of deep learning, it has gradually replaced the classical machine vision method because it can automatically extract features and has strong generalization ability.

[0003] When using supervised deep learning models to detect product image defects, samples with labeled information are required; however, in the actual manufacturing process, some types of defects of some products rarely occur, and it is almost impossible to collect samples. Labeling defects of some products requires a lot of time and manpower. For example, Chinese patents CN202110669165.8 and CN202011493629.6 describe supervised defect detection methods, which have relatively high sample collection and labeling costs.

[0004] When unsupervised deep learning models perform product image defect detection, no annotation is required, only defect-free training samples are needed, but their detection accuracy is usually not as good as that of supervised models. Chinese patent CN202110649145.4 proposes an unsupervised defect detection scheme based on autoencoders, but the reconstruction of complex images by this scheme is relatively fuzzy, which affects the detection accuracy; Chinese patent CN202110415805.2 proposes to use knowledge distillation for defect detection, but the construction of industrial data sets to be distilled is time-consuming and laborious; Chinese patent CN202110196131.1 uses generative adversarial networks to detect defects, but the problem of generative adversarial network mode collapse has been difficult to solve; represented by the document "PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization", some methods pre-train the initial network from the ImageNet dataset, and then find defects at the feature level through distillation compression, but this method ignores the difference between industrial products and natural images. Summary of the invention

[0005] The present invention provides a product image defect detection method and device based on unsupervised feature combination to solve the problem that in the actual production process, certain defect types of some products rarely appear and it is almost impossible to collect samples, and the defect labeling of some products requires a lot of time and manpower, and the problem that when an unsupervised deep learning model is used for product image defect detection, the detection accuracy is usually not as good as that of a supervised model.

[0006] The technical solution adopted by the present invention is: to disclose a product image defect detection method based on unsupervised feature combination, comprising:

[0007] S1: Collect a small number of defect-free product images as training samples and divide the images into small blocks of equal size;

[0008] S2: Each image is divided into two parts, one is the image block, and the other is the masked image obtained by masking the corresponding image block;

[0009] S3: Using the image patches and the masked images to train the memory network and the image restoration network, the features of the memory network and the image restoration network are combined, and a decoder is shared;

[0010] S4: Use the trained memory network and image restoration network to perform defect detection on the images of the products to be inspected.

[0011] As a preferred method of product image defect detection based on unsupervised feature combination, dividing the image into small blocks of equal size means dividing each image into different image areas of equal size using a grid to expand the training sample.

[0012] As a preferred method of product image defect detection based on unsupervised feature combination, S2 includes:

[0013] S2.1: Select an image of the training sample and extract a small image block on the image;

[0014] S2.2: Cover the corresponding position of the image block with a completely black block to obtain a covered image;

[0015] S2.3: The image patch and the corresponding mask image constitute a training sample pair;

[0016] S2.4: Take another small image block from the image selected in S2.1 and perform the operations in steps S2.2 and S2.3 to form another training sample pair;

[0017] S2.5: Repeat S2.4 until each image block in the image selected in S2.1 is selected. At this time, the image selected in S2.1 forms a plurality of training sample pairs, and the number of training sample pairs is equal to the number of image blocks in the image.

[0018] S2.6: Execute steps S2.1-S2.5 for each image of the training sample.

[0019] As a preferred method of product image defect detection based on unsupervised feature combination, S3 includes:

[0020] S3.1: Construct a memory network and an image restoration network. The features output by the encoder parts of the memory network and the image restoration network are combined, and they share a decoder.

[0021] S3.2: Using the training sample pairs formed in steps S2.1-S2.6 as input, train the memory network and the image restoration network so that the network reconstructs substantially identical image patches.

[0022] As a preferred method of the product image defect detection method based on unsupervised feature combination, in S3.2, training the memory network and the image restoration network includes:

[0023] The features of the memory network are replaced with the memories in the memory bank. The replacement formula is as follows:

[0024]

[0025] q represents the features extracted by the encoder of the memory network, m i Represents memory, Represents the features after replacing the memory in the memory bank, represents the weight, j represents the number of memories contained in the memory bank;

[0026] The features after memory replacement are used in the memory network and the features extracted from the image repair network are connected and combined by channel.

[0027] The loss function used for network training is:

[0028]

[0029] x represents the original image patch, Represents a small patch of the reconstructed image.

[0030] As a preferred method of product image defect detection based on unsupervised feature combination, S4 includes:

[0031] S4.1: Divide the image to be inspected into different image regions of equal size using a grid;

[0032] S4.2: converting the image to be inspected into training sample pairs consisting of small image blocks and corresponding mask images according to the steps S2.1-S2.6;

[0033] S4.3: Input the training sample pairs into the trained memory network and image restoration network to obtain a reconstructed image patch;

[0034] S4.4: The reconstructed image is spliced ​​into small blocks to form a reconstructed image, and compared with the image to be inspected, and the existence of defects in the image to be inspected and the specific location of the defects are determined based on the pixel-by-pixel differences.

[0035] As a preferred method of product image defect detection based on unsupervised feature combination, S4.4 includes:

[0036] S4.41: Calculate the pixel-by-pixel absolute difference between the reconstructed image and the image to be tested, and select a threshold value for the difference image to perform binarization;

[0037] S4.42: Use erosion and dilation operations to eliminate the effects of noise;

[0038] S4.43: Determine and output whether there are defects in the image to be inspected and the specific location of the defects. If the final difference does not exceed the threshold, the image to be inspected is output as an image of a normal product; if the final difference exceeds the threshold, the image to be inspected is output as an image of a defective product, and the corresponding area is the area where the defect exists.

[0039] The present invention also discloses a product image defect detection device based on unsupervised feature combination, comprising a camera and a detection module, wherein the camera is fixed on a product production line to capture product images; the detection module comprises a computer-readable storage medium and a processor, and is connected to the camera via a data cable; the computer-readable storage medium stores a computer program, and the computer program executes the steps in the above-mentioned product image defect detection method based on unsupervised feature combination through the processor.

[0040] The beneficial effects of the present invention are as follows: the present invention uses a memory network to extract the overall structural information of a small image block, and uses an image restoration network to extract the local field information of adjacent small image blocks, and finally integrates the overall information and the local information, so that the details of a complex image can be reconstructed, the detection accuracy is high and the false detection rate is extremely low; the present invention increases the number of samples by gridding the product images, so that only a small number of normal sample images are needed to train the model; the proposed method only performs unsupervised learning, does not require the collection of missing samples, and does not require labeling, is low in cost, easy to implement, and can more accurately determine the location of defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the product image defect detection method based on unsupervised feature combination disclosed in the present invention.

[0042] Figure 2A schematic diagram of a training sample pair consisting of an image patch and a masked image disclosed in the present invention.

[0043] Figure 3 It is a schematic diagram of the structure of the memory network and image restoration network disclosed in the present invention.

[0044] Figure 4 It is a schematic diagram of the process of the product image defect detection device based on unsupervised feature combination disclosed in the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0046] Embodiment 1:

[0047] See also Figure 1 This embodiment discloses a product image defect detection method based on unsupervised feature combination, which specifically includes the following steps:

[0048] S1: Collect a small number of defect-free product images as training samples and divide the images into small blocks of equal size.

[0049] S2: Each image is divided into two parts, one is the image block, and the other is the masked image obtained by masking the corresponding image block.

[0050] S3: Using image patches and masked images to train a memory network and an image restoration network, the features of the memory network and the image restoration network are combined, and a decoder is shared.

[0051] S4: Use the trained memory network and image restoration network to perform defect detection on the images of the products to be inspected.

[0052] In this embodiment, 120 defect-free training samples are collected.

[0053] Dividing the image into small blocks of equal size in step S1 refers to dividing each image into different image regions of equal size using a grid, thereby expanding the training samples.

[0054] In this embodiment, the image size is 256*256, and is divided into 256 image blocks of equal size using a 16*16 network.

[0055] The S2 specifically includes:

[0056] S2.1: Select an image of a training sample and extract a small image block on the image.

[0057] S2.2: Cover the corresponding position of the image block with a completely black block to obtain a covered image.

[0058] S2.3: An image patch and its corresponding mask image constitute a training sample pair.

[0059] S2.4: Take out another small image block from the image selected in S2.1, and perform the operations in steps S2.2 and S2.3 to form another training sample pair.

[0060] S2.5: Repeat S2.4 until each image block in the image selected by S2.1 is selected. At this time, the image selected by S2.1 forms multiple training sample pairs, and the number of training sample pairs is equal to the number of image blocks in the image.

[0061] S2.6: Execute steps S2.1-S2.5 for each image of the training sample.

[0062] In this embodiment, each training sample is converted into 256 training sample pairs consisting of image blocks and cover images. Figure 2 As shown, Figure 2 The image is divided into 4 image blocks of 2*2, and 4 training sample pairs are obtained after the image is subjected to steps S2.1-S2.5. Similarly, in this embodiment, an image is divided into 256 image blocks, and 256 training sample pairs are obtained.

[0063] The S3 specifically includes:

[0064] S3.1: Construct a memory network and an image restoration network. The features output by the encoder parts of the memory network and the image restoration network are combined, and they share a decoder.

[0065] S3.2: Taking the training sample pairs formed in steps S2.1-S2.6 as input, train the memory network and the image restoration network so that the network reconstructs small image blocks that are as similar as possible.

[0066] In this embodiment, the constructed network is as follows Figure 3 shown.

[0067] In this embodiment, the memory bank contains 512 memories, and the size of each memory is the same as the feature size output by the memory network encoder part.

[0068] In this embodiment, the memory network and the image restoration network respectively accept the image patch and the mask image in the training sample pair as input, and extract features through their respective encoders, wherein the features of the memory network are replaced with the memories in the memory bank, and the replacement formula is as follows:

[0069]

[0070] q represents the features extracted by the encoder of the memory network, m i Represents memory, Represents the features after replacing the memory in the memory bank, Represents weight.

[0071] In this embodiment, the features after memory replacement are used in the memory network and the features extracted from the image repair network are connected and combined by channels.

[0072] In this embodiment, the loss function used in network training is:

[0073]

[0074] x represents the original image patch, Represents a small patch of the reconstructed image.

[0075] The S4 specifically includes:

[0076] S4.1: Use a grid to divide the image to be inspected into different image regions of equal size.

[0077] S4.2: According to the steps S2.1-S2.6, the image to be inspected is converted into a training sample pair consisting of a small image block and a corresponding mask image.

[0078] S4.3: Input the training sample pairs into the trained memory network and image restoration network to obtain small reconstructed image blocks.

[0079] S4.4: The reconstructed image is spliced ​​into small blocks to form a reconstructed image, and compared with the image to be inspected, and the existence of defects in the image to be inspected and the specific location of the defects are determined based on the pixel-by-pixel differences.

[0080] In this embodiment, the image to be inspected is divided into 256 image blocks of equal size using a 16*16 network. The 256 reconstructed image blocks of the image to be inspected are sequentially spliced ​​into a reconstructed image.

[0081] In this embodiment, the pixel-by-pixel absolute difference between the reconstructed image and the image to be inspected is calculated, and a threshold is selected for the difference image to be binarized, and then corrosion and dilation operations are used to eliminate the influence of noise; if the final difference image has no white area, it means that the image to be inspected is an image of a normal product; if the final difference image has a white area, it means that the image to be inspected is an image of a defective product, and the white area is the area where the defect exists.

[0082] Example 2

[0083] This embodiment provides a product image defect detection device based on unsupervised feature combination, which is composed of a camera and a detection module, such as Figure 4 As shown, the camera is fixed on the product production line to capture product images; the detection module includes a computer-readable storage medium and a processor, and is connected to the camera via a data cable; the computer-readable storage medium stores a computer program, and the computer program executes the steps of the product image defect detection method based on unsupervised feature combination described in Example 1 through the processor.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A product image defect detection method based on unsupervised feature combination, characterized in that: include S1: Collect a small number of defect-free product images as training samples and divide the images into small blocks of equal size; S2: Each image is divided into two parts, one is the image block, and the other is the masked image obtained by masking the corresponding image block; The S2 includes: S2.1: Select an image of the training sample and extract a small image block on the image; S2.2: Cover the corresponding position of the image block with a completely black block to obtain a covered image; S2.3: The image patch and the corresponding mask image constitute a training sample pair; S2.4: Take another small image block from the image selected in S2.1 and perform the operations in steps S2.2 and S2.3 to form another training sample pair; S2.5: Repeat S2.4 until each image block in the image selected in S2.1 is selected. At this time, the image selected in S2.1 forms a plurality of training sample pairs, and the number of training sample pairs is equal to the number of image blocks in the image. S2.6: Execute steps S2.1-S2.5 for each image of the training sample; S3: Using the image patches and the masked images to train the memory network and the image restoration network, the features of the memory network and the image restoration network are combined, and a decoder is shared; The S3 includes: S3.1: Construct a memory network and an image restoration network, combine the features output by the encoder parts of the memory network and the image restoration network, and share a decoder; S3.2: using the training sample pairs formed in steps S2.1-S2.6 as input, training the memory network and the image restoration network so that the network reconstructs substantially identical image patches; S4: Use the trained memory network and image restoration network to perform defect detection on the images of the products to be inspected.

2. The product image defect detection method based on unsupervised feature combination according to claim 1, characterized in that: Dividing the image into small blocks of equal size refers to dividing each image into different image regions of equal size using a grid to expand the training samples.

3. The product image defect detection method based on unsupervised feature combination according to claim 1, characterized in that: In S3.2, training the memory network and the image restoration network includes: The features of the memory network are replaced with the memories in the memory bank. The replacement formula is as follows: q represents the features extracted by the encoder of the memory network, m i represents memory, q represents the feature after replacing the memory in the memory bank, represents the weight, j represents the number of memories contained in the memory bank; The features after memory replacement are used in the memory network and the features extracted from the image repair network are connected and combined by channel. The loss function used for network training is: x represents the original image patch, and x represents the reconstructed image patch.

4. The product image defect detection method based on unsupervised feature combination according to claim 3, characterized in that: The S4 includes: S4.1: Divide the image to be inspected into different image regions of equal size using a grid; S4.2: converting the image to be inspected into training sample pairs consisting of small image blocks and corresponding mask images according to the steps S2.1-S2.6; S4.3: Input the training sample pairs into the trained memory network and image restoration network to obtain a reconstructed image patch; S4.4: The reconstructed image is spliced ​​into small blocks to form a reconstructed image, and compared with the image to be inspected, and the existence of defects in the image to be inspected and the specific location of the defects are determined based on the pixel-by-pixel differences.

5. The product image defect detection method based on unsupervised feature combination according to claim 4, characterized in that: The S4.4 includes: S4.41: Calculate the pixel-by-pixel absolute difference between the reconstructed image and the image to be tested, and select a threshold value for the difference image to perform binarization; S4.42: Use erosion and dilation operations to eliminate the effects of noise; S4.43: Determine and output whether there are defects in the image to be inspected and the specific location of the defects. If the final difference does not exceed the threshold, the image to be inspected is output as an image of a normal product; if the final difference exceeds the threshold, the image to be inspected is output as an image of a defective product, and the corresponding area is the area where the defect exists.

6. A product image defect detection device based on unsupervised feature combination, characterized in that: It comprises a camera and a detection module, wherein the camera is fixed on a product production line to capture product images; the detection module comprises a computer-readable storage medium and a processor, and is connected to the camera via a data cable; the computer-readable storage medium stores a computer program, and the computer program executes the steps of the product image defect detection method based on unsupervised feature combination as described in any one of claims 1 to 5 through the processor.

Images