A defect detection image enhancement method based on restricted random sampling strategy

Abstract

The present invention provides a defect detection image enhancement method based on a restricted random sampling strategy, and relates to the technical field of industrial defect detection. The method comprises: obtaining normal image data from an existing image data set; performing background detection on the obtained normal image data; randomly selecting a point (x, y) in a selected area; obtaining a real abnormal image data set TA, and randomly selecting a point value from the abnormal value in the real abnormal image and assigning it to the point (x, y); performing restricted random sampling, and continuing to assign values ​​until the abnormal point number requirement is met; and obtaining a synthesized abnormal image. The defects synthesized by the method are closer to real defects, and the accuracy of detection and positioning is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial defect detection, and in particular to a defect detection image enhancement method based on a restricted random sampling strategy. Background Art

[0002] Industrial defect detection is mainly to find the appearance defects of various industrial products, ensure product quality, and maintain stable production. In the past, defect detection required manual screening, which was difficult to cover large-scale quality inspection needs, and was costly and inefficient. With the emergence of new technologies such as computer vision and deep learning, vision-based industrial defect detection technology has made great progress and has become an effective method for product appearance quality inspection, attracting widespread attention from industry and academia. Therefore, vision-based industrial defect detection has broad application prospects.

[0003] Traditional defect detection methods are mainly based on the color, shape and other features of the defect, using image processing methods or combining traditional machine learning methods for detection. For example, edge detection methods are used to locate defects, but the interior of industrial defects is often relatively stable, and edge detection-based methods can only detect the boundaries of defects. For products with simple or periodic backgrounds, Fourier transform, Gabor transform, wavelet transform and other methods can be used to convert the signal from the time domain signal to the frequency domain for detection. However, traditional methods require manual feature extraction. Due to the weakness and subjectivity of defects, manually extracted features are difficult to accurately describe defects, and may even require the design of multiple sets of defect templates. In addition, the description of defect types is limited and new defects cannot be described.

[0004] In recent years, deep learning methods, with their powerful feature extraction capabilities and high-dimensional data representation capabilities, can automatically learn features that are difficult to design manually, which not only saves costs but also greatly improves detection accuracy. According to different data labels, they can be divided into fully supervised learning models, unsupervised learning models, and other methods (semi-supervised learning models and weakly supervised learning models). At present, unsupervised methods have made exciting progress and have become one of the research focuses of industrial defect detection. Because the method based on unsupervised learning only requires normal samples that are easy to obtain for model training, and does not need to use real defect samples, it can not only solve the problem that supervised learning methods cannot find unknown defects, but also has a stronger ability to express image features than traditional methods. Related methods have achieved higher performance. However, both supervised and unsupervised methods need as much data as possible to support the performance of the model. Supervised methods also hope that the positive and negative samples in the training set are balanced. However, the total number of samples available in actual industrial production is small, and the control of defective rate is strict, which makes it very difficult to obtain defective samples. Therefore, data enhancement is used to improve the training effect and generalization of the model from the perspective of training data. For example, Tao et al. generated defect samples by adding salt and pepper noise; Lin et al. cropped defective parts from defective samples and combined them with normal samples to amplify defect data. This type of method synthesizes anomalies by replacing or mixing image patches with the content of images or image positions. However, this type of synthetic anomaly strategy has obvious discontinuity and may overfit into the inherent encoding in the operation. If Poisson images are used to synthesize anomalies, the generated anomalies may represent changes in the normal class rather than real anomalies. Therefore, the research on industrial defect detection image enhancement is still in its infancy, and the corresponding basic theory and method framework are still lacking. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a defect detection image enhancement method based on a restricted random sampling strategy in view of the deficiencies of the above-mentioned prior art, so as to achieve enhancement of the defect detection image.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a defect detection image enhancement method based on a restricted random sampling strategy, comprising the following steps:

[0007] Step 1: Obtain normal image data from an existing image dataset;

[0008] The existing image dataset contains a total of M image datasets of different categories, of which m are object categories and n are texture categories, m+n=M; each category dataset is divided into a training set, a test set and a label dataset, wherein the training set contains only normal images, and the test set contains both normal images and real abnormal images; the normal image data obtained from the existing image dataset refers to the normal image data obtained from the training set;

[0009] Step 2: Perform background detection on the acquired normal image data;

[0010] The said performing background detection on the acquired normal image data refers to performing background detection on the image data of m object categories, while the data of n texture categories does not need to be detected;

[0011] Step 2-1, converting the acquired normal image into a grayscale image;

[0012] Step 2-2, using the adaptive histogram equalization method to process the grayscale image to improve the contrast of the image;

[0013] Step 2-3: Perform global histogram equalization to make the darker image brighter overall;

[0014] Step 2-4: Perform Gaussian filtering to remove noise;

[0015] Step 2-5: Apply threshold operation to capture specific areas from the image;

[0016] Step 2-6, perform an opening operation to remove small particle noise in the normal image;

[0017] Step 2-7, perform a closing operation to remove small noise points in the normal image;

[0018] Step 2-8, perform morphological erosion and dilation operations to remove noise and fill the background in the normal image;

[0019] Step 2-9, filter the black and white areas according to the area threshold;

[0020] Step 2-10: Use dilation and erosion operations again to depict the details;

[0021] Step 2-11, if the object part in the image in step 2-10 is not black, then perform black-white inversion;

[0022] Step 3: Randomly select a point (x, y) in the selected area;

[0023] The selected area, for the object class, refers to the black area in the image obtained in step 2; for the texture class, refers to any position in the image;

[0024] Step 4: Obtain a real abnormal image dataset TA, and randomly select a point from the abnormal value in the real abnormal image and assign it to the point (x, y) in step 3;

[0025] Step 4-1, obtaining a real abnormal image dataset TA from an existing image dataset;

[0026] First, find the test set of each category of data set, then remove the normal image part from the test set, and finally the set of remaining real abnormal images is the real abnormal data set TA;

[0027] Step 4-2, obtain the label data set TAGT corresponding to the real abnormal image data from the existing image data set;

[0028] The specific method of obtaining the label data set TAGT corresponding to the real abnormal image data from the existing image data set is: first find the label data set in each category data set, and then collect all the defect class data sets to form the label data set TAGT;

[0029] Step 4-3: Randomly select a real abnormal image from the real abnormal image dataset TA obtained in step 4-1 and the corresponding label data in the label dataset TAGT, and randomly select a point value from the abnormal value position in the real abnormal image and assign it to the point (x, y) in step 3;

[0030] Step 5: Perform restricted random sampling and continue to assign values ​​until the number of abnormal points is met;

[0031] The restricted random sampling refers to randomly selecting a point from the four positions above, below, left and right of the point (x, y) as a new (x, y) point, and then assigning a value to the point according to step 4, and so on, until the number of abnormal points is met. All the selected points can constitute a defective block;

[0032] Step 6: Obtain a synthesized abnormal image;

[0033] Repeat steps 3 to 5 for multiple times to obtain multiple defective blocks; synthesize the multiple defective blocks, and the final image obtained is the synthesized abnormal image.

[0034] The beneficial effects of adopting the above technical scheme are as follows: the present invention provides a defect detection image enhancement method based on a restricted random sampling strategy, (1) a high-precision object-type data background detection method is proposed, which can accurately distinguish the background from the object, so that defect blocks can be added only on the object to prevent the detection rate from decreasing due to adding defect blocks on the background; (2) it is proposed to select abnormal points using a restricted random sampling strategy and assign values, thereby ensuring the diversity of defect shapes and matching the actual abnormal shapes; (3) the values ​​of the synthesized defect parts do not use traditional random noise values, but use actual abnormal values, so that the synthesized defects are closer to the actual defects, thereby improving the accuracy of detection and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a defect detection image enhancement method based on a restricted random sampling strategy provided by an embodiment of the present invention;

[0036] Figure 2 The process effect diagram of hazelnut type background detection provided by an embodiment of the present invention, wherein (a) is the original hazelnut image for background detection, (b) is the obtained grayscale image, (c) is the image through the adaptive equalization method, (d) is the image through the global histogram equalization method, (e) is the image through Gaussian filtering, (f) is the image after the threshold operation, (g) is the image after the opening operation, (h) is the image after the closing operation, (i) is the image after the morphological erosion and dilation operation, (j) is the image after filtering according to the area threshold, (k) is the image after the dilation and erosion operation is used again, and (l) is the image after the black and white inversion;

[0037] Figure 3 The effect diagram of image enhancement of data in 15 types of data sets provided by the embodiment of the present invention using the method of the present invention, wherein (a) is a bottle type, (b) is a cable type, (c) is a capsule type, (d) is a carpet type, (e) is a mesh type, (f) is a hazelnut type, (g) is a leather type, (h) is a metal nut type, (i) is a tablet type, (j) is a screw type, (k) is a tile type, (l) is a toothbrush type, (m) is a transistor type, (n) is a wood type, and (o) is a zipper type;

[0038] Figure 4 The present invention provides an embodiment of the present invention providing an image enhancement effect diagram of data in 15 types of data sets using a traditional method, wherein (a) is a bottle type, (b) is a cable type, (c) is a capsule type, (d) is a carpet type, (e) is a mesh type, (f) is a hazelnut type, (g) is a leather type, (h) is a metal nut type, (i) is a tablet type, (j) is a screw type, (k) is a tile type, (l) is a toothbrush type, (m) is a transistor type, (n) is a wood type, and (o) is a zipper type. DETAILED DESCRIPTION

[0039] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0040] This embodiment takes the MVTec AD data set as an example, and adopts the defect detection image enhancement method based on the restricted random sampling strategy of the present invention to achieve defect detection image enhancement.

[0041] In this embodiment, a defect detection image enhancement method based on a restricted random sampling strategy is provided. Figure 1 As shown, the following steps are included:

[0042] Step 1: Obtain normal image data from the MVTec AD dataset;

[0043] The MVTec AD dataset is a dataset proposed at the 2019 CVPR conference. It is specifically used to evaluate and test the performance of unsupervised image abnormal pattern detection and segmentation. It contains 15 different categories of image datasets, including 10 object categories and 5 texture categories, with a total of 5354 high-resolution images. Each category of dataset is divided into training set, test set and label dataset. The training set only contains normal images, and the test set contains both normal images and real abnormal images. The normal image data obtained from the MVTec AD dataset refers to the normal image data obtained from the training set.

[0044] The 10 object categories of the MVTec AD dataset are: bottles, cables, capsules, hazelnuts, screws, toothbrushes, pills, zippers, transistors, and metal nuts; the 5 texture categories are: carpets, meshes, leather, tiles, and wood.

[0045] Step 2: Perform background detection on the acquired normal image data;

[0046] Performing background detection on the acquired normal image data means performing background detection on the image data of 10 object categories, while the data of 5 texture categories do not need background detection;

[0047] Step 2-1, converting the acquired normal image into a grayscale image;

[0048] Step 2-2, using the adaptive histogram equalization method to process the grayscale image to improve the contrast of the image;

[0049] Step 2-3: Perform global histogram equalization to make the darker image brighter overall;

[0050] Step 2-4: Perform Gaussian filtering to remove noise;

[0051] Step 2-5: Apply threshold operation to capture specific areas from the image;

[0052] In this embodiment, in the threshold operation applied, the starting threshold is selected in the range of 0 to 90 according to different image categories, and the maximum value is selected as 255;

[0053] Step 2-6, perform an opening operation to remove small particle noise in the normal image;

[0054] Step 2-7, perform a closing operation to remove small noise points in the normal image;

[0055] Step 2-8, perform morphological erosion and dilation operations to remove noise and fill the background in the normal image;

[0056] Step 2-9, filter the black and white areas according to the area threshold;

[0057] In this embodiment, in filtering the black and white areas according to the area threshold, it is necessary to firstly process the image in steps 2-8 using the function for processing irregular connected areas, obtain the identification of each connected area, the starting coordinates of the circumscribed rectangle of each connected area, the width and height of the circumscribed rectangle, the number of pixels of the connected area corresponding to the identification, and other information; then obtain the area of ​​the white area and the area of ​​the black area, and adjust the threshold according to the actual object category for filtering;

[0058] Step 2-10: Use dilation and erosion operations again to depict the details;

[0059] Step 2-11, if the object part in the image in step 2-10 is not black, then perform black-white inversion;

[0060] Step 3: Randomly select a point (x, y) in the selected area;

[0061] In the selected area, for the object class, it refers to the black area in the image obtained in step 2; for the texture class, it refers to any position in the image;

[0062] Step 4: Obtain a real abnormal image dataset TA, and randomly select a point from the abnormal value in the real abnormal image and assign it to the point (x, y) in step 3;

[0063] Step 4-1, obtaining a real abnormal image dataset TA from the MVTec AD dataset;

[0064] First, find the test set of each category of data set, then remove the normal image part from the test set, and finally the set of remaining real abnormal images is the real abnormal data set TA;

[0065] Step 4-2, obtain the label dataset TAGT corresponding to the real abnormal image data from the MVTec AD dataset;

[0066] The specific method of obtaining the label dataset TAGT corresponding to the real abnormal image data from the MVTec AD dataset is as follows: first find the label dataset in each category dataset, and then collect the data of all defect categories to form the label dataset TAGT;

[0067] Step 4-3: Randomly select a real abnormal image from the real abnormal image dataset TA obtained in step 4-1 and the corresponding label data in the label dataset TAGT, and randomly select a point value from the abnormal value position in the real abnormal image and assign it to the point (x, y) in step 3;

[0068] In this embodiment, the position of the abnormal value in the real abnormal image is the corresponding position in the abnormal image corresponding to the position where the pixel value in the label is 255;

[0069] Step 5: Perform restricted random sampling and continue to assign values ​​until the number of abnormal points is met;

[0070] The restricted random sampling refers to randomly selecting a point from the four positions above, below, left and right of the point (x, y) as a new (x, y) point, and then assigning a value to the point according to step 4, and so on, until the number of abnormal points is met. All the selected points can constitute a defective block;

[0071] In this embodiment, the number of abnormal points is required to be 255;

[0072] Step 6: Obtain a synthesized abnormal image;

[0073] Repeat steps 3 to 5 for multiple times to obtain multiple defective blocks; synthesize the multiple defective blocks, and the final image obtained is the synthesized abnormal image.

[0074] In this embodiment, steps 3 to 5 are repeated three times, that is, three defective blocks are obtained.

[0075] In this embodiment, the effect of background detection for the object type of hazelnut is as follows: Figure 2 As shown, Figure 2 (a) refers to the original image of hazelnuts for background detection. Figure 2 (b) refers to the obtained grayscale image, Figure 2 (c) refers to the image obtained by the adaptive equalization method, Figure 2 (d) refers to the image processed by the global histogram equalization method. Figure 2 (e) refers to the image subjected to Gaussian filtering. Figure 2 (f) refers to the image where the threshold operation is applied, Figure 2 (g) refers to the image after opening operation. Figure 2 (h) refers to the image after the closing operation. Figure 2 (i) refers to the image on which the morphological erosion and dilation operations are performed, Figure 2 (j) refers to the image after filtering according to the area threshold. Figure 2 (k) is the image after dilation and erosion operations are performed again. Figure 2 (l) is an image after black-and-white inversion, where the black area is the object area and the white area is the background area.

[0076] In this embodiment, the enhancement method of the present invention is also compared with the traditional enhancement method, such as Figure 3 , 4 As shown, Figure 3 This is a diagram showing the effect of enhancing the data in 15 types of data sets using the method of the present invention. Figure 4 This is the enhanced effect diagram of the traditional method.

[0077] from Figure 3 It can be seen that the defect blocks in the abnormal image synthesized by the method of the present invention have various shapes and sizes, which are closer to the actual abnormality. It is more effective to use the images obtained by this enhancement method for image defect detection training. However, most traditional methods have fixed shapes, and most of the abnormal values ​​are replaced by random noise values. The position of the defect block is not necessarily on the object, so there is still a certain gap with the real abnormality.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A defect detection image enhancement method based on a restricted random sampling strategy, characterized in that: The following steps are involved: Step 1: Obtain normal image data from an existing image dataset; The existing image data set includes a total of M image data sets of different categories, of which m are object categories and n are texture categories, m+n=M; each category data set is divided into a training set, a test set and a label data set, wherein the training set only includes normal images, and the test set includes both normal images and real abnormal images; the normal image data obtained from the existing image data set refers to the normal image data obtained from the training set; step 2, performing background detection on the normal image data of the m object categories obtained to obtain the area where the object part in the image is located; Step 2-1, converting the acquired normal image into a grayscale image; Step 2-2, using the adaptive histogram equalization method to process the grayscale image to improve the contrast of the image; Step 2-3: Perform global histogram equalization to make the darker image brighter overall; Step 2-4: Perform Gaussian filtering to remove noise; Step 2-5: Apply threshold operation to capture specific areas from the image; Step 2-6, perform an opening operation to remove small particle noise in the normal image; Step 2-7, perform a closing operation to remove small noise points in the normal image; Step 2-8, perform morphological erosion and dilation operations to remove noise and fill the background in the normal image; Step 2-9, filtering the black and white areas in the image according to the area threshold; Step 2-10: Use the dilation and erosion operations again to depict the details in the image and obtain the area where the object is located; Step 2-11, if the object part in the image processed by step 2-10 is not black, then perform black-white inversion; Step 3: Randomly select a point (x, y) in the selected area of ​​the normal image; The area selected in the normal image, for the object class, refers to the location of the black area in the image obtained in step 2; for the texture class, it refers to any position of the image; Step 4: Obtain a real abnormal image dataset TA, and randomly select a point from the abnormal value in the real abnormal image and assign it to the point (x, y) in step 3; Step 5: Perform restricted random sampling and continue to assign values ​​until the number of abnormal points is met; Step 6: Obtain a synthesized abnormal image.

2. The defect detection image enhancement method based on restricted random sampling strategy according to claim 1, characterized in that: The specific method of step 4 is: Step 4-1, obtaining a real abnormal image dataset TA from an existing image dataset; First, find the test set of each category of data set, then remove the normal image part from the test set, and finally the set of remaining real abnormal images is the real abnormal data set TA; Step 4-2, obtain the label data set TAGT corresponding to the real abnormal image data from the existing image data set; First, find the label dataset in each category dataset, and then collect the datasets of all defect categories together to form the label dataset TAGT; Step 4-3: Randomly select a real abnormal image from the real abnormal image dataset TA obtained in step 4-1 and the corresponding label data in the label dataset TAGT, and randomly select a point from the abnormal value position in the real abnormal image and assign it to the point (x, y) in step 3.

3. The defect detection image enhancement method based on restricted random sampling strategy according to claim 2 is characterized in that: The specific method of step 5 is: The restricted random sampling refers to randomly selecting a point from the four positions above, below, left and right of the point (x, y) as a new (x, y) point, and then assigning a value to the point according to step 4, and so on, until the abnormal point number requirement is met, and all the selected points can constitute a defect block.

4. The defect detection image enhancement method based on restricted random sampling strategy according to claim 3 is characterized in that: The specific method of step 6 is: Steps 3 to 5 are repeatedly performed to obtain multiple defective blocks; the multiple defective blocks are synthesized, and the image finally obtained is the synthesized abnormal image.

Images