A sample augmentation method for small sample surface defect dataset

Abstract

The application discloses a sample augmentation method for small sample surface defect data sets, and mainly solves the problems of class imbalance and sample deficiency in the surface defect data set. The implementation scheme is as follows: 1) obtaining original data and artificial labels; 2) constructing a sample augmentation algorithm; and 3) sample augmentation. The sample augmentation method constructed by the application adopts eight-neighborhood description environment and calculates the environment similarity through covariance, realizes cross-image target replication without semantic conflict, solves the class imbalance problem, and realizes defect target augmentation. The method of splicing a defect subgraph and a background subgraph is adopted, the background subgraph is re-introduced into the data set, the sample complexity of the data set is improved, and sample augmentation is realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a sample augmentation method for small sample surface defect datasets. Background Technology

[0002] Sample augmentation, as a data preprocessing method, typically employs geometric transformations and sample fusion to increase the quantity and complexity of samples within a dataset, assisting algorithms in parameter optimization and thus having a wide range of applications. For example, in industrial quality inspection, sample augmentation can expand the sample size of a dataset, addressing the problem of small samples and providing higher-quality datasets for data-driven defect detection algorithms. Geometric transformation-based sample augmentation expands samples through operations such as rotation, flipping, and distortion. However, for target detection, geometric transformations only increase the geometric richness of the background and targets; the number and categories of targets within a single sample remain unchanged. Sample fusion-based sample augmentation increases the number of targets and improves background complexity by merging multiple samples into one at a certain ratio. This method alters the characteristics of defects and the background. For defect detection in industrial applications, this change introduces invalid features, increases information redundancy, and hinders algorithm optimization. Summary of the Invention

[0003] This invention fully considers the problems existing in the existing methods and aims to provide a sample augmentation method for small sample surface defect datasets. It achieves cross-image defect replication through an environment-guided defect replication algorithm, thereby achieving the augmentation of defect targets and the balance of defect categories. It also achieves size normalization of the inspection area and image background expansion through a background reuse algorithm, thereby increasing the complexity of the dataset and realizing sample augmentation.

[0004] I. Technical Principles

[0005] Existing target copying-based sample augmentation algorithms can be divided into two categories: one is intra-image copying, which avoids strong semantic conflicts due to the limited copying range, but cannot effectively improve the complexity of samples within the dataset; the other is cross-image target copying, which greatly improves the complexity of samples within the dataset by copying targets between different images, but cross-image copying causes semantic conflicts between the target and the background of the pasting location, resulting in low sample quality. To ensure semantic similarity between the target and the background of the pasting location, neural network algorithms are used to extract semantics from the target and background to guide the target to choose a suitable pasting location, but this increases the algorithm's complexity. To achieve efficient cross-image target copying, this invention designs an environment-guided defect copying algorithm. The environmental information is described by the eight-neighborhood of the defective target and the eight-neighborhood of the defective pasting region. By calculating the covariance of the corresponding neighborhoods, the similarity between the two environments is described, efficiently guiding the cross-image copying and pasting of defective targets. To address the issue of varied shapes and extreme aspect ratios in the regions to be detected in industrial datasets, a common approach is to use a sliding window to cut subimages into 1:1 aspect ratios, thus standardizing the data shape. However, this results in the discarding of most background subimages without defects, reducing the complexity of the samples within the dataset. To fully utilize background subimages, this invention designs a background reuse algorithm. By concatenating background subimages with defective subimages to create a new image, background reuse is achieved, enabling the construction of samples with more complex backgrounds and improving the overall sample quality within the dataset.

[0006] II. Based on the above principles, the present invention is implemented through the following scheme:

[0007] A sample augmentation method for small sample surface defect datasets includes the following steps:

[0008] (1) Obtaining raw data and manual labels:

[0009] Collect surface images of defective products and manually annotate the areas to be inspected and the defects in the images to construct a small sample surface defect dataset D;

[0010] (2) Construct a sample augmentation algorithm, which consists of an environment-guided defect replication algorithm and a background reuse algorithm. The specific construction process includes the following steps:

[0011] (2-a) Constructing an environment-guided defect replication algorithm: Based on manually labeled tags, extract defect target pixel blocks ST from dataset D. i and its eight neighboring pixel blocks constituting defective pixel block group G i Both i and j are positive integers, and i∈[1,M], j∈[1,8], where M represents the number of defective targets in dataset D; based on the defective pixel block group G iThe defect category of the target pixel block in the middle divides the defect pixel block group into defect pixel block groups. Defect pixel block groups of the same category belong to the same defect pixel block group set S. t Where t is a positive integer and t∈[1,C], and C represents the number of defect categories in dataset D; for any image I in dataset D, select from all defect pixel block groups. A group of defective pixel blocks, of which P t Indicates from set S t The number of randomly selected defective pixel block groups is used to generate a candidate region of the same size as the defective target pixel block in each selected defective pixel block group within the inspection area of ​​image I. The covariance Cov of the eight neighboring pixel blocks of the candidate region and the eight neighboring pixel blocks in the selected defective pixel block group is calculated to measure the environmental similarity. The specific formula is as follows:

[0012]

[0013] Among them SE j TE represents the j-th neighboring pixel block among the eight neighboring pixel blocks of the defective target pixel block. j This represents the j-th neighboring pixel block among the eight neighboring pixel blocks of the defective target pixel block to be pasted. This represents pixel-by-pixel multiplication, and E(·) represents calculating the mean. When Cov is greater than the threshold TH, the defective target pixel block in the defective pixel block group is considered compatible with the environment of the candidate region, and the defective target pixel block is pasted into the candidate region. When Cov is less than or equal to TH, the defective target pixel block and the candidate region are discarded, and the defective target pixel block and its candidate region in the next selected defective pixel block group are processed. The algorithm uses a given P t The value is used to control the number of times different types of defects are replicated, thereby achieving defect augmentation and class balance within the dataset; where P t It is an integer, and P t ∈[0, 10000], TH is a real number, and TH∈(-1, 1);

[0014] The method for extracting the eight neighboring pixel blocks in this step is as follows: calculate the x and y coordinates of the top left and bottom right corners of each neighboring pixel block, and extract rectangular neighboring pixel blocks based on the coordinates of these two corners; for defective target pixel blocks or candidate regions, the formulas for calculating the corner coordinates of their eight neighboring pixel blocks are as follows:

[0015] AE 1 =(Lx+W, Ly-H; Rx+W, Ry-H)

[0016] AE 2= (Lx, Ly-H; Rx, Ry-H)

[0017] AE 3 =(Lx-W, Ly-H; Rx-W, Ry-H)

[0018] AE 4 = (Lx+W, Ly; Rx+W, Ly)

[0019] AE 5 = (Lx-W, Ly; Rx-W, Ly)

[0020] AE 6 =(Lx+W, Ly+H; Rx+W, Ry+H)

[0021] AE 7 = (Lx, Ly+H; Rx, Ry+H)

[0022] AE 8 =(Lx-W, Ly+H; Rx-W, Ry+H)

[0023] The above 8 formulas can be uniformly represented as AE j =(index) lx index ly index rx index ry ), where AE j The index represents the corner coordinates of the j-th neighboring pixel block of the defective target pixel block or candidate region. lx index ly These represent the x and y coordinates of the top-left corner point, respectively. rx index ry Lx, Ly, Rx, Ry, W, and H represent the x-coordinate and y-coordinate of the bottom right corner point, respectively; Lx, Ly, Rx, Ry, W, and H represent the x-coordinate, y-coordinate, bottom right corner point, width, and height of the top left corner point, the bottom left corner point, the bottom right corner point, and the candidate pixel block or region, respectively.

[0024] (2-b) Constructing a background reuse algorithm: A sliding window of size L×L pixels is used, with a sliding step of S pixels each time. The region to be inspected of any image I in the dataset D is slid-cropped. Each slid-crop generates a sub-image of size L×L pixels. The sub-image is divided into background sub-image and defect sub-image according to whether it contains defects. All background sub-images constitute the background sub-image set, and all defect sub-images constitute the defect sub-image set. Three sub-images are randomly selected from the background sub-image set, and one sub-image is randomly selected from the defect sub-image set. These four selected sub-images are randomly spliced ​​into a new image with an aspect ratio of 1:1, until the number of new images generated is Q times the number of images in the defect sub-image set. All the new images generated after splicing and all defect sub-images generated by the sliding window sliding together constitute the training set in the dataset. Where L, S and Q are all positive integers, and L, S, Q∈[1,10000];

[0025] (3) Sample augmentation:

[0026] For the small sample surface defect dataset D obtained in step (1), the environment-guided defect replication algorithm constructed in step (2-a) and the background reuse algorithm constructed in step (2-b) are applied in sequence to achieve class balance and sample augmentation, and to construct the augmented surface defect dataset D_En.

[0027] In step (2-a), the threshold TH is preferably 0.5.

[0028] In step (2-b), the parameters L, S, and Q are preferably 90, 45, and 1, respectively.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] (1) The environment-guided defect replication algorithm constructed in this invention describes the environment with eight neighboring pixel blocks. By calculating the covariance between the eight neighboring pixel blocks of the defect target and the eight neighboring pixel blocks of the candidate region, the similarity of the environment is described. The target replication can be guided efficiently, and the semantic similarity between the target and the candidate region is guaranteed while improving the efficiency of the algorithm.

[0031] (2) The background reuse algorithm constructed in this invention is based on the sub-image generated by the sliding window. The background sub-image and the defect sub-image are spliced ​​together to realize background reuse, thereby improving the complexity of the samples in the dataset. Attached Figure Description

[0032] Figure 1 Flowchart of a sample augmentation method for small sample surface defect datasets according to an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the environment-guided defect replication algorithm according to an embodiment of the present invention;

[0034] Figure 3 Background reuse algorithm principle diagram of the embodiments of the present invention;

[0035] Figure 4 A comparison chart of the detection performance of the model trained on the original dataset and the model trained on the augmented dataset in this embodiment of the invention. Detailed Implementation

[0036] The specific embodiments of the present invention are described below:

[0037] Example 1

[0038] Figure 1 The diagram shows a flowchart of a sample augmentation method for small sample surface defect datasets according to an embodiment of the present invention. The specific steps are as follows:

[0039] Step 1: Obtain raw data and manual labels.

[0040] Collect surface images of defective products and manually annotate the areas to be inspected and the defects in the images to construct a small sample surface defect dataset D.

[0041] Step 2: Construct a sample augmentation algorithm, which consists of an environment-guided defect replication algorithm and a background reuse algorithm. The specific construction process includes the following steps:

[0042] (2-a) Constructing an environment-guided defect replication algorithm: Based on manually labeled tags, extract defect target pixel blocks ST from dataset D. i and its eight neighboring pixel blocks constituting defective pixel block group G i , where i and j are both positive integers, and i∈[1,12736], j∈[1,8], where 12736 is the number of defective targets in dataset D. These defects are divided into 4 categories: 523 scratch defects, 68 laser engraving defects, 11954 particle defects, and 191 pressure defects; based on the defective pixel block group G i The defect category of the target pixel block in the middle divides the defect pixel block group into defect pixel block groups. Defect pixel block groups of the same category belong to the same defect pixel block group set S. t Where t is a positive integer, representing the t-th type of defect, and t∈[1,4]; for any image I in dataset D, select from all defect pixel block sets A group of defective pixel blocks, of which P t Indicates from set S tThe number of randomly selected defective pixel block groups is used to generate a candidate region of the same size as the defective target pixel block in each selected defective pixel block group within the inspection area of ​​image I. The covariance Cov of the eight neighboring pixel blocks of the candidate region and the eight neighboring pixel blocks in the selected defective pixel block group is calculated to measure the environmental similarity. The specific formula is as follows:

[0043]

[0044] Among them SE j TE represents the j-th neighboring pixel block among the eight neighboring pixel blocks of the defective target pixel block. j This represents the j-th neighboring pixel block among the eight neighboring pixel blocks of the defective target pixel block to be pasted. This represents pixel-by-pixel multiplication, and E(·) represents calculating the mean. When Cov is greater than the threshold TH, the defective target pixel block in the defective pixel block group is considered compatible with the environment of the candidate region, and the defective target pixel block is pasted into the candidate region. When Cov is less than or equal to TH, the defective target pixel block and the candidate region are discarded, and the defective target pixel block and its candidate region in the next selected defective pixel block group are processed. The algorithm uses a given P t The value is used to control the number of times different types of defects are replicated, thereby achieving defect augmentation and class balance within the dataset; where P t It is an integer, and P t ∈[0, 10000], TH is a real number and TH∈(-1, 1), in this embodiment P1, P2, P3 and P4 are 50, 50, 0 and 100 respectively, and TH is 0.5;

[0045] The method for extracting the eight neighboring pixel blocks in this step is as follows: calculate the x and y coordinates of the top left and bottom right corners of each neighboring pixel block, and extract rectangular neighboring pixel blocks based on the coordinates of these two corners; for defective target pixel blocks or candidate regions, the formulas for calculating the corner coordinates of their eight neighboring pixel blocks are as follows:

[0046] AE 1 =(Lx+W, Ly-H; Rx+W, Ry-H)

[0047] AE 2 = (Lx, Ly-H; Rx, Ry-H)

[0048] AE 3 =(Lx-W, Ly-H; Rx-W, Ry-H)

[0049] AE 4 = (Lx+W, Ly; Rx+W, Ly)

[0050] AE 5 = (Lx-W, Ly; Rx-W, Ly)

[0051] AE 6 =(Lx+W, Ly+H; Rx+W, Ry+H)

[0052] AE 7 = (Lx, Ly+H; Rx, Ry+H)

[0053] AE 8 =(Lx-W, Ly+H; Rx-W, Ry+H)

[0054] The above 8 formulas can be uniformly represented as AE j =(index) lx index ly index rx index ry ), where AE j The index represents the corner coordinates of the j-th neighboring pixel block of the defective target pixel block or candidate region. lx index ly These represent the x and y coordinates of the top-left corner point, respectively. rx index ry Lx, Ly, Rx, Ry, W, and H represent the x-coordinate and y-coordinate of the bottom right corner point, respectively; Lx, Ly, Rx, Ry, W, and H represent the x-coordinate, y-coordinate, bottom right corner x-coordinate, bottom right corner y-coordinate, width, and height of the defective target pixel block or candidate region, respectively, where j is a positive integer.

[0055] (2-b) Constructing a background reuse algorithm: A sliding window of size L×L pixels is used, with a sliding step size of S pixels each time. The region to be inspected of any image I in the dataset D is slid-cropped. Each slid-crop generates a sub-image of size L×L pixels. The sub-image is divided into background sub-image and defect sub-image according to whether it contains defects. All background sub-images constitute the background sub-image set, and all defect sub-images constitute the defect sub-image set. Three sub-images are randomly selected from the background sub-image set, and one sub-image is randomly selected from the defect sub-image set. These four selected sub-images are randomly spliced ​​into a new image with an aspect ratio of 1:1 until the number of new images generated is Q times the number of images in the defect sub-image set. All the new images generated after splicing and all defect sub-images generated by the sliding window sliding together constitute the training set in the dataset. Where L, S and Q are all positive integers, and L, S, Q ∈ [1, 10000]. In this embodiment, L, S, Q are 90, 45 and 1, respectively.

[0056] Step 3, sample augmentation.

[0057] For the small sample surface defect dataset D obtained in step 1, the environment-guided defect replication algorithm constructed in step (2-a) and the background reuse algorithm constructed in step (2-b) are applied in sequence to achieve class balance and sample augmentation, and an augmented surface defect dataset D_En is constructed.

[0058] Example 2

[0059] The general object detection algorithm Scaled-YOLOv4 was trained using the small sample surface defect dataset D and the augmented surface defect dataset D_En constructed in Example 1, respectively. The two trained algorithms were then tested on the test set of dataset D. This experiment used Ubuntu 18.04 as the operating system and PyTorch 1.2.0 framework based on CUDA 10.0 and cuDNN 7.6.0. The training and testing were conducted on a personal computer equipped with an Intel(R) Xeon(R) W-2123 CPU (3.60GHz) and an NVIDIA GeForce RTX 2080Ti (11GB) graphics card.

[0060] This embodiment uses four metrics—mAP, mAP50, mAP75, and mAR—to evaluate the performance of the general object detection algorithm Scaled-YOLOv4 trained on the D and D_En datasets, respectively. The comparison results are shown in Table 1. It can be seen that the detection algorithm trained on the dataset augmented by the method proposed in this invention achieves optimal performance on all four evaluation metrics.

[0061] Table 1

[0062]

[0063] Figure 4 The image shows a comparison of the detection results of a general object detection algorithm trained on the original dataset and the augmented dataset. The first column shows the original image, the second column shows the manually labeled image, the third column shows the inference results of training Scaled-YOLOv4 on dataset D, and the fourth column shows the inference results of training Scaled-YOLOv4 on dataset D_En augmented by the method of this invention. The results demonstrate that the dataset augmented by this invention makes the algorithm's detection results more accurate.

[0064] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, any changes made in accordance with the shape and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A sample augmentation method for small sample surface defect dataset, characterized in that Comprising the following steps: (1) Obtain original data and artificial labels: Collect the surface images of defective products, and manually label the inspection areas and defects in the images to construct a small sample surface defect dataset D; (2) Construct a sample augmentation algorithm, which is composed of an environment-guided defect replication algorithm and a background reuse algorithm, and the specific construction process includes the following steps: (2-a) Constructing environment-guided defect replication algorithm: based on the manually annotated labels, extract defect target pixel blocks ST existing in the dataset D i and their corresponding eight neighborhood pixel blocks Form defect pixel block group G i , i and j are positive integers, and i ∈ [1, M], j ∈ [1, 8], where M represents the number of defect targets existing in the dataset D; according to the defect categories of the defect target pixel blocks in the defect pixel block group G i , the defect pixel block group is divided, and defect pixel blocks of the same category belong to the same defect pixel block group set S t , where t is a positive integer, and t ∈ [1, C], C represents the number of defect categories in the dataset D; for any one image I in the dataset D, select P defect pixel block groups from all defect pixel block group sets, where P t represents the number of defect pixel block groups randomly selected from the set S t , in the to-be-inspected area of the image I, a candidate area with the same size as the defect target pixel block is randomly generated for each defect target pixel block in the selected defect pixel block group, and the covariance Cov of the eight neighborhood pixel blocks of the candidate area and the eight neighborhood pixel blocks in the selected defect pixel block group is calculated to measure the environmental similarity, when Cov is greater than the threshold TH, it is considered that the defect target pixel block in the defect pixel block group is compatible with the environment of the candidate area, and the defect target pixel block is pasted to the candidate area, when Cov is less than or equal to TH, the defect target pixel block and the candidate area are abandoned, and the defect target pixel block in the next selected defect pixel block group and its candidate area are processed; the algorithm controls the replication times of different types of defects by the given P t value, realizes the defect augmentation and category balance in the dataset; where P t is an integer, and P t ∈ [0, 10000], TH is a real number, and TH ∈ (-1, 1). (2-b) Construct a background reuse algorithm: use a sliding window with a size of LxL pixels, and the step size of each sliding is S pixels, slide and cut the inspection area of any image I in the dataset D, and generate a sub-image with a size of LxL pixels each time. According to whether the sub-image contains defects, the sub-image is divided into background sub-images and defect sub-images. All background sub-images constitute a background sub-image set, and all defect sub-images constitute a defect sub-image set. Randomly select three sub-images from the background sub-image set and one sub-image from the defect sub-image set, and randomly splice the four selected sub-images into a new image with an aspect ratio of 1:

1. Until the number of generated new images is Q times the number of images in the defect sub-image set. All new images generated after splicing and all defect-containing sub-images generated by sliding window cutting together constitute the training set in the dataset. Wherein L, S and Q are positive integers, and L, S, Q [1, 10000]; (3) Sample augmentation: For the small sample surface defect dataset D obtained in step (1), the environment-guided defect replication algorithm constructed in step (2-a) and the background reuse algorithm constructed in step (2-b) are applied in turn to realize class balancing and sample augmentation, and an augmented surface defect dataset D_En is constructed.

2. The sample augmentation method for small sample surface defect dataset of claim 1, wherein, The eight neighborhood pixel block extraction method in step (2-a) calculates the horizontal and vertical coordinates of the upper left corner point and the horizontal and vertical coordinates of the lower right corner point of each neighborhood pixel block, and extracts the rectangular neighborhood pixel block based on the coordinates of the two corner points. For the defect target pixel block or candidate region, the corner point coordinate calculation formulas of its eight neighborhood pixel blocks are respectively: AE 1 = (Lx + W, Ly - H; Rx + W, Ry - H) AE 2 = (Lx, Ly - H; Rx, Ry - H) AE 3 = (Lx-W, Ly-H; Rx-W, Ry-H) AE 4 = (Lx + W, Ly; Rx + W, Ry) AE 5 = (Lx - W, Ly; Rx - W, Ry) AE 6 = (Lx + W, Ly + H; Rx + W, Ry + H) AE 7 = (Lx, Ly + H; Rx, Ry + H) AE 8 = (Lx - W, Ly + H; Rx - W, Ry + H) The above eight formulas can be uniformly expressed as AE j = (index lx , index ly ; index rx , index ry ), wherein AE j represents the corner point coordinates of the jth neighborhood pixel block of the defect target pixel block or the candidate region, j is a positive integer, and j ∈ [1, 8], index lx , index ly respectively represent the horizontal coordinates and the vertical coordinates of the upper left corner point, and index rx , index ry respectively represent the horizontal coordinates and the vertical coordinates of the lower right corner point; Lx, Ly, Rx, Ry, W, and H respectively represent the horizontal coordinates of the upper left corner point, the vertical coordinates of the upper left corner point, the horizontal coordinates of the lower right corner point, the vertical coordinates of the lower right corner point, the width, and the height of the defect target pixel block or the candidate region.

3. The sample augmentation method for small sample surface defect dataset of claim 1, wherein, The environment similarity calculation method in step (2-a) has the following specific formula: where SE j denotes the jth neighbor pixel block of the eight neighbor pixel blocks of the defective target pixel block, TE j denotes the jth neighbor pixel block of the eight neighbor pixel blocks of the defective target pixel block to be pasted, the symbol denotes pixel-wise multiplication, E(·) denotes computing the mean.

Images