A defect generation method for deep learning tasks

Abstract

The present application provides a defect generation method for deep learning tasks, the method comprising: obtaining a defect image based on a defect sample, the defect image being a rectangular area containing a defect within the defect sample; constructing a defect sample library based on the defect images corresponding to a preset number of defect samples; extracting at least one defect image from the defect sample library as a target image; injecting the target image into a preset background image based on the deep learning task type and transformation conditions to generate a transformed image; and generating a corresponding task image and annotation file based on the transformed image. The defect generation method provided in the present application solves the problem that when the number of defect samples and normal samples is unbalanced, deep learning training is very difficult, resulting in poor defect detection results.

Description

Technical Field

[0001] The present application relates to the field of defect detection, and in particular to a defect generation method for deep learning tasks. Background Art

[0002] In the field of defect detection, deep learning is a commonly used method for solving complex product defect detection tasks. Deep learning involves learning the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful for interpreting data such as text, images, and sound. Deep learning model training requires training on a training set, validation on a validation set, saving the optimal weights, and reading them. The accuracy of the training and validation sets is then recorded to facilitate parameter adjustment.

[0003] In actual production, there are very few defective samples that can be trained, but there are more normal samples, resulting in an imbalance in the number of defective samples and normal samples. Usually, the method uses expanding the data set, resampling the data set, or manually generating data samples. However, expanding the data set adds a large amount of sample data, resulting in an increase in workload. Resampling the data set and manually generating data samples increase manpower and material resources.

[0004] Deep learning training also requires a sufficient number of samples to ensure detection results, so deep learning training is currently very difficult, resulting in poor defect detection results. Summary of the Invention

[0005] This application provides a defect generation method for deep learning tasks to solve the problem that deep learning training is very difficult, resulting in poor defect detection results.

[0006] This application provides a defect generation method for deep learning tasks, comprising:

[0007] Acquire a defect image according to the defect sample, where the defect image is a rectangular area containing the defect inside the defect sample;

[0008] Constructing a defect sample library according to the defect images corresponding to a preset number of defect samples;

[0009] extracting at least one defect image from the defect sample library as a target image;

[0010] Injecting the target image into the preset background image according to the deep learning task type and transformation conditions to generate a transformed image;

[0011] According to the transformed image, a corresponding task image and an annotation file are generated.

[0012] Optionally, the step of obtaining a defect image according to the defect sample includes:

[0013] Generate a local area thumbnail in the defect sample, where the local area thumbnail is a rectangular area containing the defect;

[0014] Drawing the outer contour of the defect on the local area image to generate a defect image and a mask image of the same size;

[0015] Generating a minimum circumscribed rectangle of the defect outer contour according to the defect image outer contour;

[0016] The minimum circumscribed rectangle is expanded around, and the expanded rectangular area is used as the defect image.

[0017] Optionally, when the minimum circumscribed rectangle is expanded on all sides, the expanded pixel length is 3px-5px.

[0018] Optionally, the transformation conditions include: horizontal mirror transformation, vertical mirror transformation, special angle rotation transformation, affine transformation, brightness transformation, contrast transformation, blur transformation, noise transformation, and sharpening transformation, and each transformation condition includes an amplitude and a probability.

[0019] Optionally, before the step of generating the corresponding task image and annotation file, the method also includes: judging whether other transformations need to be added based on the transformed image; if so, regenerating the transformed image based on the added transformation conditions; if not, generating the corresponding task image and annotation file.

[0020] Optionally, the steps of generating the corresponding task image and annotation file include:

[0021] According to the corresponding tasks of the background image, different annotation file generation methods are adopted;

[0022] The transformation conditions of the background image and the target image are configured to generate corresponding task images and annotation files.

[0023] Optionally, generating a corresponding task image and annotation file based on the transformed image includes:

[0024] generating a corresponding auxiliary image according to the background image;

[0025] transforming the background image according to the transformation condition to generate a transformed background image;

[0026] Randomly generate an injection position according to the width and height of the background image and the width and height of the defect image;

[0027] generating an auxiliary image sub-region and a background image sub-region according to the position;

[0028] Determining whether the target image exists in the auxiliary image sub-region;

[0029] If the target image exists, randomly generating an injection position based on the width and height of the background image and the width and height of the defect image;

[0030] If the target image does not exist, performing the same transformation on the defect image and the mask image according to the transformation condition;

[0031] fusing the defect image into a sub-region of the background image;

[0032] The mask image corresponding to the defect image is fused into the sub-region of the auxiliary image to generate an auxiliary image.

[0033] Optionally, for different deep learning task types, determine whether to generate a labeling file:

[0034] If the deep learning task type is a classification task, no annotation file is generated;

[0035] If the deep learning task type is a segmentation task, saving the auxiliary image as a labeling file;

[0036] If the deep learning task type is a detection task, contour analysis is performed on the auxiliary image to obtain the minimum circumscribed rectangle of each contour as the annotation box, and a corresponding json file or xml annotation file is generated at the same time.

[0037] Optionally, the defect image is fused into a subregion of the background image, and after eroding the mask image, the fusion is performed using a Poisson function. Poisson fusion involves placing a target image into the background image, positioning it within the size of a foreground image centered at point P in the background image. The fusion process alters the color and gradient of the target image, achieving a seamless and enhanced fusion effect.

[0038] Optionally, injecting the target image into the background image according to the deep learning task type and the transformation condition to generate the transformed image includes:

[0039] generating an initial transformation list according to the transformation conditions;

[0040] Setting probabilities or adjusting transformation orders for the initial transformation list to generate a transformation list;

[0041] The background image and the target image are transformed according to the order of the transformation list.

[0042] It can be seen from the above technical solution that the present application provides a defect generation method for deep learning tasks, the method comprising: obtaining a defect image according to a defect sample, the defect image being a rectangular area containing defects inside the defect sample; constructing a defect sample library according to the defect images corresponding to a preset number of defect samples; extracting at least one defect image from the defect sample library as a target image; injecting the target image into the preset background image according to the deep learning task type and transformation conditions to generate a transformed image; generating a corresponding task image and annotation file based on the transformed image. This method utilizes a small number of defect samples (NG Sample) and a large number of normal sample images (OKSample) to generate defects on a normal sample image (OK Sample) to increase the number of defect samples, alleviate the problem of data imbalance, and improve the detection effect. The defect generation method provided by the present application solves the problem that when the number of defect samples and normal samples is unbalanced, deep learning training is very difficult, resulting in poor defect detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of the steps of a defect generation method for deep learning tasks;

[0045] Figure 2 Schematic diagram of the steps for generating the corresponding task images and annotation files;

[0046] Figure 3 is a schematic diagram of performing equivalent transformation on the defect image and the mask image;

[0047] Figure 4 Construct an interactive diagram for the defect sample library;

[0048] Figure 5 Schematic diagram of defect image generation method. DETAILED DESCRIPTION

[0049] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0050] Defect detection is a critical industrial application. Due to the diverse nature of defects, traditional machine vision algorithms struggle to fully model and transfer defect features. They are also limited in reusability and require differentiation between different working conditions, which wastes significant manpower. Deep learning has achieved remarkable results in feature extraction and localization, leading to its introduction into the field of defect detection.

[0051] Deep learning uses machine learning models with multiple hidden layers and massive amounts of training data to learn features, ultimately improving the accuracy and versatility of classification or prediction. However, training an effective deep learning model requires a large amount of labeled data, extensive graphics card resources, and lengthy training times. In many industrial scenarios, acquiring defect images is prohibitively expensive, resulting in a very limited sample size, making it difficult to directly train deep learning models. Consequently, the resulting defect detection models have high false positive and missed detection rates, making them difficult to meet the needs of businesses.

[0052] This application provides a defect generation method for deep learning tasks, see Figure 1 , which is a schematic diagram of the steps of a defect generation method for deep learning tasks, including:

[0053] S1: Obtain a defect image based on the defect sample. The defect image is a rectangular area containing the defect inside the defect sample.

[0054] Specifically, the defect sample can be a normal sample image (OK Sample) or a defect sample (NG Sample). Since there are fewer defect samples (NG Sample) and more normal sample images (OK Sample) in actual production, the defect sample normal sample images (OK Sample) account for a large proportion in this embodiment. Common defects include: planar object defects, three-dimensional object defects, such as: scratches, bumps, cracks, edge cracks and stains.

[0055] S2: Construct a defect sample library based on the defect images corresponding to a preset number of defect samples;

[0056] See also Figure 4 , which is a schematic diagram of an interactive method for constructing a defect sample library; in some embodiments, the construction of the defect sample library is carried out in an interactive manner. First, a rectangular area with defects is selected in the defect sample, which is a small local area image, such as Figure 4 At A in the figure, the outer contour of the defect is drawn on the rectangular area, as shown in Figure 4 At point B in the figure, the minimum bounding rectangle of the defect is generated according to the outer contour of the defect, as shown in Figure 4 At C in the middle.

[0057] S3: extract at least one defect image from the defect sample library as a target image;

[0058] It should be understood that the process of extracting the target image is random.

[0059] S4: inject the target image into the preset background image according to the deep learning task type and transformation conditions to generate the transformed image;

[0060] Specifically, the preset background image is a normal sample image (OK Sample);

[0061] In order to improve the diversity of defect samples, the target image is subjected to the selection of learning task type and conditional transformation.

[0062] See also Figure 5 , is a schematic diagram of the defect image generation method. In order to make the process of injecting the target image into the background image more efficient, multiple target images that have completed the learning task type and transformation conditions are superimposed, and the defects in multiple target images are injected into the same background image to generate the transformed image;

[0063] S5: Generate the corresponding task image and annotation file based on the transformed image.

[0064] Through steps S1-S5, a small number of non-normal samples (NG Sample) and a large number of normal sample images (OK Sample) are used to generate defects on the normal sample images (OK Sample), thereby increasing the number of defective samples, alleviating the problem of data imbalance, and improving the detection effect.

[0065] According to the defect sample, the steps of obtaining the defect image include:

[0066] Generate a local area thumbnail in the defect sample, where the local area thumbnail is a rectangular area containing the defect;

[0067] Draw the outer contour of the defect on the local area small image to generate a defect image and a mask image of the same size; the grayscale value is 0 only inside the defect area, and the rest of the value is 255;

[0068] According to the outer contour of the defect image, the minimum circumscribed rectangle of the defect outer contour is generated;

[0069] The minimum circumscribed rectangle is expanded and the expanded rectangular area is used as the defect image. That is, the defect image is the rectangular area inside the defect sample that contains the defect.

[0070] In some embodiments, when the minimum bounding rectangle is expanded, the pixel length of the expansion is 3px-5px. If an exception occurs during the defect outline drawing process or the defect is not fully selected, the minimum bounding rectangle is expanded to expand the selection range to ensure that the defect is within the selected area.

[0071] For example: the size of the local area thumbnail generated by the defect sample is 25px*25px. This rectangular area contains the defect. The outer contour of the defect is drawn on the 25px*25px area. After generating the defect image and mask image of the same size, for example, the minimum enclosing rectangle generated according to the outer contour of the defect is 6px*7px, and the 6px*7px minimum enclosing rectangle is expanded on all sides, and each side of the minimum enclosing rectangle is expanded by 3px-5px. After expansion, it can be 12px*13px-16px*17px, and the 12px*13px-16px*17px rectangular area is used as the defect image.

[0072] In some embodiments, the transformation conditions include: angle transformation of the simulated size position, illumination transformation, and imaging detail transformation, and each transformation condition includes an amplitude and a probability.

[0073] Specifically, for example: horizontal mirroring, vertical mirroring, special angle rotation, such as: 30 degrees, 90 degrees, 180 degrees, etc., affine transformation, such as: translation, rotation, scaling, shearing, etc., illumination transformation, such as brightness transformation, contrast transformation, imaging detail transformation, such as blur, noise, sharpening, and other transformations can be added as needed.

[0074] When the user (terminal) command is "rotate 30 degrees", the system rotates the image 30 degrees. If the generated image does not meet the requirements, the user performs the next command "scale 3px", and the system scales the image size by 3px. Based on the generated image, it is determined whether other transformations need to be added.

[0075] Before the step of generating the corresponding task image and annotation file, the method also includes: judging whether other transformations need to be added based on the transformed image, and if so, regenerating the transformed image based on the added transformation conditions; if not, generating the corresponding task image and annotation file.

[0076] The steps to generate the corresponding task image and annotation file include:

[0077] Different annotation file generation methods are adopted according to the corresponding tasks of the background image;

[0078] The corresponding tasks can be divided into classification tasks, segmentation tasks and detection tasks.

[0079] Configure the transformation conditions of the background image and the target image, and generate the corresponding task image and annotation file.

[0080] See also Figure 2 、 Figure 3 , Figure 2 Schematic diagram of the steps to generate the corresponding task image and annotation file, Figure 3 A schematic diagram of performing equivalent transformations on the defect image and the mask image, generating corresponding task images and annotation files based on the transformed images, including:

[0081] S501: Generate a corresponding auxiliary image based on the background image;

[0082] Specifically, the corresponding auxiliary images are images with grayscale values ​​of 255;

[0083] S502: transforming the background image according to the transformation condition to generate a transformed background image;

[0084] The background transformation conditions and target transformation conditions are configured interactively. The transformations are pre-designed and then selected from the pre-designed transformations. This method selects commonly used transformations, for example, horizontal mirroring of the background image to obtain the transformed background image.

[0085] S503: randomly generating an injection position according to the width and height of the background image and the width and height of the defect image;

[0086] S504: Generate an auxiliary image sub-region and a background image sub-region according to the position;

[0087] For example, set the width of the background image to W and the height to H, set the width of the defect image to w and the height to h, and randomly generate the injection position to x and y, where x = rand(0, Ww-1) and y = rand(0, Hh-1). Then the auxiliary image sub-area is (x, y, x+w, y+h).

[0088] S505: Determine whether the target image exists in the auxiliary image sub-region; that is, determine whether the number of pixels with a gray value of 0 is 0;

[0089] If the target image exists, a random injection position is generated based on the width and height of the background image and the width and height of the defect image.

[0090] If the step of determining whether the target image exists in the auxiliary image sub-region has reached 10 times and the satisfied point x and y cannot be obtained, then skip the defect sample and proceed to the next defect sample determination;

[0091] If the target image does not exist, the defect image and the mask image are transformed identically according to the transformation conditions;

[0092] For example, if the target image does not exist, the brightness of the defect image and the mask image can be transformed, and whether the next transformation needs to be added can be determined based on the actual situation.

[0093] S506: Fusing the defect image into the sub-region of the background image;

[0094] S507: Fusing the mask image corresponding to the defect image into the sub-region of the auxiliary image to generate an auxiliary image.

[0095] Determine whether to generate annotation files for different deep learning task types:

[0096] If the deep learning task type is a classification task, no annotation file is generated;

[0097] If the deep learning task type is a segmentation task, save the auxiliary image as a labeling file;

[0098] If the deep learning task type is a detection task, contour analysis is performed on the auxiliary image to obtain the minimum enclosing rectangle of each contour as the annotation box, and the corresponding json file or xml annotation file is generated at the same time.

[0099] When the user (terminal) does not need to generate an annotation file, the system selects the deep learning task type as a classification task. When the user (terminal) needs to generate an annotation file in png format, the system will select the deep learning task type as a segmentation task. When the user (terminal) needs to generate an annotation file in json or xml format, the system will select the deep learning task type as a detection task.

[0100] Annotation files are generally divided into mat, json, xml, png, CSV, xlsx, txt, word and other formats. In this embodiment, json, xml or png format is generated.

[0101] Contour analysis, also known as contour detection, involves detecting object boundaries in an image and obtaining topological information. The first step involves binarizing the image; the second step involves labeling connected components; and the third step involves determining the contour's centroid, area, perimeter, and minimum enclosing rectangle.

[0102] The defect image is fused into the sub-region of the background image, and the mask image is eroded and then fused using the Poisson function.

[0103] The mask image is preprocessed. Image erosion can eliminate the boundary points of the object and shrink the boundary inward. It can remove objects smaller than the structural element and separate two objects with small connections. If there is a small connection between the two objects, the two objects can be separated when the structure is large enough.

[0104] In some embodiments, the defect image can be fused into a sub-area of ​​the background image by copying. The copying method may result in poor fusion between images, but Poisson fusion is to place a target image into the background image. The placement is within the size range of a foreground image centered on point P in the background image. The fusion process will change the color and gradient in the target image, so as to achieve a seamless fusion effect with better fusion effect.

[0105] According to the deep learning task type and transformation conditions, the target image is injected into the background image to generate the transformed image, including:

[0106] Generate an initial transformation list based on the transformation conditions;

[0107] Among them, the transformation range that can be selected for background image transformation and target image transformation is the same, namely size position angle transformation (horizontal mirror, vertical mirror, special angle rotation, affine transformation), illumination transformation (brightness transformation, contrast transformation) and imaging detail transformation (blur, noise, sharpening). For example, if horizontal mirror and vertical mirror are selected for background image transformation, any one or more of horizontal mirror, vertical mirror, special angle rotation, affine transformation, brightness transformation, contrast transformation, blur, noise and sharpening can be selected for target image transformation. Different transformation amplitudes and transformation probabilities are set for background transformation conditions and target transformation conditions, which can be freely combined. It should be noted that each transformation condition contains an amplitude and probability.

[0108] For example, if the background transformation of an image selects horizontal mirroring, translation transformation, 90-degree rotation, brightness transformation, blur transformation, and noise transformation in sequence, the initial background transformation list is shown in Table 1:

[0109] ID Change Name Transformation Amplitude Transformation probability 0 Horizontal Mirror none 1 1 Translation Transformation [-10,10] 1 2 Special angle transformation 90 degrees 1 3 Brightness Transformation [-30,30] 1 4 Fuzzy Transform [3,7] 1 5 Noise Transform [10,20] 1

[0110] Table 1

[0111] For example, if the target transformation of an image selects vertical mirroring, scaling transformation, 180-degree rotation, contrast transformation, blur transformation, and sharpening transformation in sequence, the initial target transformation list is shown in Table 2:

[0112] ID Change Name Transformation Amplitude Transformation probability 0 Mirror vertically none 1 1 Scaling Transform [-0.5,0.5] 1 2 Special angle transformation 180 degrees 1 3 Contrast transformation [0.5,1.5] 1 4 Fuzzy Transform [3,7] 1 5 Sharp transformation [3,7] 1

[0113] Table 2

[0114] Setting the probability of the initial transformation list or adjusting the transformation order to generate a transformation list;

[0115] Transforms the background image and the target image according to the order of the transformation list.

[0116] The steps for generating the transformed image according to the background transformation conditions and the target transformation conditions are as follows:

[0117] Set the probability for each transformation or adjust the transformation order to generate a transformation list;

[0118] In particular, for transformations with a transformation range, an intermediate value is randomly generated each time for transformation;

[0119] Specifically, the defect image and the mask image are transformed according to the target transformation list. Assuming that the probability of each transformation is not changed or the order of the transformation is not adjusted, the transformation is performed in the order listed in Table 2.

[0120] Transform the background image according to the background transformation list. Assuming that the probability of each transformation is not changed or the order of the transformation is not adjusted, the transformation is performed in the order listed in Table 1.

[0121] Before executing each transformation, a probability value between 0 and 1 is randomly generated;

[0122] If the probability value is less than the probability value set by the transformation, the transformation is executed; otherwise, it is not executed.

[0123] If the probability value is less than the probability value set for the transformation, the transformation is executed, otherwise it is not executed. It can be seen from the above technical solution that the present application provides a defect generation method for deep learning tasks, the method comprising: obtaining a defect image according to a defect sample, the defect image being a rectangular area containing defects inside the defect sample; constructing a defect sample library according to the defect images corresponding to a preset number of defect samples; extracting at least one defect image from the defect sample library as a target image; injecting the target image into the preset background image according to the deep learning task type and transformation conditions to generate a transformed image; generating a corresponding task image and annotation file based on the transformed image. This method realizes the generation of defects on a normal sample image (OK Sample) using a small number of non-normal samples (NG Sample) and a large number of normal sample images (OK Sample) to increase the number of defect samples, alleviate the problem of data imbalance, and improve the detection effect. The defect generation method provided by the present application solves the problem that when the number of defect samples and normal samples is unbalanced, deep learning training is very difficult, resulting in poor defect detection effect.

[0124] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A defect generation method for deep learning tasks, characterized in that: include: Acquire a defect image according to the defect sample, where the defect image is a rectangular area containing the defect inside the defect sample; According to the defect sample, the steps of obtaining the defect image include: Generate a local area thumbnail in the defect sample, where the local area thumbnail is a rectangular area containing the defect; Drawing the outer contour of the defect on the local area image to generate a defect image and a mask image of the same size; Generating a minimum circumscribed rectangle of the defect outer contour according to the defect image outer contour; Expanding the minimum circumscribed rectangle around the minimum circumscribed rectangle, and taking the expanded rectangular area as the defect image; Constructing a defect sample library according to the defect images corresponding to a preset number of defect samples; extracting at least one defect image from the defect sample library as a target image; The target image is injected into a preset background image according to the deep learning task type and transformation conditions to generate a transformed image, including: generating an initial transformation list according to the transformation conditions; Setting probabilities or adjusting transformation orders for the initial transformation list to generate a transformation list; transforming the background image and the target image according to the order of the transformation list; The transformation conditions include: at least one of horizontal mirror transformation, vertical mirror transformation, special angle rotation transformation, affine transformation, brightness transformation, contrast transformation, blur transformation, noise transformation and sharpening transformation, and each transformation condition includes an amplitude and a probability; According to the transformed image, a corresponding task image and an annotation file are generated.

2. The defect generation method for deep learning tasks according to claim 1, characterized in that: When the minimum circumscribed rectangle is expanded on all sides, the expanded pixel length is 3px-5px.

3. The defect generation method for deep learning tasks according to claim 1, characterized in that: Before the step of generating the corresponding task image and annotation file, the method also includes: judging whether other transformations need to be added based on the transformed image, and if so, regenerating the transformed image based on the added transformation conditions; if not, generating the corresponding task image and annotation file.

4. The defect generation method for deep learning tasks according to claim 1, characterized in that: The steps to generate the corresponding task image and annotation file include: According to the corresponding tasks of the background image, different annotation file generation methods are adopted; The transformation conditions of the background image and the target image are configured to generate corresponding task images and annotation files.

Images