Industrial product surface defect data set expansion method and device based on background modeling

Abstract

The invention relates to the technical field of image visual processing, in particular to an industrial product surface defect data set expansion method and device based on background modeling, and the method comprises the steps: extracting background texture information of a defect image through employing a background modeling method, and obtaining a background texture image data set of the defect image; training a generative adversarial network by using the background texture image data set of the defect image, and generating a new background texture image by using the trained generative adversarial network; performing difference operation on the defect image and the corresponding background texture image to obtain a defect foreground image; and fusing the defect foreground image with a background texture image generated by the generative adversarial network to obtain a fused defect image. According to the defect image obtained by the method, background texture information is reserved, defect features in the image are highlighted, diversity is achieved, and meanwhile the quality of the defect features is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image vision processing, and in particular to a method and device for expanding an industrial product surface defect data set based on background modeling. Background Art

[0002] Deep learning models have been developed very maturely in the mainstream field of computer vision and are usually applied in the scene of industrial product surface defect detection. Training a deep learning model often requires a large amount of rich data. In the scene of industrial product surface defect detection, the number of some surface defects is small, the structure of the appearance is single, and the imbalance of data restricts the improvement of deep learning model performance. Therefore, it is necessary to expand the data set of industrial product surface defects.

[0003] Currently, common methods for expanding industrial product surface defect datasets include data augmentation technology, synthetic data generation, image stitching, semi-supervised learning and self-supervised learning, database sharing and open source datasets, and transfer learning.

[0004] However, the surface defects of industrial products have their own unique characteristics and complexity. When expanding the defect dataset, traditional image processing methods often cannot effectively retain the complete information of the defects and their background, which may have an adverse effect on the quality of the generated images. Figure 1 As shown, the defect features within the box are of poor quality.

[0005] In recent years, some studies have used generative adversarial networks (GANs) as a method to expand industrial product surface defect datasets. However, due to the limitations of generative adversarial networks in capturing defect features, the generated industrial product surface defect images cannot meet high quality requirements. Summary of the invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the surface defect images of industrial products generated in the prior art cannot retain background information and have low defect quality.

[0007] In order to solve the above technical problems, the present invention provides a method for expanding an industrial product surface defect dataset based on background modeling, comprising:

[0008] The background texture information of the defect image is extracted by using the background modeling method to obtain a background texture image dataset of the defect image;

[0009] The background texture image dataset of the defect image is used to train a generative adversarial network, and a new background texture image is generated using the trained generative adversarial network;

[0010] Perform a difference operation between the defect image and its corresponding background texture image to obtain a defect foreground image;

[0011] The defect foreground image is fused with the background texture image generated by the generative adversarial network to obtain a fused defect image.

[0012] Preferably, the background texture information of the defect image is extracted by using a background modeling method to obtain a background texture image of the defect image, and the formula is:

[0013] BK (i,j) =θ T M(i,j)

[0014] Among them, BK is the background texture image of the defect image, θ T is the transpose of the regression coefficient matrix, M is the input defect image, and (i, j) is the pixel value coordinate of the image.

[0015] Preferably, a local weighted regression algorithm is used to calculate the regression coefficient matrix of the defect image.

[0016] Preferably, the loss function of the generative adversarial network is:

[0017]

[0018] Among them, G(z) is the fake sample generated by the generator, D is the discriminator, G is the generator, x~p data is the real sample, z~p z is the input noise sample, and AE is the sample space.

[0019] Preferably, if the input background texture image has a smooth surface, noise is introduced when a new background texture image is generated using the trained generative adversarial network.

[0020] Preferably, a difference operation is performed between the defect image and its corresponding background texture image to obtain a defect foreground image, and the formula is:

[0021] FG=|BK-M|

[0022] Among them, FG is the defect foreground image, BK is the background texture image of the defect image, and M is the input defect image.

[0023] Preferably, after obtaining the defect foreground image, the noise of the defect foreground image in a preset interval is filtered out to obtain an optimized defect foreground image, and the formula is:

[0024]

[0025] Among them, α is the preset threshold, and FG1 is the defect foreground image with noise filtered out.

[0026] Preferably, morphological operations are used to further improve the quality of the defect foreground image, and the formula is:

[0027]

[0028] Among them, FG2 is the defect foreground image after morphological operation optimization, S is the structural element, represents the morphological erosion operation, Represents the morphological dilation operation.

[0029] Preferably, the size of the structure element S is 2×4, expressed as

[0030] The present invention also provides an industrial product surface defect data set expansion device based on background modeling, comprising:

[0031] A background texture extraction module is used to extract background texture information of the defect image using a background modeling method to obtain a background texture image data set of the defect image;

[0032] A background texture image generation module is used to train a generative adversarial network using a background texture image dataset of a defect image, and generate a new background texture image using the trained generative adversarial network;

[0033] The defect foreground image acquisition module is used to perform a difference operation between the defect image and its corresponding background texture image to obtain the defect foreground image;

[0034] The fusion module is used to fuse the defect foreground image with the background texture image generated by the generative adversarial network to obtain a fused defect image.

[0035] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0036] The method for expanding the industrial product surface defect data set based on background modeling described in the present invention first extracts a background texture image from an existing defect image by a background modeling method, then uses a generative adversarial network to learn background texture information, and then generates more background texture images that conform to real scenes, and performs a differential operation on the defect image and the background texture image to obtain a defect foreground image, and finally fuses the defect foreground image and the background texture image generated by the generative adversarial network to obtain a fused defect image. The present invention can effectively expand the industrial product surface defect data set by continuously generating background texture images and then fusing them to obtain new defect images, and the defect images obtained by the present invention retain the background texture information and highlight the defect features in the image, and have diversity while ensuring the quality of the defect features. Using the defect images generated by the present invention as a training data set improves the robustness and accuracy in the industrial product surface defect detection and classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 It is a defect image generated by the traditional defect dataset expansion method;

[0039] Figure 2 It is a flow chart of a method for expanding a data set of industrial product surface defects based on background modeling according to the present invention;

[0040] Figure 3 It is the background texture image of the defect image extracted by the background modeling method;

[0041] Figure 4 It is a schematic diagram of the structure of the generative adversarial network;

[0042] Figure 5 This is the fusion defect image obtained by the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0044] Reference Figure 2 As shown, the present invention provides a method for expanding an industrial product surface defect data set based on background modeling, the steps comprising:

[0045] S1. Using a background modeling method to extract background texture information of a defect image, a background texture image training set of the defect image is obtained.

[0046] The background modeling method has significant advantages in dealing with complex industrial environments. It can effectively eliminate background interference and highlight defect areas. Based on the above relationship, the regression coefficient matrix θ' is calculated, and the background modeling of the defect image is performed to obtain the background texture image of the defect image. The formula is:

[0047] BK (i,j) =θ T M(i,j)

[0048] Among them, BK is the background texture image of the defect image, θ' T is the transpose of the regression coefficient matrix, M is the input defect image, and (i, j) is the pixel value coordinate of the image.

[0049] Preferably, the regression coefficient matrix of the defect image is calculated using a local weighted regression algorithm, and the formula includes:

[0050]

[0051] Among them, ω i is the i-th weight coefficient, and f(θ') is used to solve the weight coefficient ω i The loss function, x i is the i-th pixel, n is the total number of pixels to be calculated, y t In pixels x i is the center, the target value that needs to be fitted within the length interval of the pixel sequence length t, θ is the regression coefficient matrix; x' is the index value of the pixel sequence, and k is the Gaussian function parameter.

[0052] Convert the function f(θ) into matrix form:

[0053] f(θ)=(y-xθ)ω(y-xθ)

[0054] Then we get the relationship between the regression coefficient matrix and the weight matrix:

[0055] θ=(x T ωxθ) -1 x T ωy

[0056] Among them, x is the index value matrix of the pixel sequence, y is the pixel value matrix of the pixel sequence, and ω is the weight coefficient matrix.

[0057] The above formula only gives a mathematical explanation of the key steps of the local weighted regression algorithm and does not include the entire derivation process.

[0058] The background texture image reference of the defect image extracted by the background modeling method Figure 3 shown.

[0059] The purpose of background modeling is to better separate defect features and use background texture images to provide a reference baseline for defect detection, making subsequent defect detection and identification more accurate and efficient.

[0060] Preferably, before S1 extracts background texture information of the defect image using a background modeling method, the method further includes:

[0061] Collect defect data images:

[0062] First, use a CCD linear array industrial camera to collect image data of defective industrial products. In the image acquisition process, in order to ensure the convenience and accuracy of subsequent data processing, the light source of the acquisition environment should be kept good to reduce the impact of ambient light on data quality. A stable light source can effectively reduce the interference of light changes on image data and ensure that the acquired images are of high quality and consistency.

[0063] Make a defect dataset:

[0064] After obtaining the defect data, it is necessary to further process and annotate the data. First, the defect data is preprocessed to remove useless image data containing non-defects. Then, the retained defect image data is annotated in detail to ensure that the characteristics of each defect are accurately recorded. Finally, the annotated defect data is made into a standardized data set for subsequent analysis and processing.

[0065] S2. Use the background texture image dataset of the defect image to train a generative adversarial network, and use the trained generative adversarial network to generate a new background texture image.

[0066] Generative adversarial network is a powerful generative model that can generate high-quality background texture images that are highly similar to the real background texture by learning the background texture images of defect images. Figure 4 shown.

[0067] The generative adversarial network includes a discriminator and a generator. The discriminator can be used for training or testing, while the generator can only be used for testing. The generator captures the distribution of real samples and generates new fake samples based on the distribution; the discriminator is a binary classifier used to determine whether the input sample is a real sample or a fake sample. Through continuous adversarial training, the generator and the discriminator enable the discriminator to correctly identify the source of the training samples, and at the same time make the fake samples generated by the generator more similar to the real samples. The loss function of the generative adversarial network is:

[0068]

[0069] Among them, G(z) is the fake sample generated by the generator, D is the discriminator, G is the generator, x~p data is the real sample, z~p z is the input noise sample, and AE is the sample space.

[0070] For example, the number of rounds of generative adversarial network training in this embodiment is set to 200 times, the hardware environment is GPU GeForce RTX 3060, the processor is AMD Ryzen 75800H, the software environment is Python 3.7, and Pytorch is used as the learning framework.

[0071] In this step, the background texture image of the defect image is input into the generative adversarial network and trained for multiple iterations. The generative adversarial network gradually learns to generate realistic background texture images by learning and modeling a large number of background texture images, and finally generates a background texture image that is highly similar to the real background, ensuring that the generated background texture image is sufficiently authentic and consistent with the real background texture image, providing rich background information, which can be further effectively used for subsequent defect image generation.

[0072] It should be noted that when the background texture information of the defect image is extracted using the background modeling method in S1, the obtained background texture image may have a problem of too smooth texture. This overly smooth background texture image may cause the subsequently generated defect image to lack sufficient details and realism, thereby affecting the effect of defect detection. In order to solve this problem, this embodiment introduces moderate noise when generating background texture images using a generative adversarial network. The introduction of noise can increase the texture details of the background texture image and avoid distortion caused by being too smooth. It can not only enhance the authenticity of the background image, but also increase the diversity and complexity of the generated defect image to meet the actual needs of defect detection samples.

[0073] S3. Perform a differential operation on the defect image and its corresponding background texture image to obtain a defect foreground image.

[0074] The defect image is differentially calculated with the background texture image of the defect image to obtain the defect foreground image. The formula includes:

[0075] FG=|BK-M|

[0076] Among them, FG is the defect foreground image, BK is the background texture image of the defect image, and M is the input defect image.

[0077] Preferably, in order to improve the quality of the defect foreground image, the noise of the defect foreground image in a preset interval can also be filtered out to obtain an optimized defect foreground image, and the formula is:

[0078]

[0079] Wherein, α is the preset threshold, and FG1 is the defect foreground image after filtering out noise. The setting of the preset threshold α is often related to the quality of the defect foreground image. When the noise of the defect foreground is large, the preset threshold α should be appropriately increased.

[0080] Preferably, in order to improve the quality of the defect foreground image, morphological operation may be further used for denoising, so that the defect foreground image can achieve a better effect. The expression of the structure element S used in this embodiment is:

[0081]

[0082] The size of the structural element S is 2×4, and the kernel function of this structure is particularly conducive to denoising in special scenes such as track surfaces.

[0083] The formula for optimizing the defect foreground image based on the structural element S is:

[0084]

[0085] Among them, FG2 is the defect foreground image after morphological operation optimization. represents the morphological erosion operation, Represents the morphological dilation operation.

[0086] In this step, by calculating the pixel difference between the defect image and the background texture image of the defect image, the defect area is highlighted to obtain the defect foreground image, making the defect features more obvious and easier to identify. The defect foreground image retains the main features of the defect and provides important defect information for subsequent defect image generation and data set expansion.

[0087] S4, fusing the defect foreground image with the background texture image generated by the generative adversarial network to obtain a fused defect image. Figure 5 shown.

[0088] Fusion of defect foreground images with background texture images generated by generative adversarial networks not only retains the key features of the defects, but also integrates high-quality background texture information, thereby generating more realistic and diverse defect images, greatly enriching the diversity and representativeness of defect datasets. This method can effectively make up for the lack of diversity and authenticity in traditional data augmentation methods, making the fused defect images more in line with the actual situation and better meeting the needs of defect detection.

[0089] The method for expanding the industrial product surface defect data set based on background modeling described in the present invention first extracts a background texture image from an existing defect image by a background modeling method, then uses a generative adversarial network to learn background texture information, and then generates more background texture images that conform to real scenes, and performs a differential operation on the defect image and the background texture image to obtain a defect foreground image, and finally fuses the defect foreground image and the background texture image generated by the generative adversarial network to obtain a fused defect image. The present invention can effectively expand the industrial product surface defect data set by continuously generating background texture images and then fusing them to obtain new defect images, and the defect images obtained by the present invention retain the background texture information and highlight the defect features in the image, and have diversity while ensuring the quality of the defect features. The defect images generated by the present invention are used as training data sets to improve the robustness and accuracy in industrial product surface defect detection and classification tasks.

[0090] Based on the above-mentioned method for expanding an industrial product surface defect dataset based on background modeling, this embodiment further provides an industrial product surface defect dataset expansion device based on background modeling, including:

[0091] A background texture extraction module is used to extract background texture information of the defect image using a background modeling method to obtain a background texture image data set of the defect image;

[0092] A background texture image generation module is used to train a generative adversarial network using a background texture image dataset of a defect image, and generate a new background texture image using the trained generative adversarial network;

[0093] The defect foreground image acquisition module is used to perform a difference operation between the defect image and its corresponding background texture image to obtain the defect foreground image;

[0094] The fusion module is used to fuse the defect foreground image with the background texture image generated by the generative adversarial network to obtain a fused defect image.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for expanding the industrial product surface defect dataset based on background modeling. It is characterized in that include: The background texture information of the defect image is extracted by using the background modeling method to obtain a background texture image dataset of the defect image; The background texture image dataset of the defect image is used to train a generative adversarial network, and a new background texture image is generated using the trained generative adversarial network; Perform a difference operation between the defect image and its corresponding background texture image to obtain a defect foreground image; The defect foreground image is fused with the background texture image generated by the generative adversarial network to obtain a fused defect image.

2. According to the method for expanding the industrial product surface defect dataset based on background modeling according to claim 1, It is characterized in that The background texture information of the defect image is extracted using the background modeling method to obtain the background texture image of the defect image. The formula is: BK (i,j) =θ T M(i,j) Among them, BK is the background texture image of the defect image, θ T is the transpose of the regression coefficient matrix, M is the input defect image, and (i, j) is the pixel value coordinate of the image.

3. The method for expanding the industrial product surface defect dataset based on background modeling according to claim 2, It is characterized in that The regression coefficient matrix of the defect image is calculated using the local weighted regression algorithm.

4. The method for expanding an industrial product surface defect dataset based on background modeling according to claim 1, It is characterized in that The loss function of the generative adversarial network is: Among them, G(z) is the fake sample generated by the generator, D is the discriminator, G is the generator, x~p data is the real sample, z~p z is the input noise sample, and AE is the sample space.

5. The method for expanding an industrial product surface defect dataset based on background modeling according to claim 1, It is characterized in that If the input background texture image has a smooth surface, noise will be introduced when generating a new background texture image using the trained generative adversarial network.

6. The method for expanding an industrial product surface defect dataset based on background modeling according to claim 1, It is characterized in that Perform a differential operation on the defect image and its corresponding background texture image to obtain the defect foreground image. The formula is: FG=|BK-M Among them, FG is the defect foreground image, BK is the background texture image of the defect image, and M is the input defect image.

7. The method for expanding an industrial product surface defect dataset based on background modeling according to claim 6, It is characterized in that After obtaining the defect foreground image, the noise of the defect foreground image in the preset interval is filtered out to obtain the optimized defect foreground image. The formula is: Among them, α is the preset threshold, FG 1 The defect foreground image with noise filtered out.

8. The method for expanding an industrial product surface defect dataset based on background modeling according to claim 7, It is characterized in that Morphological operations are used to further improve the quality of defect foreground images. The formula is: Among them, FG 2 is the defect foreground image after morphological operation optimization, S is the structural element, represents the morphological erosion operation, Represents the morphological dilation operation.

9. The method for expanding an industrial product surface defect dataset based on background modeling according to claim 8, It is characterized in that The size of the structure element S is 2×4, and the expression is 10. A device for expanding industrial product surface defect dataset based on background modeling. It is characterized in that include: The background texture extraction module is used to extract the background texture information of the defect image using the background modeling method. Obtain a background texture image dataset of defect images; A background texture image generation module is used to train a generative adversarial network using a background texture image dataset of a defect image, and generate a new background texture image using the trained generative adversarial network; The defect foreground image acquisition module is used to perform a difference operation between the defect image and its corresponding background texture image to obtain the defect foreground image; The fusion module is used to fuse the defect foreground image with the background texture image generated by the generative adversarial network. Get the fused defect image.

Images