An industrial defect image generation method

Abstract

The application provides an industrial defect image generation method. The method comprises: acquiring a defect foreground image and a good product image; inputting the defect foreground image and the good product image into an industrial defect image generation model, wherein the model generates a simulated industrial defect image; the model comprises: an encoding module, a feature fusion module, a feature diffusion module and a decoding module; the encoding module is used for extracting feature maps of the defect foreground image and the good product image; the feature fusion module is used for fusing the feature maps of the defect foreground image and the good product image extracted by the encoding module to obtain fused features; the feature diffusion module is used for realizing deep fusion of the fused features and generating diffusion features; and the decoding module is used for decoding the diffusion features to output the simulated industrial defect image. The method can generate a large number of simulated industrial defect images in a short time only by using a few images of real defect samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an industrial defect image generation method. BACKGROUND

[0002] In the production scene of high yield such as consumer electronics, new energy automobile batteries, a sufficient number of high-quality training samples containing various defects are usually required for the training of a deep visual model to realize accurate detection of industrial image defects. However, in the actual production process, the number of industrial defect images is small, and the time required for manually collecting the above industrial defect images is long and the cost is high. In addition, due to the uncertainty of the above defects, as well as the characteristics of the diversification of defects and the serious imbalance of categories, the difficulty of obtaining high-quality samples is greatly increased, thereby affecting the detection accuracy of the related model, and even seriously affecting the delivery schedule of the related engineering project. Therefore, how to obtain high-quality training data (i.e. type complete and balanced training samples) in a short time is a problem for the training of a visual model. The biggest problem of high-quality training data is that the number of part defect category image samples is small, which is imbalanced, and thus it is difficult to meet the requirements of constructing a good visual detection model. The traditional acquisition method for samples with a small number of defect categories includes: (1) collecting on the production line by manual method: manually obtaining target samples of the required defect type on the production line until the number reaches the requirement, which is the most direct collection method, but this method takes too long and cannot meet the time requirements of the project delivery; (2) using image processing software (such as Photoshop) to make target type defect samples: technical personnel use software to make target type defect samples, which is high in labor cost, and the number of generated defect samples is limited, and the authenticity of the defect is not high; (3) artificially manufacturing target type defects on normal products: this method will cause irreversible damage to good products, especially for high-value products, and the cost of damage is unbearable for enterprises. Therefore, there is a strong demand for intelligent defect generation in the visual detection market.

[0003] In the case of lack of training data or insufficient number of samples of the required defect category, it is difficult for the deep visual detection model to be trained to achieve the target accuracy, therefore, it is beneficial to balance the defect features of the training set, realize the rapid iteration of the detection model, reduce the over-detection and missed detection of the model to the samples, and improve the detection performance by quickly and effectively expanding the training data or part of the training samples and increasing the diversity of the training data.

[0004] In the deep way, most of the current defect generation systems use the method based on the generative adversarial network. Generally, the processing method is to form the collected target defect samples into training samples, randomly initialize the noise input of the generator of the generative adversarial network, generate the simulation image, the discriminator of the generative adversarial network accepts the generated simulation image and the corresponding target defect sample as two inputs, then discriminates, and finally trains under the control of the adversarial loss function of the generative adversarial network; in inference, the noise input of the trained generator is randomly initialized, and the final simulation industrial defect image is generated. The defect image generation method based on the generative adversarial network has the following disadvantages: first, the mode collapse problem, that is, the generated samples tend to be the same and lack diversity; second, the flexibility is poor, and the user cannot well control the position and shape of the generated defects; third, the generated defect image cannot automatically generate corresponding labels, and manual labeling is required in the later stage, increasing the cost. In view of the above three aspects, the overall effect of this generation method is poor.

[0005] Therefore, it is necessary to improve the prior art. SUMMARY

[0006] The present application proposes a new industrial defect image generation method to solve the problem of insufficient number of high-quality training samples containing various defects for deep visual models or class imbalance.

[0007] According to a first aspect, an embodiment provides an industrial defect image generation method. The industrial defect image generation method comprises:

[0008] obtaining a defect foreground image and a good product image;

[0009] inputting the defect foreground image and the good product image into an industrial defect image generation model which has been trained, the industrial defect image generation model generating a simulation industrial defect image; wherein the simulation industrial defect image is an industrial product image containing a simulation defect;

[0010] wherein the defect foreground image refers to an image of a real defect in an industrial product image containing the real defect and / or an image of the real defect pre-stored in a defect memory bank in the industrial defect image generation model, and the good product image is obtained based on an industrial product image without the real defect corresponding to the defect foreground image;

[0011] wherein the industrial defect image generation model comprises an encoding module, a feature fusion module, a feature diffusion module and a decoding module;

[0012] The encoding module is used to extract feature maps of the defect foreground image and the good product image;

[0013] The feature fusion module is used to fuse the feature maps of the defective foreground image and the good product image extracted by the encoding module to obtain fused features;

[0014] The feature diffusion module is used to achieve deep fusion of the fused features and generate diffused features;

[0015] The decoding module is used to decode the diffusion features to output the simulated industrial defect image.

[0016] In one embodiment, the encoding module includes a first background feature extraction submodule, a second background feature extraction submodule, a first foreground feature extraction submodule, and a second foreground feature extraction submodule, and the feature fusion module includes a first feature fusion submodule and a second feature fusion submodule;

[0017] in,

[0018] The first background feature extraction submodule is used to extract features from the good product image to obtain a first background feature map;

[0019] The first foreground feature extraction submodule is used to extract features from the defective foreground image to obtain a first foreground feature map;

[0020] The first feature fusion submodule is used to fuse the first background feature map and the first foreground feature map to obtain a first fused feature map;

[0021] The second background feature extraction submodule is used to extract features from the first fused feature map to obtain a second background feature map;

[0022] The second foreground feature extraction submodule is used to extract features from the first foreground feature map to obtain a second foreground feature map;

[0023] The second feature fusion submodule performs feature fusion on the second background feature map and the second foreground feature map to obtain a second fused feature map;

[0024] The images of real defects in the industrial product images containing real defects and / or the images of real defects pre-stored in the defect memory are obtained by setting the pixels in the industrial product images containing real defects (excluding those pre-marked as the locations of the real defects) to 0, to random numbers, or to the average value of the pixels in the industrial product images. The good product images are obtained by setting the pixels in the industrial product images without real defects (corresponding to the locations of the real defects in the defect foreground images) to 0, to random numbers, or to the average value of the pixels in the industrial product images.

[0025] In an embodiment, the first background feature extraction sub-module and the first foreground feature extraction sub-module each include three standard convolution layers; wherein the standard convolution layer includes a first convolution layer, a batch standardization layer and an activation layer connected in sequence;

[0026] The second background feature extraction sub-module and the second foreground feature extraction sub-module each include a convolution layer, a maximum pooling layer and two residual modules connected in series, each of the residual modules includes two convolution sub-layers; the input feature map of the maximum pooling layer is the output feature map of the convolution layer, the output of each of the residual modules is added to the output of the upper-level residual module to serve as the input of the lower-level residual module, wherein the input feature map of the first residual module is the output feature map of the maximum pooling layer, and the input feature map of the second residual module is the feature map obtained by adding the output feature map of the first residual module to the output feature map of the maximum pooling layer.

[0027] In an embodiment, the first feature fusion sub-module performs feature fusion on the first background feature map and the first foreground feature map to obtain a first fusion feature map, including:

[0028] respectively performing spatial attention enhancement operations on the first background feature map and the first foreground feature map to obtain a first enhanced background feature map and a first enhanced foreground feature map respectively;

[0029] splicing the first enhanced background feature map and the first enhanced foreground feature map in the channel dimension to obtain a first enhanced background and foreground feature map;

[0030] performing feature extraction on the first enhanced background and foreground feature map through a second convolution layer to obtain the first fusion feature map;

[0031] wherein,

[0032] The second feature fusion sub-module performs feature fusion on the second background feature map and the second foreground feature map to obtain a second fusion feature map, including:

[0033] respectively performing spatial attention enhancement operations on the second background feature map and the second foreground feature map to obtain a second enhanced background feature map and a second enhanced foreground feature map respectively;

[0034] splicing the second enhanced background feature map and the second enhanced foreground feature map in the channel dimension to obtain a second enhanced background and foreground feature map;

[0035] performing feature extraction on the second enhanced background and foreground feature map through a third convolution layer to obtain the second fusion feature map.

[0036] In an embodiment, the performing a spatial attention enhancement operation on the first background feature map and the first foreground feature map respectively to obtain a first enhanced background feature map and a first enhanced foreground feature map comprises:

[0037] The first background feature map and the first foreground feature map are respectively taken as a feature map to be enhanced, and the following steps are performed: performing a max-pooling operation and an average-pooling operation on the input feature map to be enhanced in the channel dimension to obtain a first max-pooling feature map and a first average-pooling feature map corresponding to the feature map to be enhanced; concatenating the first max-pooling feature map and the first average-pooling feature map in the channel dimension to obtain a first concatenated feature map; performing feature extraction on the first concatenated feature map through a fourth convolutional layer to obtain a first convolutional feature map; performing a Sigmoid operation on the first convolutional feature map to obtain a first Sigmoid feature map; performing an element-wise multiplication operation on the first Sigmoid feature map and the input feature map to be enhanced to obtain an enhanced feature map; the first enhanced background feature map is the enhanced feature map obtained when the feature map to be enhanced is the first background feature map, and the first enhanced foreground feature map is the enhanced feature map obtained when the feature map to be enhanced is the first foreground feature map.

[0038] In an embodiment, the feature diffusion module comprises a fifth convolutional layer, and the decoding module comprises a plurality of decoding sub-modules, wherein the output of the decoding sub-module of the previous stage is taken as the input of the decoding sub-module of the next stage; wherein each decoding sub-module performs the following steps: performing an inverse convolution operation or an up-sampling operation on the diffusion feature map to obtain an inverse convolution feature map; performing a first convolution operation on the inverse convolution feature map and then performing a first ReLU activation operation to obtain a first result; performing a second convolution operation on the first result and then performing a second ReLU activation operation to obtain a second result; wherein the decoding module outputs the simulated industrial defect image based on the second result output by the decoding sub-module of the last stage.

[0039] In an embodiment, the method further comprises:

[0040] Before inputting the defect foreground image and the good product image into the trained industrial defect image generation model, one or more of the following operations are performed on the defect foreground image: flipping, rotating, elastic deformation, adjusting size, and adjusting position.

[0041] In an embodiment, the trained industrial defect image generation model is trained by the following steps:

[0042] Obtaining training sample images and corresponding annotation data; wherein the training sample images are the industrial product images containing real defects.

[0043] performing a 0-setting or a random number setting or a mean value setting operation on the pixels corresponding to the positions pre-labeled as real defects in the training sample image to obtain a training good image, and performing a 0-setting or a random number setting or a mean value setting operation on the pixels other than the positions pre-labeled as real defects in the training sample image to obtain a training defect foreground image;

[0044] inputting the training defect foreground image and the training good image into the industrial defect image generation model, performing feature extraction on the training good image and the training defect foreground image respectively to obtain a first background feature map and a first foreground feature map respectively, performing feature fusion on the first background feature map and the first foreground feature map to obtain a first fusion feature map, performing feature extraction on the first fusion feature map to obtain a second background feature map, performing feature extraction on the first foreground feature map to obtain a second foreground feature map, and performing feature fusion on the second background feature map and the second foreground feature map to obtain a second fusion feature map; wherein the defect memory bank in the industrial defect image generation model can automatically save the training defect foreground image and take the training defect foreground image as the image of the real defect;

[0045] adding a labeled noise map with random intensity to the second fusion feature map to obtain a predicted noise map, inputting the predicted noise map into the feature diffusion module to obtain the diffusion feature map, and outputting the simulated industrial defect image through the processing of the decoding module based on the diffusion feature map;

[0046] training the industrial defect image generation model according to a total loss function to obtain the trained industrial defect image generation model, wherein the total loss function is determined by a first loss function and a second loss function, the first loss function is determined according to the predicted noise map and the labeled data, and the second loss function is determined according to the simulated industrial defect image and the training sample image.

[0047] In an embodiment, after adding the labeled noise map with random intensity to the second fusion feature map to obtain a predicted noise map, inputting the predicted noise map into the feature diffusion module to obtain the diffusion feature map based on the predicted noise map, and obtaining the diffusion feature map through the corresponding processing of the feature diffusion module, the method comprises:

[0048] adding a labeled noise map with random intensity to the second fusion feature map to obtain a second fusion noise feature map; wherein the corresponding labeled data comprises the labeled noise map.

[0049] The feature diffusion module predicts the labeled noise map in the second fused noise feature map to obtain a predicted noise map; and the feature diffusion module outputs the diffusion feature map; wherein the diffusion feature map is obtained by subtracting the predicted noise map from the second fused noise feature map.

[0050] In an embodiment, an expression of the total loss function is:

[0051] ,

[0052] wherein an expression of the first loss function is:

[0053] ,

[0054] wherein the m represents a total number of pixel points of the labeled noise map t n or the predicted noise map p n , and the i represents an index of a pixel point of the labeled noise map t n or the predicted noise map p n .

[0055] In an embodiment, an expression of the second loss function is:

[0056] ,

[0057] wherein the L MSE represents a second content loss function, and the second content loss function is a mean square error loss function; the L t represents a second texture structure loss function; the δ 1 and δ 2 are respectively a preset weight coefficient of the second content loss function and a preset weight coefficient of the second texture structure loss function.

[0058] In an embodiment, an expression of the second content loss function is:

[0059] ,

[0060] wherein the n represents a total number of pixel points of the simulated industrial defect image y or the training sample image x , and the j represents a total number of pixel points of the simulated industrial defect imagey Or the training sample image x The index of the pixel.

[0061] In one embodiment, the expression for the second texture structure loss function is:

[0062] ,

[0063] Among them, the T ( y, x The expression for ) is:

[0064] ,

[0065] Among them, the T ( y, x ( ) is a structural similarity loss function, which is used to measure the structural similarity loss function of the simulated industrial defect image. y and training sample images x The similarity between them, the μ y and μ x The simulated industrial defect images are respectively y and training sample images x The average value of the middle pixels, the and The simulated industrial defect images are respectively y and training sample images x The variance of the middle pixel, σ yx For the simulated industrial defect image y and training sample images x The covariance of the middle pixels, the c 1 and c 2 This is the default value.

[0066] According to a second aspect, one embodiment provides a computer-readable storage medium. The storage medium includes a program executable by a processor to perform the methods described in any of the embodiments herein.

[0067] The beneficial effects of this application are:

[0068] The industrial defect image generation method of the present application comprises: acquiring a defect foreground image and a good product image; inputting the defect foreground image and the good product image into an industrial defect image generation model that has been trained, the industrial defect image generation model generating a simulated industrial defect image; that is, the industrial defect image generation method of the present application only needs a few images of real defect samples, and can generate a large number of simulated industrial defect images in a short time, and the generated simulated industrial defect images maintain high consistency and similarity with the images of real defects, thereby enabling rapid construction of a deep visual defect detection model, ultimately improving the detection performance of the deep visual defect detection model, and achieving cost reduction and efficiency improvement for production enterprises. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A module schematic diagram of the industrial defect image generation model in the industrial defect image generation method of an embodiment;

[0070] Figure 2 A flowchart schematic diagram of the industrial defect image generation method of an embodiment;

[0071] Figure 3 A module schematic diagram of the second background feature extraction sub-module or the second foreground feature extraction sub-module of an embodiment;

[0072] Figure 4 A flowchart schematic diagram of obtaining a first fused feature map of an embodiment;

[0073] Figure 5 A flowchart schematic diagram of training the industrial defect image generation model of an embodiment;

[0074] Figure 6 A schematic diagram of a real defect image and a simulated industrial defect image of an embodiment, wherein, Figure 6 The image on the left side of the middle is a real defect image (i.e. an image of a real defect in an industrial product image containing a real defect or an image of a real defect pre-stored in a defect memory bank of the industrial defect image generation model), Figure 6 The image on the right side of the middle is a simulated industrial defect image generated by the industrial defect image generation model. DETAILED DESCRIPTION

[0075] The application will be described in further detail below with specific reference to the drawings. Like elements in different embodiments are denoted by like reference numerals. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, those skilled in the art will readily recognize that many of the specific details presented herein are not in every case necessary in order to practice the present application. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the application.

[0076] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various embodiments. Meanwhile, the steps or actions in the method description can also be sequentially adjusted or changed in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment, and do not mean that the sequence is necessary, unless otherwise stated that a certain sequence must be followed.

[0077] The serial numbers of components in this paper, such as "first", "second", etc., are only used to distinguish the described objects, and do not have any sequence or technical meaning. The "connection" and "coupling" in this paper include direct and indirect connection (coupling) unless otherwise specified.

[0078] The technical purpose of the industrial defect image generation method provided by the present application is to directly generate high-fidelity simulated defect samples (i.e. simulated industrial defect images in the following) using the above industrial defect image generation method in the case of only a small amount of real defect sample data (i.e. industrial product images containing real defects in the following), and the simulated defect samples can be directly used as training samples, thereby enhancing the diversity of training samples, constructing a high-quality training sample set (the training sample set is composed of the above training samples), and finally realizing the rapid training and performance improvement of the defect detection model.

[0079] The technical concept of the industrial defect image generation method provided by the present application is that the above industrial defect image generation method is based on deep learning technology to build a network structure, and the industrial defect image generation model involved in the industrial defect image generation method adopts a "coding-decoding" architecture similar to U-Net as a whole. Please refer to Figure 1The industrial defect image generation model mainly includes an encoding module E (Encoder), a feature fusion module R, a feature diffusion module S, and a decoding module D (Decoder). The encoding module E is mainly used to extract multi-layer features of the input background image (good product image and training good product image mentioned below) and foreground image (defect foreground image and training defect foreground image mentioned below), and the feature fusion module R is used to accurately fuse the multi-layer features. The feature diffusion module S is responsible for adding artificial noise to the fused features obtained from the feature fusion module R and performing diffusion operation and corresponding training. Finally, the features output by the feature diffusion module S (i.e., diffusion feature map c ) are decoded by the decoding module D. The decoding module D is essentially used to realize image reconstruction and restore image quality. The "multi-layer features" refer to a collection of feature maps of multiple scales or resolutions. The "multi-layer features" are extracted by a neural network and are an abstract representation of the original image.

[0080] The technical solutions of the present application will be described in detail below with reference to the embodiments.

[0081] Please refer to Figure 2 , the industrial defect image generation method proposed in the present application includes:

[0082] Step S100: obtaining a defect foreground image and a good product image;

[0083] Step S200: inputting the defect foreground image and the good product image into an industrial defect image generation model that has been trained, and the industrial defect image generation model generates a simulated industrial defect image; wherein the simulated industrial defect image is an industrial product image containing a simulated defect.

[0084] The industrial defect image generation model in step S200 includes an encoding module, a feature fusion module, a feature diffusion module, and a decoding module. The encoding module is used to extract feature maps of the defect foreground image and the good product image. The feature fusion module is used to fuse the feature maps of the defect foreground image and the good product image extracted by the encoding module to obtain fused features. The feature diffusion module is used to realize deep fusion of the fused features and generate diffusion features. The decoding module is used to decode the diffusion features to output the simulated industrial defect image.

[0085] The defect foreground image in step S100 refers to the image of the real defect in the industrial product image containing the real defect and / or the image of the real defect pre-stored in the defect memory library of the industrial defect image generation model. The good product image is obtained based on the industrial product image without real defects corresponding to the defect foreground image. In some embodiments, the image of the real defect in the industrial product image containing the real defect and / or the real defect pre-stored in the defect memory library is obtained by performing a zero operation on the pixels in the industrial product image containing the real defect except the positions pre-labeled as the real defect, and the good product image is obtained by performing a zero operation on the pixels in the above industrial product image without real defects corresponding to the positions in the defect foreground image labeled as the real defect. That is, the defect foreground image refers to an industrial product image in which only the pixel values at the positions of the real defects are retained and the rest of the pixels are zeroed, and the defect foreground image serves as the foreground image; the pixels in the above industrial product image without real defects corresponding to the positions of the defect foreground image are zeroed and the rest of the pixel values are retained to obtain the good product image (i.e., the background image). The defect foreground image and the good product image are input into the industrial defect image generation model which has been trained to generate a simulated industrial defect image.

[0086] For example, please refer to Figure 1 The image of the real defect is only needed to be input into the industrial defect image generation model in the training stage. When the industrial defect image generation model performs inference, the image of the real defect is not needed to be input to obtain the defect foreground image and the good product image. When the industrial defect image generation model performs inference, there are two ways to obtain the defect foreground image: one is to obtain the defect foreground image (the training defect foreground image below) by performing a zero operation on the pixels in the training image with the real defect pre-stored by the industrial defect image generation model in the training stage except the positions of the real defect labels; the other is to obtain the defect foreground image by performing a zero operation on the pixels in the industrial product image containing the real defect input by the user except the positions pre-labeled as the real defect. When the industrial defect image generation model performs inference, there is only one way to obtain the good product image: performing a zero operation or other operations with similar effects to the zero operation on the pixels in the industrial product image without real defects input by the user corresponding to the positions in the defect foreground image labeled as the real defect to obtain the good product image.

[0087] In some embodiments, the image of the real defect in the image of the industrial product containing the real defect and / or the image of the real defect pre-stored in the defect memory bank can also be obtained by performing the operation of setting the pixels in the image of the industrial product containing the real defect to a random number or the mean value of the pixels in the above-mentioned image of the industrial product except for the positions pre-labeled as real defects. Similarly, the good product image can also be obtained by performing the operation of setting the pixels in the image of the industrial product without real defects to a random number or the mean value of the pixels in the above-mentioned image of the industrial product at the positions labeled as real defects corresponding to the defect foreground image.

[0088] It should be noted that the purpose of the above-mentioned operation of setting to 0, or the operation of setting to a random number, or the operation of setting to the mean value of the pixels is to destroy the original regularity of the pixels in the above-mentioned image corresponding to the operation, so that the original physical meaning is lost, which is beneficial to network learning.

[0089] It should be noted that the reason for performing the above-mentioned operation of setting the pixels in the corresponding defect position of the foreground image corresponding to the input of the real image without defects to 0 is: 1) the real image without defects (such as the image of the industrial product) exists in large quantities and is easy to obtain, while the image of the real defect is limited in quantity, and the present application hopes to use a small amount of real defect image to prepare a large number of simulated industrial defect images, and the simulated industrial defect image is an industrial product image containing a simulated defect, and the real image without defects acts as a background image here; 2) if the operation of setting the pixels in the corresponding defect position of the foreground image corresponding to the input of the defect foreground image to 0 is directly performed on the image of the real defect, the number of the obtained background image (i.e. the good product image) will be limited, which cannot meet the requirement of generating a large number of simulated industrial defect images.

[0090] When the industrial defect image generation model performs inference, the defect foreground image of the target type is used as the foreground image of the inference input; the operation of setting the pixels in the corresponding defect position of the defect foreground image corresponding to the input to 0 or the above-mentioned other corresponding operation is performed on the real image without defects to obtain the background image of the inference input. The industrial defect image generation model that has been trained well obtains two kinds of image (i.e. the defect foreground image and the good product image) input and directly generates high-quality simulated industrial defect images in an end-to-end manner.

[0091] In some embodiments, in step S200, i.e., when the industrial defect image generation model is performing inference, the user can also select the foreground image of the real defect image as the defect foreground image, and take corresponding operations on the foreground of the defect foreground image to control the real defect, and the corresponding operations include one or more of flipping, rotating, elastic deformation, resizing, adjusting position, etc. When the industrial defect image generation model is performing inference, if the user does not take any operation to control the real defect image (i.e., the real defect image in the industrial product image containing the real defect and / or the real defect image pre-stored in the defect memory library in the industrial defect image generation model), only the real defect-free industrial product image needs to be input, and the industrial defect image generation model will automatically perform all the required operations on the real defect-free industrial product image to facilitate the user's use; if the user selectively controls one or more of the shape, position, and size of the real defect, the industrial defect image generation model can perform inference according to the user's specific input.

[0092] It should be noted that a person skilled in the art can set all the required operations according to actual needs, for example, the required operations can include preset and random operations (such as rotation, flipping, and shifting operations). For each real defect, the industrial defect image generation model can randomly select one or more operations for processing.

[0093] Therefore, after the industrial defect image generation model is trained, the industrial defect image generation model can directly generate high-quality simulated industrial defect images in an end-to-end manner, so as to achieve the purpose of effectively and quickly expanding the target type samples.

[0094] Please refer to Figure 1 The encoding module of the industrial defect image generation model includes a first background feature extraction submodule P 1, a second background feature extraction submodule P 2, a first foreground feature extraction submodule Q 1, and a second foreground feature extraction submodule Q 2, a feature fusion module including a first feature fusion submodule R 1 and a second feature fusion submodule R 2.

[0095] The first background feature extraction submodule P 1 is used for feature extraction on the good product image x 1 to obtain a first background feature map b 1;

[0096] The first foreground feature extraction submodule Q 1 is used for feature extraction on the defect foreground imagex 2perform feature extraction to obtain a first foreground feature map f 1;

[0097] a first feature fusion sub-module R 1for performing feature fusion on the first background feature map b 1and the first foreground feature map f 1to obtain a first fused feature map b 11 ;

[0098] a second background feature extraction sub-module P 2for performing feature extraction on the first fused feature map b 11 to obtain a second background feature map b 2;

[0099] a second foreground feature extraction sub-module Q 2for performing feature extraction on the first foreground feature map f 1to obtain a second foreground feature map f 2;

[0100] a second feature fusion sub-module R 2for performing feature fusion on the second background feature map b 2and the second foreground feature map f 2to obtain a second fused feature map b 22 .

[0101] the first background feature map b 1and the first foreground feature map f 1are used to abstractly represent low-level features in the good product image x 1and the defect foreground image x 2.

[0102] the first fused feature map b 11 is used to abstractly represent features after low-level feature fusion of the good product image x 1and the defect foreground image x 2.

[0103] the first background feature extraction sub-module P 1and the first foreground feature extraction sub-module Q 1have the same network structure but do not share parameters therebetween. The second background feature extraction sub-module P 2and the second foreground feature extraction sub-module Q 2have the same network structure but do not share parameters therebetween. Wherein, P 1or QThe network structure of 1 is shown in Table 1 below:

[0104]

[0105] In Table 1 above, Conv+BN+ReLU represents a combination of Conv (convolutional layer), BN (batch normalization layer), and ReLU (ReLU activation function layer). The convolutional kernel of Conv (convolutional layer) can be any kernel, thus Conv+BN+ReLU in Table 1 can be considered as a 3x3 standard convolutional layer. Therefore, the first background feature extraction submodule... P 1 or the first foreground feature extraction submodule Q The network structure of 1 mainly consists of three 3x3 standard convolutional layers.

[0106] In some embodiments, the stride of the first and third standard convolutional layers can be 2, and the stride of the second standard convolutional layer can be 1. Since image generation is a pixel-intensive task, pooling operations are not used to reduce the feature map resolution; instead, convolutional layers with a stride of 2 are used, which helps to maintain the positional features of the features.

[0107] It should be noted that the first background feature extraction submodule P 1 or the first foreground feature extraction submodule Q The stride of the second standard convolutional layer, set to 1, serves as a transition for features. Using convolution to reduce the resolution of the feature map better preserves the corresponding positional information than using pooling operations. Furthermore, the stride of the three 3x3 standard convolutional layers mentioned above is merely an example; other stride values ​​are possible.

[0108] In some embodiments, compared to a good product image x 1 and defect foreground image x Input resolution of 2, first background feature map b 1 and the first foreground feature map f A resolution of 1 can be used to distinguish between good product images. x 1 and defect foreground image x 2 is 1 / 4 of the input resolution.

[0109] It should be noted that those skilled in the art can set the first background feature map according to actual needs. b 1 and the first foreground feature map f A resolution of 1, i.e., the first background feature map. b 1 and the first foreground feature map f A resolution of 1 does not necessarily indicate a good image. x 1 and defect foreground image x 2 is 1 / 4 of the input resolution.

[0110] The first background feature extraction submodule of the industrial defect image generation model P 1 and the first foreground feature extraction submodule Q 1 each include three standard convolution layers; wherein the standard convolution layer includes a first convolution layer, a batch standardization layer and an activation layer connected in sequence.

[0111] In some embodiments, referring to Figure 3 , the second background feature extraction submodule and the second foreground feature extraction submodule each include a convolution layer 101, a maximum pooling layer 102 and two residual modules 103 connected in series, each residual module 103 including two convolution sub-layers; the input feature map of the maximum pooling layer is the output feature map of the convolution layer, and the output of each residual module 103 is added to the output of the upper-level residual module 103 to serve as the input of the lower-level residual module 103, wherein the input feature map of the first residual module 103 is the output feature map of the maximum pooling layer, and the input feature map of the second residual module 103 is the feature map obtained by adding the output feature map of the first residual module 103 to the output feature map of the maximum pooling layer.

[0112] It should be noted that the above-mentioned second background feature extraction submodule P 2 and the second foreground feature extraction submodule Q 2 are example structures adopted by the present application, and the above-mentioned second background feature extraction submodule P 2 and the second foreground feature extraction submodule Q 2 can also adopt other feature extractors conforming to the principle requirements, for example, common Res2Net and HRNet feature extractors.

[0113] In some embodiments, the person skilled in the art can also select the specific structural parameters (such as the stride P output channel number, etc.) of the convolution layer and the maximum pooling layer in the second background feature extraction submodule Q 2 and the second foreground feature extraction submodule 、 2 according to the actual application scene. For example, the size of the convolution kernel of the convolution layer of the second background feature extraction submodule P 2 or the second foreground feature extraction submodule Q 2 can be 7x7 ， , the stride can be 2, and the output channel number can be 64. The pooling window size of the maximum pooling layer of the second background feature extraction submodule P 2 or the second foreground feature extraction submodule Q 2 can be 3x3 ，The step size is 2. The input feature map of the first residual module 103 is the output feature map of the max pooling layer. The input feature map of the second residual module 103 is the feature map obtained by adding the output feature map of the first residual module 103 to the output feature map of the max pooling layer. The output of each residual module 103 after the first residual module 103 is added to the output of the previous residual module 103 as the input of the next residual module 103. The output feature map of the second residual module 103 is added to the output feature map of the first residual module 103 as the second background feature extraction submodule. P 2 or second foreground feature extraction submodule Q 2. Output. Each residual module 103 includes two convolutional sub-layers, where both convolutional sub-layers can be 3×3 convolutional layers (i.e., the kernel size in both convolutional sub-layers is 3×3), and the number of output channels in the two convolutional sub-layers in the residual module 103 is the same. That is, the feature map input to the residual module 103 first undergoes convolution processing by the previous convolutional sub-layer in the residual module 103; the feature result obtained from the convolution processing of the previous convolutional sub-layer is input to the next convolutional sub-layer in the residual module 103 for further convolution processing; the output result of the next convolutional sub-layer in the residual module 103 is used as the output of the residual module 103.

[0114] It should be noted that the function of the feature fusion module is to accurately fuse low-level and high-level features from the good product image and the defective foreground image. Low-level features contain relatively good positional information, while high-level features contain relatively good semantic information. The first feature fusion submodule... R 1 and second feature fusion submodule R The network structure of 2 is consistent, the only difference being the number of input and output channels, and the first feature fusion submodule. R 1 and second feature fusion submodule R The parameters for 2 are not shared. First Feature Fusion Submodule R 1. Output features (i.e., the first fused feature map) b 11 The first feature is a low-level feature, and the second feature fusion submodule is a low-level feature. R 2. Output features (i.e., the second fused feature map) b 22 Low-level features are high-level features. In neural networks, low-level features generally have higher resolution and fewer input and output channels corresponding to them; high-level features have lower resolution and more input and output channels corresponding to them.

[0115] It should be noted that the first feature fusion submodule R 1 and second feature fusion submodule R The function of 2 is to extract images of good products. x1 and defect foreground image x Key feature information in 2 (e.g., images that represent good products) x 1 and defect foreground image x The first feature fusion submodule (which extracts the abstract features of feature 2) suppresses other irrelevant information. R 1 and second feature fusion submodule R Both methods employ spatial attention enhancement to amplify relevant features and filter out irrelevant information. Then, they are combined and a 1x1 convolutional layer is used to extract comprehensive information (i.e., the extracted good-quality image). x 1 and defect foreground image x (All feature information after fusing certain high-dimensional / key information of 2), ultimately achieving precise feature fusion.

[0116] Please refer to Figure 4 First feature fusion submodule R 1 pair of first background feature maps b 1 and the first foreground feature map f 1. Perform feature fusion to obtain the first fused feature map. b 11 ,include:

[0117] Step S11: Process the first background feature map respectively b 1 and the first foreground feature map f 1. Perform spatial attention enhancement operations to obtain a first enhanced background feature map and a first enhanced foreground feature map, respectively;

[0118] Step S12: The first enhanced background feature map and the first enhanced foreground feature map are concatenated along the channel dimension to obtain the first enhanced background and foreground feature map;

[0119] Step S13: Extract features from the first enhanced background and foreground feature maps using a second convolutional layer to obtain the first fused feature map. b 11 .

[0120] The kernel size of the second convolutional layer is 1x1. It should be noted that the spatial attention enhancement operation (SA) can employ existing techniques in this field.

[0121] Second Feature Fusion Submodule R 2 pairs of second background feature maps b 2. Second foreground feature map f 2. Perform feature fusion to obtain a second fused feature map. b 22 ,include:

[0122] The second background feature mapb 2 and the second foreground feature map f 2 performs a spatial attention enhancement operation to obtain a second enhanced background feature map and a second enhanced foreground feature map, respectively;

[0123] The second enhanced background feature map and the second enhanced foreground feature map are spliced in the channel dimension to obtain a second enhanced background and foreground feature map;

[0124] A third convolutional layer is used to extract features from the second enhanced background and foreground feature map to obtain a second fusion feature map b 22 .

[0125] The third convolutional layer has a kernel size of 1x1.

[0126] In an embodiment of the present application, another method of spatial attention enhancement operation is provided. In this embodiment, the above-mentioned spatial attention enhancement operation is performed on the first background feature map b 1 and the first foreground feature map f 1 to obtain a first enhanced background feature map and a first enhanced foreground feature map, respectively, including:

[0127] The first background feature map b 1 and the first foreground feature map f 1 are taken as the input feature maps to perform the following steps: a maximum pooling operation Max and a mean pooling operation Mean are performed on the input feature maps in the channel dimension to obtain a first maximum pooled feature map and a first mean pooled feature map corresponding to the input feature maps, respectively; the first maximum pooled feature map and the first mean pooled feature map are spliced in the channel dimension to obtain a first spliced feature map; a fourth convolutional layer is used to extract features from the first spliced feature map to obtain a first convolutional feature map; a Sigmoid operation is performed on the first convolutional feature map to obtain a first Sigmoid feature map; the first Sigmoid feature map is multiplied element-wise with the input feature map to obtain an enhanced feature map; the first enhanced background feature map is the enhanced feature map obtained when the input feature map is the first background feature map b 1, and the first enhanced foreground feature map is the enhanced feature map obtained when the input feature map is the first foreground feature map f 1.

[0128] It should be noted that after the maximum pooling operation is performed on the input in the channel dimension, the number of channels is reduced to 1. After the average pooling operation is performed on the input in the channel dimension, the number of channels is also reduced to 1. Then, the first maximum-pooled feature map and the first average-pooled feature map with the number of channels reduced to 1 are spliced in the channel dimension, and the first spliced feature map with the number of channels being 2 is output.

[0129] wherein the convolution kernel size of the fourth convolution layer is 1x1. After the first spliced feature map is extracted by the fourth convolution layer, the number of channels is reduced to 1.

[0130] It should be noted that the first feature fusion sub-module R 1 and the second feature fusion sub-module R 2 have the same structure and processing flow. Therefore, in order to avoid repetition, the structure and processing flow of the first feature fusion sub-module R 1 are described as an example, and the structure and processing flow of the second feature fusion sub-module R 2 will not be described again. That is, the spatial attention enhancement operation on the second background feature map b 2 and the second foreground feature map f 2 will not be described again. b f b f 1 will not be described again.

[0131] It should be noted that the main function of the feature diffusion module S is to realize further fusion within the feature (i.e., the second fusion feature map b 22 ), so that the background feature and the foreground feature in the second fusion feature map b 22 can be more naturally and faithfully fused in texture and details.

[0132] ​​​The feature diffusion module of the industrial defect image generation model comprises a fifth convolutional layer. The decoding module comprises a plurality of decoding sub-modules, wherein the output of a preceding decoding sub-module is taken as the input of a subsequent decoding sub-module; wherein each decoding sub-module performs the following steps: performing an inverse convolution operation or an up-sampling operation on the diffusion feature map to obtain an inverse convolution feature map; performing a first convolution operation on the inverse convolution feature map and then performing a first ReLU activation operation to obtain a first result; performing a second convolution operation on the first result and then performing a second ReLU activation operation to obtain a second result. The convolution kernel size of the fifth convolutional layer is 1x1. The second result output by the last decoding sub-module is the simulated industrial defect image required, without the need for other post-processing measures on the second result output by the last decoding sub-module. It should be noted that the specific manner of "outputting the simulated industrial defect image based on the diffusion feature map through the processing of the decoding module" described above can refer to the structure and execution steps of the decoding module (such as the steps performed by the decoding sub-module described above). That is, the second result output by the last decoding sub-module is the simulated industrial defect image.

[0133] It should be noted that the decoding module D adopts the decoding module of the classic U-Net network structure, and the decoding module D is consistent with the decoding module of the U-Net network in structure, but the input is only the diffusion feature map c , and the intermediate layer feature short connection input of the decoding module in the original U-Net network is removed, and the output result of the decoding module D is y . When the industrial defect image generation model is in the training stage, the above y and x complete training under the control of the corresponding loss function L D ; when the industrial defect image generation model is in the inference stage, the decoding module D directly generates the simulated industrial defect image.

[0134] It should be noted that the structure of the decoding module is only exemplary, and the decoding module can also adopt the decoding module of the common DeepLab or other decoding modules that meet the principle requirements.

[0135] The diffusion feature map c is used for abstract representation of the fusion feature (i.e., the second fusion feature map b 22 ) of the good product image x 1 and the defect foreground image x 2.

[0136] In some embodiments, the industrial defect image generation method of the present application further comprises: before inputting the defect foreground image and the good product image into the industrial defect image generation model that has been trained, performing one or more of the following operations on the defect foreground image: flipping, rotating, elastic deformation, adjusting size, and adjusting position. Adjusting size refers to adjusting the size of the defect foreground in the defect foreground image. Adjusting position refers to adjusting the position of the defect foreground in the defect foreground image.

[0137] It should be noted that the user can freely decide whether to perform the above one or more of the operations on the defect foreground image according to actual needs.

[0138] Please refer to Figure 5 The industrial defect image generation model that has been trained in the above step S200 is obtained by the following steps:

[0139] Step S21: obtaining training sample images and corresponding annotation data; wherein the training sample images are industrial product images containing real defects;

[0140] Step S22: performing a set-to-zero operation or a set-to-random number operation or a set-to-mean operation on the pixels corresponding to the positions pre-annotated as real defects in the training sample images to obtain a good product image for training, and performing a set-to-zero operation or a set-to-random number operation or a set-to-mean operation on the pixels other than the positions annotated as real defects in the training sample images to obtain a defect foreground image for training;

[0141] Step S23: inputting the defect foreground image for training and the good product image for training into the industrial defect image generation model, and performing feature extraction on the good product image for training x 1 and the defect foreground image for training x 2 to obtain a first background feature map b 1 and a first foreground feature map f 1 respectively; performing feature fusion on the first background feature map b 1 and the first foreground feature map f 1 to obtain a first fusion feature map b 11 ; performing feature extraction on the first fusion feature map b 11 to obtain a second background feature map b 2; performing feature extraction on the first foreground feature map f 1 to obtain a second foreground feature map f 2; performing feature fusion on the second background feature map b 2 and the second foreground feature map f 2 to obtain a second fusion feature mapb 22 ; wherein the defect memory in the industrial defect image generation model can automatically save the defect foreground image for training x 2, and the defect foreground image for training x 2 as an image of a real defect;

[0142] Step S24: adding a labeled noise map with random intensity in the second fused feature map b 22 t n and inputting the feature diffusion module to obtain a predicted noise map; based on the predicted noise map, the diffusion feature map is obtained through the corresponding processing of the feature diffusion module c ; based on the diffusion feature map c , the simulation industrial defect image is output through the processing of the decoding module D;

[0143] Step S25: training the industrial defect image generation model according to the total loss function to obtain the trained industrial defect image generation model; wherein the total loss function is determined by the first loss function L S and the second loss function L D , the first loss function L S is determined according to the predicted noise map and the labeled data, and the second loss function L D is determined according to the simulation industrial defect image and the training sample image.

[0144] It should be noted that the above-mentioned fifth convolutional layer in the network structure of the feature diffusion module is a convolutional layer with a convolutional kernel of 1x1. In the second fused feature map b 22 , a labeled noise map with random intensity is added t n to obtain the above-mentioned second fused noise feature map. The above-mentioned fifth convolutional layer of the feature diffusion module predicts the above-mentioned second fused noise feature map, and further obtains the above-mentioned predicted noise map.

[0145] For example, please refer to Figure 1 , x , the image of the real defect only needs to be input into the industrial defect image generation model in the training stage. When the industrial defect image generation model is trained, the image of the real defect with segmentation annotation x can be executed foreground separation operation. That is, the image x ​Pixels marked as real defect positions are subjected to a 0 operation or a random number operation or an operation of setting to the mean value of the pixels in the industrial product image, and a good product image for training is obtained x 1; at the same time, the image x Pixels other than those marked as real defect positions are subjected to a 0 operation or a random number operation or an operation of setting to the mean value of the pixels in the industrial product image, and a defect foreground image for training is obtained x 2.

[0146] When the industrial defect image generation model is trained, the good product image for training x 1 and the defect foreground image for training x 2 are used as common inputs of the industrial defect image generation model. The encoding module E of the industrial defect image generation model extracts multi-layer features from the good product image for training x 1 and the defect foreground image for training x 2, and performs corresponding fusion operations by the feature fusion module to obtain fused features. The feature diffusion module S is responsible for adding artificial noise to the final fused features and performing diffusion operations, and completes training under the control of a corresponding loss function. Finally, the obtained features are reconstructed by the decoding module D to restore image quality, and the decoding module D generates simulated industrial defect images. y The corresponding target is x .

[0147] In essence, the purpose of training the industrial defect image generation model is to enable the industrial defect image generation model to learn to fuse the features of the background image and the foreground image described above, so as to achieve high-quality simulation generation of simulated industrial defect images.

[0148] In some embodiments, during the training of the industrial defect image generation model, the defect memory bank in the industrial defect image generation model saves the defect foreground image for training of the training sample image, and then when the industrial defect image generation model is trained and inferences, the user can not need to import the image of the real defect as the defect foreground image for inference, and the industrial defect image generation model can automatically select the defect foreground image for training from the defect memory bank as the defect foreground image for inference.

[0149] It should be noted that when the industrial defect image generation model is trained, the learning goal of the industrial defect image generation model is the input real sample defect image (i.e. the defect foreground image used for training described above). That is, essentially, it is hoped that the industrial defect image generation model learns the fusion of the above-mentioned good product image used for training and the above-mentioned defect foreground image in high-dimensional features, and then makes the industrial defect image generation model learn the sufficient fusion of the defect foreground image and the good product image, so that the industrial defect image generation model can finally generate a simulated industrial defect image very close to the above-mentioned input real sample defect image, and the generated simulated industrial defect image is more faithful and natural, and then completes the simulation generation task of the defect image.

[0150] The above-mentioned second fusion feature map b 22 The label noise map with random intensity is added in the second fusion feature map t n The feature diffusion module S is used for inputting the second fusion feature map to obtain a predicted noise map; and a diffusion feature map is obtained based on the predicted noise map c , comprising:

[0151] The label noise map with random intensity is added in the second fusion feature map b 22 The label noise map with random intensity is added in the second fusion feature map t n to obtain a second fusion noise feature map; wherein the corresponding label data comprises a label noise map t n ;

[0152] The feature diffusion module S predicts the label noise map in the second fusion noise feature map t n to obtain a predicted noise map p n ; the feature diffusion module outputs a diffusion feature map c ; wherein the diffusion feature map c is obtained by subtracting the predicted noise map from the second fusion noise feature map p n .

[0153] In some embodiments, the noise type of the label noise map t n is Gaussian noise. The noise type of the label noise map t n is not limited here, and a person skilled in the art can select the noise type of the label noise map t n according to actual needs.

[0154] It should be noted that introducing a feature diffusion module to predict noise is a common practice for feature diffusion module S. The mechanism of the feature diffusion module is as follows: by adding noise perturbation, the prediction results of the industrial defect image generation model change significantly. The industrial defect image generation model predicts noise, then removes the noise. By repeating this training process, the industrial defect image generation model learns to ignore noise, becomes less sensitive to unimportant and non-critical information in the data, and learns to synthesize and summarize key information, making the industrial defect image generation model more robust and with stronger generalization ability. Specifically, the purpose of introducing the feature diffusion module is to obtain a diffusion feature map. c This allows for a better understanding of the diffusion feature map. c As a good product image x 1 and defect foreground image x 2. Fusion features (i.e., the second fusion feature map) b 22 The abstract representation of the image is used to improve the image generation quality of the final simulated industrial defect image. Adding noise of random intensity aims to train the industrial defect image generation model to learn to summarize and represent key information under various noise conditions, unaffected by noise, thus improving the diffusion characteristics. c This results in a more robust and generalizable industrial defect image generation model.

[0155] This application designs a corresponding combined loss function for an industrial defect image generation model to complete the training of the model. Specifically, the expression for the overall loss function is as follows:

[0156] ,

[0157] Wherein, the first loss function L S The loss function is expressed in the form of mean squared error. L S The expression is:

[0158] ,

[0159] in, m Noise map annotation t n Or predict the noise map p n The total number of pixels, i Noise map annotation t n Or predict noise map p n The index of the pixel.

[0160] It should be noted that the first loss function L S for calculating the annotated noise map t n and the predicted noise map p n is the overall average of the square of the difference between the corresponding pixels. The first loss function L S can make the feature diffusion module S effectively predict the noise, make the industrial defect image generation model pay more attention to the key part of the feature to realize the sufficient fusion of the feature, and be beneficial to improve the quality of the finally generated simulated industrial defect image, so that the generated simulated industrial defect image is more natural and faithful.

[0161] The expression of the above-mentioned second loss function L D is:

[0162] ,

[0163] wherein, L MSE indicates the second content loss function, and the second content loss function is a mean square error loss function. L t indicates the second texture structure loss function; δ1 and δ2 are respectively a preset weight coefficient of the second content loss function and a preset weight coefficient of the second texture structure loss function.

[0164] In some embodiments, δ1 = 0.7 and δ2 = 0.3.

[0165] It should be noted that the specific parameters of the above-mentioned δ1 and δ2 are only example parameters. Those skilled in the art can adjust the specific parameters of the above-mentioned δ1 and δ2.

[0166] The expression of the above-mentioned second content loss function is:

[0167] ,

[0168] wherein, n indicates the total number of pixel points of the simulated industrial defect image y or the training sample image x , j indicates the index of the pixel point of the simulated industrial defect image y or the training sample image x .

[0169] It should be noted that the main role of the second content loss function L MSE is to make the pixel value of the generated simulated industrial defect image y more close to the pixel value of the target image (such as the training sample imagex ) as close as possible. That is, the second content loss function L MSE for calculating the generated simulation industrial defect image y and the image corresponding to the label x , and thus the quality of the generated simulation industrial defect image can be effectively controlled.

[0170] The expression of the above-mentioned second texture structure loss function is:

[0171] ,

[0172] wherein, T ( y , x ) is the expression of:

[0173] ,

[0174] wherein, T ( y , x ) is a structural similarity loss function, the structural similarity loss function is used to measure the similarity between the simulation industrial defect image y and the training sample image x , μ y and μ x are the mean values of the pixels in the simulation industrial defect image y and the training sample image x , and are the variances of the pixels in the simulation industrial defect image y and the training sample image x , and σ yx is the covariance of the pixels in the simulation industrial defect image y and the training sample image x , c 1 and c 2 are preset values.

[0175] It should be noted that the above-mentioned c 1 and c 2 are two small constants, and the purpose of setting the above-mentioned c 1 and c 2 is to avoid the above-mentioned T ( y , xThe second texture structure loss function can measure the similarity of the reconstructed image (i.e., the simulated industrial defect image) and the target image (i.e., the label image corresponding to the simulated industrial defect image) from the aspects of brightness, contrast and structure, thereby effectively improving the quality of image reconstruction.

[0176] Please refer to Figure 6 , Figure 6 The image on the left side in FIG. 1 is a real defect image (i.e., an image of a real defect in an industrial product image containing a real defect or an image of a real defect pre-stored in a defect memory of an industrial defect image generation model), Figure 6 The image on the right side in FIG. 1 is a simulated industrial defect image generated by the industrial defect image generation model.

[0177] The industrial defect image generation method provided in the present application designs a corresponding combined loss function (i.e., the overall loss function described above) for the industrial defect image generation model, wherein the feature diffusion module and the decoding module of the industrial defect image generation model are respectively designed with corresponding loss functions (i.e., the first loss function and the second loss function) to train the industrial defect image generation model. The industrial defect image generation method provided in the present application can realize high-quality generation of target type defect images (i.e., simulated industrial defect images) based on the design idea of data fusion, wherein the combined design of the encoding module, the feature diffusion module and the feature fusion module is the key to the entire model structure. In addition, the industrial defect image generation model in the industrial defect image generation method can complete training on a large-scale industrial image. When the user uses the industrial defect image generation model, the corresponding pre-training weight (wherein the pre-training weight only contains the weight parameter of the industrial defect image generation model, and the pre-training weight does not include the defect memory in the pre-training stage; when the user specifically trains (fine-tunes) the industrial defect image generation model, the industrial defect image generation model will automatically save the related defect memory) can be loaded, and then only a small amount of real defect samples (such as industrial product images containing real defects) are needed to fine-tune the industrial defect image generation model, and the industrial defect image generation model can be used for inference, thereby generating target type high-quality sample data (i.e., simulated industrial defect images).

[0178] It can be seen that the present application proposes a new industrial defect image generation method for the problem of insufficient number of high-quality training samples containing various defects or class imbalance for training of a deep visual model. The industrial defect image generation method only needs a few images of real defect samples, and can generate a large number of simulated industrial defect images that meet user needs (such as defects with specific settings such as different positions and shapes) in a short time, and the generated simulated industrial defect images maintain high consistency and similarity with the images of real defects. That is, the generated simulated industrial defect images can be directly used as training samples, achieving rapid expansion of target training data, enhancing the diversity of training samples, and achieving rapid construction of a deep visual defect detection model, ultimately improving the fitting ability, generalization performance and detection performance of the deep visual defect detection model, and achieving cost reduction and efficiency improvement of production enterprises.

[0179] It can be seen that, compared with the generation method based on the generative adversarial network, the purpose of the industrial defect image generation method provided by the present application is to quickly generate high-quality defect samples of the target type required by the user. The advantages of the industrial defect image generation method provided by the present application mainly include the following aspects:

[0180] (1) The industrial defect image generation method provided by the present application requires a small number of real defect samples, and is therefore very suitable for scenarios where target type defect samples are not easy to collect, and saves labor costs;

[0181] (2) Easy to use and easy to control: the user can flexibly control the type, position and size of the defect, and support related combinations of defect types, making it easy for users to generate complex composite defect samples;

[0182] (3) The generated simulated industrial defect images have annotations, and the generation efficiency is high: the generated defect samples are one-to-one corresponding to the annotations, and the annotations are automatically generated, so the user does not need to perform tedious pixel-level annotation operations, greatly improving the work efficiency. The industrial defect image generation method provided by the present application adopts a novel solution for the problem of insufficient and unbalanced training data in the industry, generates simulated defect samples with high authenticity, can quickly and directly expand target type data, and can be directly used as training samples to improve the detection performance of the defect detection model.

[0183] It should be noted that first, the "label noise map" of the feature diffusion module is irrelevant to the description "the generated defect sample is one-to-one corresponding to the label, and the label is automatically generated" in advantage (3) here. That is, the "label noise map" of the feature diffusion module only plays a role in the feature diffusion module; second, the description "the generated defect sample is one-to-one corresponding to the label, and the label is automatically generated" in advantage (3) here means that the position, size, type, etc. of the defect in the defect foreground image are determined before the industrial defect image generation model is input into the trained industrial defect image generation model. This also means that the generated defect sample (i.e. the simulated industrial defect image) has a corresponding defect position, size and type, so "the generated defect sample is one-to-one corresponding to the label, and the label is automatically generated".

[0184] It can be seen that, compared with the prior art, the industrial defect image generation method provided by the present application has the following advantages:

[0185] (1) A small amount of real defect samples are used to generate defect samples, that is, only a few real defect samples are needed, that is, not only a large number of required defect samples (i.e. simulated industrial defect images) can be generated in a short time, but also the diversity of training data can be enhanced, which is highly synergistic with the deep visual detection model, and the detection capability of the related defect detection model is improved;

[0186] (2) High fidelity of the simulated defect sample (i.e. the simulated industrial defect image). The present application combines the coding module, the feature diffusion module and the feature fusion module, so that the simulated defect (i.e. the simulated industrial defect image) generated by the industrial defect image generation model can maintain high consistency and similarity with the defect quality of the real defect image, and the generated defect sample can be directly used as the training sample of the deep visual detection model;

[0187] (3) Strong applicability, and the application scenarios of the technology are wide, that is, the industrial defect image generation method provided by the present application can be used for different products and different types of defects, and the application scenarios are wide. The user can flexibly and conveniently generate defect samples of target types of different products according to their own needs.

[0188] The above is some description of the industrial defect image generation method. Some embodiments of the present application also disclose a computer readable storage medium. The storage medium includes a program that can be executed by a processor to implement the method of any embodiment described herein.

[0189] Various exemplary embodiments are described herein. However, it will be recognized by those skilled in the art that changes and modifications can be made to the exemplary embodiments without departing from the scope of the present disclosure. For example, various operational steps and components for carrying out the operational steps can be implemented in different sequences and / or omitted, combined, or combined in various ways than presented in the figures and / or descriptions without departing from the principles of the present disclosure.

[0190] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. In addition, as understood by those skilled in the art, the principles herein can be reflected in a computer program product on a computer readable storage medium preloaded with computer readable program code. Any tangible, non-transitory computer readable storage medium can be used, including magnetic storage devices (hard disk, floppy disk, etc.), optical storage devices (CD-ROM, DVD, Blu Ray disk, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general purpose computer, a special purpose computer, or other programmable data processing apparatus to form a machine, so that these instructions executed on the computer or other programmable data processing apparatus can generate a device that implements the specified functions. These computer program instructions can also be stored in a computer readable storage medium, which can instruct the computer or other programmable data processing apparatus to operate in a specific way, so that the instructions stored in the computer readable storage medium can form a manufactured item, including an implementation device that implements the specified functions. Computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so as to execute a series of operational steps on the computer or other programmable data processing apparatus to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus can provide steps for implementing the specified functions.

[0191] Although the principles herein have been illustrated in various embodiments, many modifications in structure, arrangement, proportions, elements, materials, and components specially adapted to specific environments and operational requirements can be used without departing from the principles and scope of the present disclosure. The above modifications and other changes or modifications will be included within the scope of the present disclosure.

[0192] The foregoing detailed description has been presented for purposes of illustration and description. However, various modifications and changes are possible in the implementation of the disclosure. Accordingly, the disclosure is intended to embrace all modifications and alterations within the scope and spirit of the disclosure. Thus, the scope of the disclosure is not intended to be limited to the particular form set forth herein, but includes all features that might be provided within the scope and spirit of the disclosure. Likewise, a variety of advantages and features have been set forth in the description herein with reference to various embodiments. It is to be understood that not necessarily all advantages can be achieved in accordance with any particular embodiment. Further, solutions to problems can have been set forth in one or more embodiments; however, it is to be understood that not necessarily all such solutions are within the scope of the disclosure. Fruits of the benefits, solutions to problems, and any elements that might contribute to the acuteness of the benefits as a whole or any other stated characteristic, are not to be construed as required, essential, or critical. The terms "comprises," "comprising," "includes," "including," and the like as used herein are specifically intended to be construed as inclusive, rather than exclusive. That is, it is intended that the process, method, article, or apparatus include the listed elements but not exclude other elements that are not expressly listed or inherent to such process, method, system, article, or apparatus. Also, the use of the terms "coupled" and "coupling," and the like, as used herein, are intended to mean a physical connection, an electrical connection, a magnetic connection, an optical connection, a communicative connection, a functional connection, and / or any other connection.

[0193] Those skilled in the art will recognize that many modifications might be made to the details of the above-described embodiments without departing from the underlying principles of the present application. The scope of the present application should, therefore, be determined only for the appended claims.

Claims

1. A method for generating industrial defect images, characterized in that, include: Obtain foreground images of defects and images of good products; The defect foreground image and the good product image are input into a pre-trained industrial defect image generation model, which generates a simulated industrial defect image; wherein, the simulated industrial defect image is an image of an industrial product containing simulated defects; Wherein, the defect foreground image refers to the image of the real defect in the industrial product image containing the real defect and / or the image of the real defect pre-stored in the defect memory bank in the industrial defect image generation model, and the good product image is obtained based on the industrial product image without real defects corresponding to the defect foreground image; The industrial defect image generation model includes: an encoding module, a feature fusion module, a feature diffusion module, and a decoding module; The encoding module is used to extract the feature map of the defective foreground image and the feature map of the good product image; The feature fusion module is used to fuse the feature maps of the defective foreground image and the good product image extracted by the encoding module to obtain fused features; The feature diffusion module is used to achieve deep fusion of the fused features and generate diffused features; The decoding module is used to decode the diffusion features to output the simulated industrial defect image.

2. The industrial defect image generation method as described in claim 1, characterized in that, The encoding module includes a first background feature extraction submodule, a second background feature extraction submodule, a first foreground feature extraction submodule, and a second foreground feature extraction submodule; the feature fusion module includes a first feature fusion submodule and a second feature fusion submodule. in, The first background feature extraction submodule is used to extract features from the good product image to obtain a first background feature map; The first foreground feature extraction submodule is used to extract features from the defective foreground image to obtain a first foreground feature map; The first feature fusion submodule is used to fuse the first background feature map and the first foreground feature map to obtain a first fused feature map; The second background feature extraction submodule is used to extract features from the first fused feature map to obtain a second background feature map; The second foreground feature extraction submodule is used to extract features from the first foreground feature map to obtain a second foreground feature map; The second feature fusion submodule performs feature fusion on the second background feature map and the second foreground feature map to obtain a second fused feature map; The images of real defects in the industrial product images containing real defects and / or the images of real defects pre-stored in the defect memory are obtained by setting the pixels in the industrial product images containing real defects (excluding those pre-marked as the locations of the real defects) to 0, to random numbers, or to the average value of the pixels in the industrial product images. The good product images are obtained by setting the pixels in the industrial product images without real defects (corresponding to the locations of the real defects in the defect foreground images) to 0, to random numbers, or to the average value of the pixels in the industrial product images.

3. The industrial defect image generation method as described in claim 2, characterized in that, Both the first background feature extraction submodule and the first foreground feature extraction submodule include three standard convolutional layers; wherein, the standard convolutional layer includes a first convolutional layer, a batch normalization layer and an activation layer connected in sequence; The second background feature extraction submodule and the second foreground feature extraction submodule both include a convolutional layer, a max pooling layer, and two cascaded residual modules. Each residual module includes two convolutional sublayers. The input feature map of the max pooling layer is the output feature map of the convolutional layer. The output of each residual module is added to the output of the previous residual module as the input of the next residual module. Specifically, the input feature map of the first residual module is the output feature map of the max pooling layer, and the input feature map of the second residual module is the feature map obtained by adding the output feature map of the first residual module to the output feature map of the max pooling layer.

4. The industrial defect image generation method as described in claim 2, characterized in that, The first feature fusion submodule performs feature fusion on the first background feature map and the first foreground feature map to obtain a first fused feature map, including: Spatial attention enhancement operations are performed on the first background feature map and the first foreground feature map respectively to obtain the first enhanced background feature map and the first enhanced foreground feature map respectively; The first enhanced background feature map and the first enhanced foreground feature map are concatenated along the channel dimension to obtain the first enhanced background and foreground feature map; The first enhanced background and foreground feature maps are extracted by a second convolutional layer to obtain the first fused feature map; in, The second feature fusion submodule performs feature fusion on the second background feature map and the second foreground feature map to obtain a second fused feature map, including: Spatial attention enhancement operations are performed on the second background feature map and the second foreground feature map respectively to obtain the second enhanced background feature map and the second enhanced foreground feature map respectively; The second enhanced background feature map and the second enhanced foreground feature map are concatenated along the channel dimension to obtain the second enhanced background and foreground feature map; The second enhanced background and foreground feature maps are processed by a third convolutional layer to extract features, thereby obtaining the second fused feature map.

5. The industrial defect image generation method as described in claim 4, characterized in that, The step of performing spatial attention enhancement operations on the first background feature map and the first foreground feature map respectively to obtain the first enhanced background feature map and the first enhanced foreground feature map respectively includes: The following steps are performed using the first background feature map and the first foreground feature map as feature maps to be enhanced: Max pooling and average pooling operations are performed on the input feature maps to be enhanced along the channel dimension to obtain a first max pooling feature map and a first average pooling feature map corresponding to the feature maps to be enhanced; the first max pooling feature map and the first average pooling feature map are concatenated along the channel dimension to obtain a first concatenated feature map; features are extracted from the first concatenated feature map through a fourth convolutional layer to obtain a first convolutional feature map; a Sigmoid operation is performed on the first convolutional feature map to obtain a first Sigmoid feature map; the first Sigmoid feature map is multiplied element-wise with the input feature map to be enhanced to obtain an enhanced feature map; the first enhanced background feature map is the enhanced feature map obtained when the feature map to be enhanced is the first background feature map, and the first enhanced foreground feature map is the enhanced feature map obtained when the feature map to be enhanced is the first foreground feature map.

6. The industrial defect image generation method as described in claim 1, characterized in that, The feature diffusion module includes a fifth convolutional layer, and the decoding module includes multiple decoding sub-modules, wherein the output of the previous level decoding sub-module serves as the input of the next level decoding sub-module; wherein each decoding sub-module performs the following steps: performing a deconvolution operation or an upsampling operation on the diffused feature map to obtain a deconvolutional feature map; performing a first convolution operation on the deconvolutional feature map followed by a first ReLU activation operation to obtain a first result; performing a second convolution operation on the first result followed by a second ReLU activation operation to obtain a second result; wherein the decoding module outputs the simulated industrial defect image based on the second result output by the last level decoding sub-module.

7. The industrial defect image generation method as described in claim 1, characterized in that, The method further includes: Before inputting the defect foreground image and the good product image into the pre-trained industrial defect image generation model, one or more of the following operations are performed on the defect foreground image: flipping, rotating, elastically deforming, resizing, and repositioning.

8. The industrial defect image generation method as described in claim 2, characterized in that, The pre-trained industrial defect image generation model is obtained through the following steps: Acquire training sample images and corresponding labeled data; wherein, the training sample images are the industrial product images containing real defects; A good training image is obtained by setting the pixels corresponding to the locations of real defects in the training sample image to 0, random numbers, or the average value of the pixels in the industrial product image. A defect foreground image is obtained by setting the pixels in the training sample image other than those marked as real defects to 0, random numbers, or the average value of the pixels in the industrial product image. The training defect foreground image and the training good product image are input into the industrial defect image generation model. Feature extraction is performed on the training good product image and the defect foreground image used for training to obtain a first background feature map and a first foreground feature map, respectively. Feature fusion is performed on the first background feature map and the first foreground feature map to obtain a first fused feature map. Feature extraction is performed on the first fused feature map to obtain a second background feature map. Feature extraction is performed on the first foreground feature map to obtain a second foreground feature map. Feature fusion is performed on the second background feature map and the second foreground feature map to obtain a second fused feature map. The defect memory in the industrial defect image generation model can automatically save the defect foreground image used for training and use the defect foreground image used for training as the image of the real defect. After adding a labeled noise map of random intensity to the second fused feature map, the image is input into the feature diffusion module to obtain a predicted noise map; based on the predicted noise map, the diffused feature map is obtained through the corresponding processing of the feature diffusion module; based on the diffused feature map, the simulated industrial defect image is output through the processing of the decoding module. The industrial defect image generation model is trained according to the total loss function to obtain the trained industrial defect image generation model. The total loss function is determined by a first loss function and a second loss function. The first loss function is determined based on the predicted noise map and the labeled data, and the second loss function is determined based on the simulated industrial defect image and the training sample image.

9. The industrial defect image generation method as described in claim 8, characterized in that, The step of adding a labeled noise map of random intensity to the second fused feature map and then inputting it into the feature diffusion module to obtain a predicted noise map; based on the predicted noise map, the diffused feature map is obtained through corresponding processing by the feature diffusion module, including: A random intensity of labeled noise map is added to the second fused feature map to obtain a second fused noise feature map; wherein, the corresponding labeled data includes the labeled noise map; The feature diffusion module predicts the labeled noise map in the second fused noise feature map to obtain the predicted noise map; the feature diffusion module outputs the diffusion feature map; wherein the diffusion feature map is obtained by subtracting the predicted noise map from the second fused noise feature map.

10. The industrial defect image generation method as described in claim 8, characterized in that, The expression for the overall loss function is: ， The expression for the first loss function is: ， Among them, the m The labeled noise map t n Or predict the noise map p n The total number of pixels, the i The labeled noise map t n Or predict noise map p n The index of the pixel.

11. The industrial defect image generation method as described in claim 8, characterized in that, The expression for the second loss function is: ， Among them, the L MSE This represents the second content loss function, which is the mean squared error loss function; L t The second texture structure loss function is represented; δ 1 and δ 2 represents the preset weighting coefficients of the second content loss function and the second texture structure loss function, respectively.

12. The industrial defect image generation method as described in claim 11, characterized in that, The expression for the second content loss function is: ， Among them, the n The simulated industrial defect image represents y Or the training sample image x The total number of pixels, the j The simulated industrial defect image represents y Or the training sample image x The index of the pixel.

13. The industrial defect image generation method as described in claim 11, characterized in that, The expression for the second texture structure loss function is: ， Among them, the T ( y, x The expression for ) is: ， Among them, the T ( y, x ( ) is a structural similarity loss function, which is used to measure the structural similarity loss function of the simulated industrial defect image. y and training sample images x The similarity between them, the μ y and μ x The simulated industrial defect images are respectively y and training sample images x The average value of the middle pixels, the and The simulated industrial defect images are respectively y and training sample images x The variance of the middle pixel, σ yx For the simulated industrial defect image y and training sample images x The covariance of the middle pixels, the c 1 and c 2 is the default value.

14. A computer-readable storage medium, characterized in that, Includes a program that can be executed by a processor to implement the method as described in any one of claims 1 to 13.

Images