Defect image generation method and system based on attribute semantic separation, and storage medium

Abstract

A defect image generation method, system, and storage medium based on attribute semantic separation, the method comprising: obtaining a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image; using the defect annotation data of each attribute to perform a mask operation on the defect image to obtain a defect source image of each attribute; obtaining the skeleton information of the image to be generated, and inputting the skeleton information of the defect source image and the image to be generated into a trained defect image generation model for fusion to obtain a defect composite image. By setting the annotation data of each defect attribute, a defect composite image that conforms to the distribution characteristics of the defect attribute can be constructed; by inputting images to be generated with different skeleton morphologies, the morphology of the defect can be effectively expanded. In summary, the technical solution of the present invention can generate defect images with strong randomness and diversity, enriching the training samples of deep learning.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a defect image generation method and system based on attribute semantic separation, and a storage medium. Background Art

[0002] With the development and widespread adoption of deep learning, this technology has been applied to a growing number of fields. For example, deep learning has played a significant role in object defect detection. However, there are still significant differences between current deep learning and human intelligence. For example, in the defect detection process, there are many small defects to detect. These small defects and noise interference can cause images to appear similar, resulting in many over-detection errors. Another example is image classification. Humans can abstract new concepts from a small number of images. For example, suppose a human doesn't know the differences between Labradors and Corgis. After seeing a few images of Labradors and Corgis, they can quickly distinguish between Labradors and Corgis in subsequent images. They can even acquire new concepts without even looking at the images. For example, if a person is told that a striped horse is called a zebra, they will be able to distinguish zebras as a type of horse. Deep learning cannot achieve this level of intelligence. Existing common deep learning models require a large number of sample images for training, and this extensive learning process is required to achieve relatively accurate detection and recognition.

[0003] However, in industrial production processes, defect detection is difficult and expensive to collect, resulting in a small number of defect samples. This makes it very difficult to improve the accuracy and robustness of deep learning training. If the number of sample images used for training is reduced, using only a small number of sample images, the gap between deep learning and human intelligence will increase significantly. Furthermore, because the number of sample images is too small for deep learning, overfitting is easily caused. Summary of the Invention

[0004] The present invention provides a defect image generation method and system based on attribute semantic separation, as well as a storage medium, which mainly solves the technical problem of lack of training samples in the training process of deep learning for defect detection.

[0005] According to the first aspect, an embodiment provides a method for generating a defect image based on attribute semantic separation, comprising:

[0006] Acquire a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image;

[0007] Using the defect annotation data of each attribute to perform a mask operation on the defect image to obtain a defect source image of each attribute;

[0008] Obtaining skeleton information of the image to be generated;

[0009] Inputting the defect source image and the skeleton information of the image to be generated into a trained defect image generation model to obtain a defect composite image;

[0010] The defect image generation model includes a first branch and a second branch, wherein the first branch is a feature extraction network and the second branch is an attribute feature image generation network. Inputting the defect source image and the skeleton information of the image to be generated into the trained defect image generation model to obtain a defect composite image includes:

[0011] The defect source image of each attribute is input into the first branch respectively, and the feature extraction network is used to extract the features to obtain the corresponding defect attribute feature map. The defect attribute feature maps are then fused to obtain a multi-attribute defect feature map.

[0012] The skeleton information of the image to be generated and the multi-attribute defect feature map are input into the second branch, the position feature of the skeleton information is encoded through the attribute feature image generation network to obtain a skeleton feature map, the skeleton feature map and the multi-attribute defect feature map are subjected to feature fusion based on attribute feature migration to obtain a fused feature map, and the fused feature map is used to reconstruct the image to obtain a defect composite image.

[0013] In one embodiment, the feature extraction network includes a first feature extraction branch and a second feature extraction branch, wherein the first feature extraction branch is a feature extraction network pre-trained using a public data set, and its parameters remain unchanged during the training phase, and the second feature extraction branch is a feature extraction network trained using a sample image set, the first feature extraction branch and the second feature extraction branch have the same structure, and the output of each network layer of the first feature extraction branch and the output of the corresponding network layer of the second feature extraction branch are channel-concatenated and used as the input of the next network layer;

[0014] The method of inputting the defect source image of each attribute into the first branch for feature extraction to obtain the corresponding defect attribute feature map includes: for the defect source image of each attribute, inputting it into the first feature extraction branch and the second feature extraction branch respectively, performing channel splicing on the feature maps output by the first feature extraction branch and the second feature extraction branch, and then performing average pooling to obtain the corresponding defect attribute feature map.

[0015] In one embodiment, fusing the defect attribute feature maps to obtain a multi-attribute defect feature map includes: channel-splicing the defect attribute feature maps and then performing a 1×1 convolution operation to obtain the multi-attribute defect feature map.

[0016] In one embodiment, the second branch is a convolutional neural network composed of a position feature encoder, an attribute feature embedding module, an attention module, and a decoder. The skeleton information of the image to be generated and the multi-attribute defect feature map are input into the second branch, the skeleton information is encoded with position features through the attribute feature image generation network to obtain a skeleton feature map, the skeleton feature map and the multi-attribute defect feature map are subjected to feature fusion based on attribute feature migration to obtain a fused feature map, and the fused feature map is used to perform image reconstruction to obtain a defect composite image, including:

[0017] Inputting the skeleton information into the position feature encoder to extract position features to obtain a skeleton feature map;

[0018] Inputting the skeleton feature map and the multi-attribute defect feature map into the attribute feature embedding module to perform attribute feature migration to obtain an attribute embedding feature map;

[0019] The attribute embedding feature map is input into the attention module for feature enhancement processing to obtain a fused feature map;

[0020] The fused feature map is input into the decoder for image reconstruction to obtain a defect composite image.

[0021] In one embodiment, the attribute feature embedding module includes one or more attribute feature embedding submodules, each of which includes a first convolutional layer, a first AdaIN operation layer, an activation layer, a second convolutional layer, and a second AdaIN operation layer connected in sequence, wherein the first AdaIN operation layer is used to perform an AdaIN operation on the output feature map of the first convolutional layer and the multi-attribute defect feature map, and the second AdaIN operation layer is used to perform an AdaIN operation on the output feature map of the second convolutional layer and the multi-attribute defect feature map;

[0022] The input of each attribute feature embedding submodule is the output of the previous attribute feature embedding submodule and the multi-attribute defect feature map, where the input of the first attribute feature embedding submodule is the skeleton feature map and the multi-attribute defect feature map; the output of each attribute feature embedding submodule is the result of adding the output of the second AdaIN operation layer and the output of the previous attribute feature embedding submodule, and the output of the last attribute feature embedding submodule is the attribute embedding feature map.

[0023] In one embodiment, the attention module includes a combined network layer, a fourth convolutional layer, a second Sigmoid operation layer, a fifth convolutional layer, and a third Sigmoid operation layer, wherein the combined network layer includes a third convolutional layer, a batch processing layer, and a first Sigmoid operation layer connected in sequence;

[0024] The step of embedding the attribute into the feature map and inputting it into the attention module for feature enhancement processing to obtain a fused feature map includes:

[0025] Performing average pooling in the X direction and average pooling in the Y direction on the attribute embedding feature map, respectively, splicing the two obtained pooled feature maps and inputting them into the combined network layer for processing to obtain a first intermediate feature map;

[0026] Separating the first intermediate feature map by channel to obtain a second intermediate feature map and a third intermediate feature map;

[0027] Processing the second intermediate feature map sequentially through the fourth convolution layer and the second Sigmoid operation layer to obtain a first attention map, and processing the third intermediate feature map sequentially through the fifth convolution layer and the third Sigmoid operation layer to obtain a second attention map;

[0028] Perform pixel-level multiplication on the first attention map, the second attention map, and the attribute embedding feature map to obtain the fused feature map.

[0029] In one embodiment, the output of the decoder includes the defect composite image and a scoring matrix, each element of the scoring matrix represents the probability that the corresponding area of ​​the defect composite image is a real defect image, and the mean of all elements of the scoring matrix represents the probability that the defect composite image as a whole is a real defect image.

[0030] In one embodiment, the defect image generation model is trained according to a preset total loss function, where the total loss function is composed of a reconstruction loss function, an adversarial loss function, and a texture loss function;

[0031] The reconstruction loss function is:

[0032]

[0033] Where N represents the total number of pixels of the defect composite image and the image to be generated, and N = W × H × C, W, H and C represent the width, height and number of channels of the defect composite image and the image to be generated, respectively, y i represents the i-th pixel of the defect composite image, x i represents the i-th pixel of the image to be generated, || ||1 represents the L1 distance;

[0034] The adversarial loss function is:

[0035]

[0036] Where G represents the generator, which is composed of the position feature encoder, the attribute feature embedding module, and the attention module. D represents the discriminator, which is composed of the decoder. x represents the defect image, x′ represents the image to be generated, E[·] represents the expectation, D(x) represents the result obtained by inputting the defect image into the discriminator, G(x′) represents the defective synthetic image obtained by inputting the image to be generated into the generator, y=G(x′), D(y) represents the result obtained by inputting the defective synthetic image into the discriminator, x~p(x) represents that x obeys the distribution of defective images in the sample image set, y~p G(x′) Indicates that y follows the distribution of defect composite images obtained from the images to be generated in the sample image set;

[0037] The texture loss function is:

[0038]

[0039] where φ l Represents the output feature map of the lth layer of the feature extraction network, N l represents the total number of network layers of the selected feature extraction network, W l 、H l 、C l and represent the width, height and number of channels of the output feature map of the lth layer of the feature extraction network, respectively, l (x) u,v,w represents the pixel at the coordinate (u, v, w) of the output feature map of the lth layer when the defect image is input into the feature extraction network, φ l (y) u,v,w represents the pixel at coordinate (u, v, w) of the output feature map of layer l when the defect composite image is input into the feature extraction network;

[0040] The total loss function is:

[0041] L=L rec +L GAN (G,D)+L con .

[0042] According to the second aspect, an embodiment provides a defect image generation system based on attribute semantic separation, comprising:

[0043] A data acquisition module, configured to acquire a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image;

[0044] A mask module, configured to perform a mask operation on the defect image using the defect annotation data of each attribute to obtain a defect source image of each attribute;

[0045] A skeleton information acquisition module, used to acquire the skeleton information of the image to be generated;

[0046] A defect image synthesis module, configured to input the defect source image and the skeleton information of the image to be generated into a trained defect image generation model to obtain a defect synthesis image;

[0047] The defect image generation model includes a first branch and a second branch, the first branch is a feature extraction network, the second branch is an attribute feature image generation network, and the defect image synthesis module includes an attribute feature extraction submodule and a synthesis submodule;

[0048] The attribute feature extraction submodule is used to input the defect source image of each attribute into the first branch respectively, perform feature extraction through the feature extraction network, obtain the corresponding defect attribute feature map, and fuse the defect attribute feature maps to obtain a multi-attribute defect feature map;

[0049] The synthesis submodule is used to input the skeleton information of the image to be generated and the multi-attribute defect feature map into the second branch, perform position feature encoding on the skeleton information through the attribute feature image generation network to obtain a skeleton feature map, perform feature fusion based on attribute feature migration on the skeleton feature map and the multi-attribute defect feature map to obtain a fused feature map, and use the fused feature map to reconstruct the image to obtain a defect composite image.

[0050] According to a third aspect, an embodiment provides a computer-readable storage medium, on which a program is stored. The program can be executed by a processor to implement the defect image generation method as described in the first aspect.

[0051] According to the defect image generation method / system based on attribute semantic separation of the above-mentioned embodiment, firstly, the defect image and the image to be generated, as well as the defect annotation data of one or more attributes of the defect image are obtained, and the defect annotation data of each attribute are used to perform a mask operation on the defect image to obtain the defect source image of each attribute, and the skeleton information of the image to be generated is obtained, and the skeleton information of the defect source image and the image to be generated are input into the trained defect image generation model for fusion to obtain a defect composite image, so that by setting the annotation data of each defect attribute separately, a defect composite image that conforms to the distribution characteristics of the defect attribute can be constructed; the skeleton information determines the morphology and basic posture of the generated defect, so by inputting the image to be generated with different skeleton morphologies, the morphology of the defect can be effectively expanded. In summary, the defect image generation method provided by the present invention enhances the randomness and diversity of defect morphology, broadens the distribution of defect sample images, and can enrich the training samples of deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1A flowchart of a defect image generation method based on attribute semantic separation according to an embodiment;

[0053] Figure 2 A schematic structural diagram of a defect image generation model according to an embodiment;

[0054] Figure 3 A flowchart of an embodiment of inputting the skeleton information of the defect source image and the image to be generated into a trained defect image generation model to obtain a defect composite image;

[0055] Figure 4 A schematic diagram of the structure of a feature extraction network according to an embodiment;

[0056] Figure 5 A schematic diagram of the process of extracting features from defect source images of various attributes and then fusing them to obtain a multi-attribute defect feature map;

[0057] Figure 6 A schematic diagram of the structure of an attribute feature image generation network according to an embodiment;

[0058] Figure 7 A flowchart of an embodiment of the present invention in which an attribute feature image generation network processes skeleton information and a multi-attribute defect feature map to obtain a defect composite image;

[0059] Figure 8 A schematic diagram of the structure of an attribute feature embedding submodule according to an embodiment;

[0060] Figure 9 Schematic diagram of the structure of an attention module according to an embodiment;

[0061] Figure 10 Schematic diagram of the structure of a defect image generation system based on attribute semantic separation according to an embodiment;

[0062] Figure 11 It is a structural diagram of a defect image generation system based on attribute semantic separation according to another embodiment. DETAILED DESCRIPTION

[0063] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0064] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0065] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0066] Currently, there are two main methods for improving training results in scenarios where training samples are scarce. The first is to increase the number of samples through data augmentation and regularization. The idea behind this method is straightforward: since there is little training data, additional training data is added. For example, existing sample images are processed through image processing methods such as rotation, translation, contrast conversion, blurring, and adding white noise to obtain new sample images. Since overfitting can occur, a regularization term is added to alleviate the overfitting problem. However, the images generated by this method are highly similar to the original images, which is equivalent to reusing data from multiple similar images during the training process. This makes it difficult to address the issues of sample diversity and single distribution.

[0067] Another approach, currently the more mainstream, is meta-learning. The goal of meta-learning is to leverage previously learned knowledge to solve new problems. This approach is based on human learning mechanisms, which build on existing knowledge rather than starting from scratch like deep learning. If existing prior knowledge can be leveraged to help solve new problems, then learning for the new problem requires fewer examples, thus addressing the sample shortage issue. However, meta-learning requires additional data to learn prior knowledge. For example, for a classification problem, assuming data with 100 categories, 60 categories can be used to learn prior knowledge, 20 categories for validation, and the remaining 20 categories for testing. These 20 categories used for testing should be completely different from the previous 80 categories—that is, new classes, new concepts, and new problems. Each of these 20 categories should have only a few images. The previous 80 categories are used to train the model and determine hyperparameters, essentially learning prior knowledge that helps solve the new problem. The effectiveness of this method is related to the similarity between the two data sets. If the two data sets are very similar, the learned prior knowledge can well solve the problem of lack of samples in the new data set. If the difference is large, the effect may be general.

[0068] To address the shortage of defect sample images, the present invention provides a defect image generation method based on attribute semantic separation to address the lack of training samples during the training process. This method can generate defect sample images that conform to actual scenarios based on a small number of acquired defect images with different attributes, meeting the needs of data generation in actual production processes where samples are scarce. Based on the attributes of the input defect source image and the skeleton information of the image to be generated, the generated defect image can have different attributes and different morphologies, thereby obtaining attribute-separated defect sample images and broadening the distribution of defect sample images.

[0069] Please refer to Figure 1 In one embodiment, the method includes steps 110 to 140, which are described in detail below.

[0070] Step 110: Acquire a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image.

[0071] A defect image is an image of an object with defects. The attributes of a defect image can refer to the defect type, such as dents, scratches, or cracks, or the location of the defect within an entire object. For example, a bicycle as a whole includes handlebars, a seat, wheels, and other parts, each of which can be considered an attribute. A defect image can have multiple different attributes.

[0072] Each attribute of the defect image can be annotated separately to obtain the defect annotation data of the attribute. Figure 5 , Figure 5 The defect image in includes two types of defects: one is a strip defect on the left, and the other is a ring defect on the right. Defect annotation data 1 marks the strip defect, and defect annotation data 2 marks the ring defect. Figure 4 and Figure 5 As shown, the defect annotation data can be similar to a mask image of the same size as the defect image, where the pixel value of the defective part is 1 and the pixel value of the non-defective part is 0.

[0073] The image to be generated can be an image of an object in the actual scene, which is used to extract skeleton information. The defect features of each attribute of the defect image will be fused to the skeleton to generate a defect composite image with the skeleton morphology.

[0074] Step 120: Use the defect annotation data of each attribute to perform a mask operation on the defect image to obtain a defect source image of each attribute.

[0075] Specifically, the defect annotation data and the defect image can be multiplied at the pixel level, that is, the corresponding pixels are multiplied. Let the defect annotation data of the i-th attribute be M i , the defect image is I s , the corresponding defect source image of the i-th attribute can be obtained through pixel-level multiplication operation, which can be expressed as:

[0076]

[0077] Step 130: Obtain skeleton information of the image to be generated.

[0078] The skeleton information determines the shape and basic posture of the generated defect. By inputting the image to be generated with different skeleton shapes, the shape of the defect can be effectively expanded. The skeleton information can be obtained by using a feature map P of the same size as the image to be generated. t Skeleton information can be manually drawn, with the user drawing the corresponding graphics for the desired defect form, such as a circle or arc. Traditional image processing or deep learning can also be used to extract it from the image to be generated. For details, please refer to existing technologies. For example, the image to be generated can be input into a pre-trained key point extraction module to obtain a key point feature map as skeleton information.

[0079] Step 140: Input the defect source image and the skeleton information of the image to be generated into the trained defect image generation model to obtain a defect composite image.

[0080] The defect image generation model is a deep learning neural network model, please refer to Figure 2In one embodiment, the model includes two branches: a first branch and a second branch. The first branch is a feature extraction network that extracts features from the defect source image; the second branch is an attribute feature image generation network that fuses the extracted features of the defect source image with skeleton information to generate a composite defect image with corresponding attribute features.

[0081] Please refer to Figure 3 In one embodiment, step 140 includes steps 141 to 142, which are introduced below respectively.

[0082] Step 141: Input the defect source image of each attribute into the first branch respectively, perform feature extraction through the feature extraction network to obtain the corresponding defect attribute feature map, and fuse the defect attribute feature maps to obtain a multi-attribute defect feature map.

[0083] Please refer to Figure 4 The feature extraction network of one embodiment of the present invention includes a first feature extraction branch and a second feature extraction branch. The first feature extraction branch is a feature extraction network pre-trained using a public data set, and the second feature extraction branch is a feature extraction network trained using a sample image set. When training the model, the model is trained using a sample image set composed of collected defect images and images to be generated. During the training phase, the parameters of the first feature extraction branch remain unchanged and do not participate in the training. The second feature extraction branch is the network structure to be trained, and the parameters of the second feature extraction branch are updated to obtain its network parameters.

[0084] like Figure 4 As shown, the first feature extraction branch and the second feature extraction branch have the same structure. The output of each network layer of the first feature extraction branch and the output of the corresponding network layer of the second feature extraction branch are channel-joined and serve as the input of the next network layer.

[0085] Please refer to Figure 4 In this embodiment, the defect source image of each attribute is input into the first branch respectively, and feature extraction is performed through the feature extraction network to obtain the corresponding defect attribute feature map. The specific process is as follows: for the defect source image of each attribute, it is input into the first feature extraction branch and the second feature extraction branch respectively, the feature maps output by the first feature extraction branch and the second feature extraction branch are channel-spliced, and then average pooling is performed to obtain the corresponding defect attribute feature map. Figure 4 In the figure, ⊙ represents pixel-level multiplication. Indicates channel splicing.

[0086] This embodiment sets two branches for the feature extraction network, one of which is a feature extraction network pre-trained using a public data set, so that the entire feature extraction network has a certain feature extraction capability at the initial stage. The second feature extraction branch is guided by the pre-trained feature extraction network for training, thereby accelerating the training speed.

[0087] Please refer to Figure 5 After extracting the features of each attribute of the defect source image through the feature extraction network to obtain the corresponding defect attribute feature map, the defect attribute feature maps are fused to obtain a multi-attribute defect feature map. For the fusion processing of the defect attribute feature map, methods such as channel splicing can be used, and then a 1×1 convolution operation is performed to obtain a multi-attribute defect feature map. The multi-attribute defect feature map is denoted by C fusion express.

[0088] Step 142: Input the skeleton information and the multi-attribute defect feature map of the image to be generated into the second branch, perform position feature encoding on the skeleton information through the attribute feature image generation network to obtain a skeleton feature map, perform feature fusion based on attribute feature migration on the skeleton feature map and the multi-attribute defect feature map to obtain a fused feature map, use the fused feature map to reconstruct the image, and obtain a defect composite image.

[0089] Please refer to Figure 6 In one embodiment, the attribute feature image generation network of the second branch is a convolutional neural network composed of a position feature encoder, an attribute feature embedding module, an attention module, and a decoder. The position feature encoder is mainly used to extract the position feature and extract the position of the skeleton. The attribute feature embedding module is mainly used to fuse the position features of the skeleton and the texture features of the multi-attribute defect feature map. The attention module is mainly used to enhance the key features. The decoder is mainly used to reconstruct the image based on the features and generate a sample image of the defect.

[0090] Please refer to Figure 7 In this embodiment, step 142 specifically includes steps 1421 to 1424, which are described in detail below.

[0091] Step 1421: Input the skeleton information into the position feature encoder to extract the position features to obtain a skeleton feature map.

[0092] The position feature encoder is mainly composed of convolutional layers, pooling layers, and nonlinear activation functions. It is used to extract the position features of the skeleton information and output the skeleton feature map C1. The skeleton feature map C1 can be a feature map of the same size as the image to be generated.

[0093] Step 1422: Input the skeleton feature map and the multi-attribute defect feature map into the attribute feature embedding module to perform attribute feature migration to obtain an attribute embedding feature map.

[0094] The multi-attribute defect feature map reflects the defect features of each attribute of the defect image, including texture features. The attribute feature embedding module receives the skeleton feature map and the multi-attribute defect feature map to perform attribute feature migration, which means migrating the defect features of each attribute to the extracted skeleton.

[0095] Please refer to Figure 7 and Figure 8 In one embodiment, the attribute feature embedding module includes one or more attribute feature embedding submodules, each of which includes a first convolutional layer, a first AdaIN operation layer, an activation layer, a second convolutional layer, and a second AdaIN operation layer, connected in sequence. The first AdaIN operation layer and the second AdaIN operation layer are used to perform AdaIN operations, and the activation layer can specifically use a ReLU activation function.

[0096] The input of each attribute feature embedding submodule consists of two parts: one is the output H of the previous attribute feature embedding submodule t-1 , one is the multi-attribute defect feature map C fusion , where t represents the serial number of the attribute feature embedding submodule. For the first attribute feature embedding submodule, H0=C1, that is, the input of the first attribute feature embedding submodule is the skeleton feature map C1 and the multi-attribute defect feature map C fusion The output H of the previous attribute feature embedding submodule t-1 After convolution by the first convolutional layer, the output feature map is input into the first AdaIN operation layer and the multi-attribute defect feature map C fusion Perform AdaIN operation, the operation result is processed by the activation layer and then enters the first convolution layer for convolution, and the output feature map obtained is input into the second AdaIN operation layer and the multi-attribute defect feature map C fusion Perform AdaIN operation, the output of the second AdaIN operation layer is combined with the output H of the previous attribute feature embedding submodule t-1 Add up to get the output H of the current attribute feature embedding submodule t The output of the last attribute feature embedding submodule is used as the attribute embedding feature map. The operation of each attribute feature embedding submodule can be expressed as:

[0097]

[0098] in It represents the comprehensive operation process of the first convolutional layer, the first AdaIN operation layer, the activation layer, the second convolutional layer and the second AdaIN operation layer of the t-th attribute feature embedding submodule.

[0099] For the AdaIN operation, it can be expressed as:

[0100]

[0101] Where x represents the output feature map of the first convolutional layer or the second convolutional layer, y = C fusion , μ(x) and μ(y) represent the mean vector composed of the mean of each channel of the feature map x, and the multi-attribute defect feature map C fusion The mean vector composed of the mean of each channel, σ(x) and σ(y) represent the standard deviation vector composed of the standard deviation of each channel of the feature map x, and the multi-attribute defect feature map C fusion The standard deviation vector is composed of the standard deviation of each channel. The operation of the feature map x and each mean vector and standard deviation vector in the formula represents the operation of each pixel of each channel of the feature map x and the mean and standard deviation of the corresponding channel.

[0102] For the mean and standard deviation, taking the feature graph x as an example, they can be calculated according to the following formula:

[0103]

[0104]

[0105] Where subscript n is the index of feature map x in a batch, subscript c is the index of feature map x channel, H x is the height of the feature map x, W x is the width of the feature map x, x nchw It represents the pixel value at the coordinate (h, w) on the x channel c of the nth feature map in a batch, and ε is a preset small error value, which can be 0.01.

[0106] Multi-attribute defect feature map C fusion The mean and standard deviation can be calculated similarly using the above formula.

[0107] It can be understood that the feature map H output by each attribute feature embedding submodule t The scale of the image may change, so by connecting multiple attribute feature embedding sub-modules for processing, features of different scales can be fused, and the image generation model can be used to reconstruct images using features of multiple scales.

[0108] Step 1423: Input the attribute embedding feature map into the attention module for feature enhancement processing to obtain a fused feature map.

[0109] Position information has a great influence on the accuracy of image reconstruction. Therefore, an embodiment of the present invention designs a special attention module to embed position information into channel attention, so as to achieve accurate reconstruction of defect image attribute level. Figure 9, may include a combined network layer CBS, a fourth convolution layer, a second Sigmoid operation layer, a fifth convolution layer and a third Sigmoid operation layer, wherein the combined network layer CBS includes a third convolution layer (Convolution), a batch processing layer (Batch Normalization) and a first Sigmoid operation layer connected in sequence.

[0110] Please refer to Figure 9 Based on the above structure, the specific process of step 1423 is as follows: First, average pooling (AP) in the X direction and average pooling in the Y direction are performed on the attribute embedding feature map, that is, the pooling window is moved in the X and Y directions respectively to perform pooling. The two pooled feature maps obtained are concatenated and input into the combined network layer CBS for processing to obtain a first intermediate feature map. The first intermediate feature map is separated by channel to obtain a second intermediate feature map and a third intermediate feature map. Specifically, the channel separation can be averaged. For example, if the first intermediate feature map has C channels, C / 2 of the channels are used as the second intermediate feature map, and the remaining C / 2 channels are used as the third intermediate feature map. The second intermediate feature map is processed sequentially through the fourth convolutional layer and the second sigmoid operation layer to obtain the first attention map. The third intermediate feature map is processed sequentially through the fifth convolutional layer and the third sigmoid operation layer to obtain the second attention map. Finally, pixel-level multiplication is performed on the first attention map, the second attention map, and the attribute embedding feature map to obtain a fused feature map.

[0111] The attention module of this embodiment can fuse channel and position features, which is conducive to extracting key feature information such as position information for reconstruction of defect images. In essence, the attention module of this embodiment divides the vertical and horizontal features of the input feature map into two independent direction-aware feature maps. These two feature maps embedded with specific direction information are then encoded into two attention maps. Each attention map captures the long-range dependency of the input feature map along a spatial direction. Therefore, the position information is stored in the attention map. Finally, the two attention maps are multiplied by the input feature map to enhance the representation ability of the input feature map. This can help the model more accurately locate and identify targets of interest, thereby achieving accurate image reconstruction.

[0112] In addition, the attention module of this embodiment separates the first intermediate feature map into two feature maps by channel, performs convolution and Sigmoid operations on the two feature maps to obtain two attention maps, and then multiplies both attention maps with the attribute embedding feature map. Since the separated feature map has fewer parameters, the amount of calculation is greatly reduced compared to using the complete first intermediate feature map for the whole process calculation, and similar effects can be achieved.

[0113] Step 1424: Input the fused feature map into the decoder for image reconstruction to obtain a defect composite image.

[0114] The decoder is primarily used to reconstruct images based on the feature maps obtained by the previous modules, generating a composite defect image. The decoder can consist of several convolutional and upsampling layers. After the fused feature maps enter the decoder, they undergo multiple convolution operations and upsampling to produce the final composite defect image. The composite defect image can be the same size as the image to be generated.

[0115] The decoder and the previous modules can form a generative adversarial network, in which the generator is composed of a position feature encoder, an attribute feature embedding module, and an attention module, and the discriminator is composed of a decoder. The discriminator is used to output a score value to evaluate the quality of the defective synthetic image. In one embodiment, in addition to the defective synthetic image, the output of the decoder may also include a scoring matrix, in which each element of the scoring matrix represents the probability that the corresponding area of ​​the defective synthetic image is a real defective image, and the mean of all elements of the scoring matrix represents the probability that the defective synthetic image as a whole is a real defective image. Specifically, the scoring matrix can be an M×M matrix, where M is an integer not less than 2. For example, if it is a 3×3 matrix, it means that the defective synthetic image is divided into nine equal parts, and each element of the scoring matrix represents the probability that the corresponding position area is a real defective image. The mean of the 9 elements represents the probability that the defective synthetic image as a whole is a real defective image.

[0116] In previous generative adversarial network learning, the output of the discriminator was a scalar between 0 and 1, representing the probability that the image generated by the generator was a real image. To improve the reconstruction and representation capabilities of images, the decoder of this embodiment outputs an M×M matrix, where each element corresponds to a region and a receptive field of the image. Finally, the average of the scores of each region is taken to represent the probability that the generated image is a real image overall. In other words, instead of using a single value to evaluate the entire image, an M×M matrix is ​​now used to evaluate the entire image. Intuitively, this can be understood as follows: the discriminator of a conventional generative adversarial network only outputs a single number, like a one-man show, while the discriminator of this embodiment outputs a matrix, and the elements of the matrix are averaged to evaluate whether the generated image is a real image, taking into account the influence of different parts of the image, just like considering the suggestions of multiple people before making a decision.

[0117] The discriminator in a traditional generative adversarial network determines whether an image is synthetic or real by considering the overall image, potentially overlooking local details. However, the discriminator in this embodiment outputs a two-dimensional matrix, where each element represents a judgment about a local region of the image. This allows the local texture details of the image to be highlighted and enhanced.

[0118] In fact, some studies have shown that the discriminator of a conventional generative adversarial network is not suitable for image fields that require high resolution and high-definition details. Therefore, this embodiment proposes a region-based discriminator whose receptive field corresponds to a small area of ​​the image. In other words, it corresponds to the discriminator's judgment of a small area of ​​the image. Training with a scoring matrix can enable the model to pay more attention to image details. This mechanism proposed in this embodiment integrates local image features with overall image characteristics, and determines the authenticity of each region. This realizes the extraction and characterization of local image features, which is conducive to the production of higher-resolution images. At the same time, after averaging the elements of the scoring matrix, it is also possible to compare the real image and the generated image.

[0119] When training the defect image generation model, a sample image set is used according to a preset total loss function. The sample image set includes both the actual acquired defect image and the image to be generated. The defect image and the image to be generated are input into the defect image generation model to generate a composite defect image. The loss function value is then calculated and the model is updated. In one embodiment, the defect image, the image to be generated, and the composite defect image are of equal size, and the total loss function is composed of a reconstruction loss function, an adversarial loss function, and a texture loss function.

[0120] The reconstruction loss function is:

[0121]

[0122] Where N represents the total number of pixels of the defect composite image and the image to be generated, and N = W × H × C, W, H, and C represent the width, height, and number of channels of the defect composite image and the image to be generated, respectively. That is, the specifications of the defect composite image and the image to be generated are both W × H × C, and the total number of pixels is N; y i represents the i-th pixel of the defect composite image, x i represents the i-th pixel of the image to be generated, and ||||1 represents the L1 distance.

[0123] The adversarial loss function is:

[0124]

[0125] Where G represents the generator, D represents the discriminator, x represents the defect image, x′ represents the image to be generated, E[·] represents the expectation, D(x) represents the result obtained by inputting the defect image x into the discriminator, G(x′) represents the defect composite image obtained by inputting the image to be generated x′ into the generator, y=G(x′), D(y) represents the result obtained by inputting the defect composite image into the discriminator, x~p(x) represents that x obeys the distribution of defect images in the sample image set, y~p G(x′) It means that y follows the distribution of defect composite images obtained from the images to be generated in the sample image set.

[0126] N of the feature extraction network can be pre-selected before training l The layers are used to calculate the texture loss function. The texture loss function is:

[0127]

[0128] where φ l Represents the output feature map of the lth layer of the feature extraction network, N l represents the total number of network layers of the selected feature extraction network, W l 、H l 、C l and represent the width, height and number of channels of the output feature map of the lth layer of the feature extraction network, respectively, l (x) u,v,w represents the pixel at the coordinate (u, v, w) of the output feature map of the lth layer when the defect image is input into the feature extraction network, φ l (y) u,v,w Represents the pixel at coordinate (u, v, w) of the output feature map of layer l when the defect synthesis image is input into the feature extraction network.

[0129] Then the total loss function is:

[0130] L=L rec +L GAN (G,D)+L con .

[0131] Based on the above-mentioned defect image generation method based on attribute semantic separation, the present invention also provides a defect image generation system based on attribute semantic separation, please refer to Figure 10 In one embodiment, the system includes a data acquisition module 1, a mask module 2, a skeleton information acquisition module 3 and a defect image synthesis module 4, which are described below respectively.

[0132] The data acquisition module 1 is used to acquire a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image.

[0133] The mask module 2 is used to perform a mask operation on the defect image using the defect annotation data of each attribute to obtain the defect source image of each attribute.

[0134] The skeleton information acquisition module 3 is used to obtain the skeleton information of the image to be generated.

[0135] The defect image synthesis module 4 is used to input the defect source image and the skeleton information of the image to be generated into the trained defect image generation model to obtain a defect synthesis image. The defect image generation model includes a first branch and a second branch. The first branch is a feature extraction network for extracting features from the defect source image; the second branch is an attribute feature image generation network, which mainly fuses the extracted features of the defect source image with the skeleton information to generate a defect synthesis image with corresponding attribute features. Please refer to Figure 10 The defect image synthesis module includes an attribute feature extraction submodule 41 and a synthesis submodule 42.

[0136] The attribute feature extraction submodule 41 is used to input the defect source image of each attribute into the first branch respectively, perform feature extraction through the feature extraction network, obtain the corresponding defect attribute feature map, and fuse the defect attribute feature maps to obtain a multi-attribute defect feature map.

[0137] In one embodiment, the feature extraction network includes a first feature extraction branch and a second feature extraction branch. For details, please refer to the corresponding introduction above. The attribute feature extraction submodule 41 is specifically used to input the defect source image of each attribute into the first feature extraction branch and the second feature extraction branch respectively, and perform channel splicing on the feature maps output by the first feature extraction branch and the second feature extraction branch, and then perform average pooling to obtain the corresponding defect attribute feature map.

[0138] In one embodiment, the attribute feature extraction submodule 41 is further configured to perform channel splicing on each defect attribute feature map and then perform a 1×1 convolution operation to obtain a multi-attribute defect feature map.

[0139] The synthesis submodule 42 is used to input the skeleton information and multi-attribute defect feature map of the image to be generated into the second branch, perform position feature encoding on the skeleton information through a convolutional neural network to obtain a skeleton feature map, perform feature fusion based on attribute feature migration on the skeleton feature map and the multi-attribute defect feature map to obtain a fused feature map, and use the fused feature map to reconstruct the image to obtain a defect composite image.

[0140] In one embodiment, the attribute feature image generation network of the second branch is a convolutional neural network composed of a position feature encoder, an attribute feature embedding module, an attention module and a decoder. For details, please refer to the corresponding introduction above. The synthesis submodule 42 is specifically used to: input the skeleton information into the position feature encoder for position feature extraction to obtain a skeleton feature map; input the skeleton feature map and the multi-attribute defect feature map into the attribute feature embedding module for attribute feature migration to obtain an attribute embedding feature map; input the attribute embedding feature map into the attention module for feature enhancement processing to obtain a fused feature map; input the fused feature map into the decoder for image reconstruction to obtain a defect composite image.

[0141] In one embodiment, the attribute feature embedding module includes one or more attribute feature embedding submodules, each attribute feature embedding submodule includes a first convolutional layer, a first AdaIN operation layer, an activation layer, a second convolutional layer, and a second AdaIN operation layer connected in sequence, the first AdaIN operation layer is used to perform AdaIN operation on the output feature map of the first convolutional layer and the multi-attribute defect feature map, and the second AdaIN operation layer is used to perform AdaIN operation on the output feature map of the second convolutional layer and the multi-attribute defect feature map. The input of each attribute feature embedding submodule is the output of the previous attribute feature embedding submodule and the multi-attribute defect feature map, wherein the input of the first attribute feature embedding submodule is the skeleton feature map and the multi-attribute defect feature map; the output of each attribute feature embedding submodule is the result of adding the output of the second AdaIN operation layer to the output of the previous attribute feature embedding submodule, and the output of the last attribute feature embedding submodule is used as the attribute embedding feature map. For the specific working process of the attribute feature embedding module, please refer to the corresponding introduction above and will not be repeated here.

[0142] In one embodiment, the attention module includes a combined network layer, a fourth convolutional layer, a second Sigmoid operation layer, a fifth convolutional layer, and a third Sigmoid operation layer, wherein the combined network layer includes a third convolutional layer, a batch processing layer, and a first Sigmoid operation layer connected in sequence. The synthesis submodule 42 is used to: perform average pooling in the X direction and average pooling in the Y direction on the attribute embedding feature map, splice the two obtained pooling feature maps, and input them into the combined network layer for processing to obtain a first intermediate feature map; separate the first intermediate feature map by channel to obtain a second intermediate feature map and a third intermediate feature map; process the second intermediate feature map sequentially through the fourth convolutional layer and the second Sigmoid operation layer to obtain a first attention map, and process the third intermediate feature map sequentially through the fifth convolutional layer and the third Sigmoid operation layer to obtain a second attention map; perform pixel-level multiplication on the first attention map, the second attention map, and the attribute embedding feature map to obtain a fused feature map.

[0143] In one embodiment, the output of the decoder includes a defective composite image and a scoring matrix, wherein each element of the scoring matrix represents the probability that the corresponding area of ​​the defective composite image is a real defective image, and the mean of all elements of the scoring matrix represents the probability that the defective composite image as a whole is a real defective image.

[0144] Please refer to Figure 11In one embodiment, the defect image generation system based on attribute semantic separation further includes a model training module 5, which is configured to train a defect image generation model based on a preset total loss function, wherein the total loss function is composed of a reconstruction loss function, an adversarial loss function, and a texture loss function. For details about the total loss function, the reconstruction loss function, the adversarial loss function, and the texture loss function, please refer to the corresponding description above and will not be repeated here.

[0145] According to the defect image generation method / system based on attribute semantic separation of the above-mentioned embodiment, the defect image and the image to be generated, as well as the defect annotation data of one or more attributes of the defect image, are first obtained. The defect annotation data of each attribute is used to perform a mask operation on the defect image to obtain the defect source image of each attribute, and the skeleton information of the image to be generated is obtained. The skeleton information of the defect source image and the image to be generated are input into a trained defect image generation model for fusion to obtain a defect composite image. By separately setting the annotation data of each defect attribute, a defect composite image that conforms to the distribution characteristics of the defect attribute can be constructed. In addition, because the skeleton information determines the morphology and basic posture of the generated defect, the morphology of the defect can be effectively expanded by inputting images to be generated with different skeleton morphologies.

[0146] In summary, the defect image generation method provided by the present invention enhances the randomness and diversity of defect morphology, broadens the distribution of defect sample images, and can obtain other defect images that conform to actual scenarios by using a small number of real defect images. It can enrich the training sample data for deep learning and meet the needs of data generation in sample-scarce scenarios in actual production processes.

[0147] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0148] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A defect image generation method based on attribute semantic separation, characterized in that: include: Acquire a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image; Using the defect annotation data of each attribute to perform a mask operation on the defect image to obtain a defect source image of each attribute; Obtaining skeleton information of the image to be generated; Inputting the defect source image and the skeleton information of the image to be generated into a trained defect image generation model to obtain a defect composite image; The defect image generation model includes a first branch and a second branch, wherein the first branch is a feature extraction network and the second branch is an attribute feature image generation network. Inputting the defect source image and the skeleton information of the image to be generated into the trained defect image generation model to obtain a defect composite image includes: The defect source image of each attribute is input into the first branch respectively, and the feature extraction network is used to extract the features to obtain the corresponding defect attribute feature map. The defect attribute feature maps are then fused to obtain a multi-attribute defect feature map. The skeleton information of the image to be generated and the multi-attribute defect feature map are input into the second branch, the position feature of the skeleton information is encoded through the attribute feature image generation network to obtain a skeleton feature map, the skeleton feature map and the multi-attribute defect feature map are subjected to feature fusion based on attribute feature migration to obtain a fused feature map, and the fused feature map is used to reconstruct the image to obtain a defect composite image.

2. The defect image generation method according to claim 1, wherein: The feature extraction network includes a first feature extraction branch and a second feature extraction branch, wherein the first feature extraction branch is a feature extraction network pre-trained using a public data set, and its parameters remain unchanged during the training phase, and the second feature extraction branch is a feature extraction network trained using a sample image set, the first feature extraction branch and the second feature extraction branch have the same structure, and the output of each network layer of the first feature extraction branch and the output of the corresponding network layer of the second feature extraction branch are channel-concatenated and used as the input of the next network layer; The method of inputting the defect source image of each attribute into the first branch for feature extraction to obtain the corresponding defect attribute feature map includes: for the defect source image of each attribute, inputting it into the first feature extraction branch and the second feature extraction branch respectively, performing channel splicing on the feature maps output by the first feature extraction branch and the second feature extraction branch, and then performing average pooling to obtain the corresponding defect attribute feature map.

3. The defect image generation method according to claim 1 or 2, wherein: The fusing of the defect attribute feature maps to obtain a multi-attribute defect feature map includes: channel-splicing the defect attribute feature maps and then performing a 1×1 convolution operation to obtain the multi-attribute defect feature map.

4. The defect image generation method according to claim 1, wherein: The second branch is a convolutional neural network composed of a position feature encoder, an attribute feature embedding module, an attention module, and a decoder. The skeleton information of the image to be generated and the multi-attribute defect feature map are input into the second branch, the skeleton information is encoded with position features through the attribute feature image generation network to obtain a skeleton feature map, the skeleton feature map and the multi-attribute defect feature map are subjected to feature fusion based on attribute feature migration to obtain a fused feature map, and the fused feature map is used to perform image reconstruction to obtain a defect composite image, including: Inputting the skeleton information into the position feature encoder to extract position features to obtain a skeleton feature map; Inputting the skeleton feature map and the multi-attribute defect feature map into the attribute feature embedding module to perform attribute feature migration to obtain an attribute embedding feature map; The attribute embedding feature map is input into the attention module for feature enhancement processing to obtain a fused feature map; The fused feature map is input into the decoder for image reconstruction to obtain a defect composite image.

5. The defect image generation method according to claim 4, wherein: The attribute feature embedding module includes one or more attribute feature embedding submodules, each of which includes a first convolutional layer, a first AdaIN operation layer, an activation layer, a second convolutional layer, and a second AdaIN operation layer connected in sequence, wherein the first AdaIN operation layer is used to perform an AdaIN operation on the output feature map of the first convolutional layer and the multi-attribute defect feature map, and the second AdaIN operation layer is used to perform an AdaIN operation on the output feature map of the second convolutional layer and the multi-attribute defect feature map; The input of each attribute feature embedding submodule is the output of the previous attribute feature embedding submodule and the multi-attribute defect feature map, where the input of the first attribute feature embedding submodule is the skeleton feature map and the multi-attribute defect feature map; the output of each attribute feature embedding submodule is the result of adding the output of the second AdaIN operation layer and the output of the previous attribute feature embedding submodule, and the output of the last attribute feature embedding submodule is the attribute embedding feature map.

6. The defect image generation method according to claim 5, wherein: The formula for the AdaIN operation is: Wherein, x represents the output feature map of the first convolutional layer or the second convolutional layer, y represents the multi-attribute defect feature map, μ(x) and μ(y) respectively represent the mean vector composed of the means of each channel of the feature map x and the mean vector composed of the means of each channel of the multi-attribute defect feature map, σ(x) and σ(y) respectively represent the standard deviation vector composed of the standard deviations of each channel of the feature map x and the standard deviation vector composed of the standard deviations of each channel of the multi-attribute defect feature map. The operation between the feature map x and each mean vector and standard deviation vector in the formula represents the operation between each pixel of each channel of the feature map x and the mean and standard deviation of the corresponding channel.

7. The defect image generation method according to claim 4, wherein: The attention module includes a combined network layer, a fourth convolutional layer, a second Sigmoid operation layer, a fifth convolutional layer, and a third Sigmoid operation layer, wherein the combined network layer includes a third convolutional layer, a batch processing layer, and a first Sigmoid operation layer connected in sequence; The step of embedding the attribute into the feature map and inputting it into the attention module for feature enhancement processing to obtain a fused feature map includes: Performing average pooling in the X direction and average pooling in the Y direction on the attribute embedding feature map, respectively, splicing the two obtained pooled feature maps and inputting them into the combined network layer for processing to obtain a first intermediate feature map; Separating the first intermediate feature map by channel to obtain a second intermediate feature map and a third intermediate feature map; Processing the second intermediate feature map sequentially through the fourth convolution layer and the second Sigmoid operation layer to obtain a first attention map, and processing the third intermediate feature map sequentially through the fifth convolution layer and the third Sigmoid operation layer to obtain a second attention map; Perform pixel-level multiplication on the first attention map, the second attention map, and the attribute embedding feature map to obtain the fused feature map.

8. The defect image generation method according to claim 4, wherein: The output of the decoder includes the defect composite image and a scoring matrix, wherein each element of the scoring matrix represents the probability that the corresponding area of ​​the defect composite image is a real defect image, and the mean of all elements of the scoring matrix represents the probability that the defect composite image as a whole is a real defect image.

9. The defect image generating method according to any one of claims 4 to 8, wherein: The defect image generation model is trained according to a preset total loss function, wherein the total loss function is composed of a reconstruction loss function, an adversarial loss function, and a texture loss function; The reconstruction loss function is: Where N represents the total number of pixels of the defect composite image and the image to be generated, and N = W × H × C, W, H and C represent the width, height and number of channels of the defect composite image and the image to be generated respectively; y i represents the i-th pixel of the defect composite image, x i represents the i-th pixel of the image to be generated, || ||1 represents the L1 distance; The adversarial loss function is: Where G represents the generator, which is composed of the position feature encoder, the attribute feature embedding module, and the attention module. D represents the discriminator, which is composed of the decoder. x represents the defect image, x′ represents the image to be generated, E[·] represents the expectation, D(x) represents the result obtained by inputting the defect image into the discriminator, G(x′) represents the defective synthetic image obtained by inputting the image to be generated into the generator, y=G(x′), D(y) represents the result obtained by inputting the defective synthetic image into the discriminator, x~p(x) represents that x obeys the distribution of defective images in the sample image set, y~p G(x′) Indicates that y follows the distribution of defect composite images obtained from the images to be generated in the sample image set; The texture loss function is: where φ l Represents the output feature map of the lth layer of the feature extraction network, N l represents the total number of network layers of the selected feature extraction network, W l 、H l 、C l and represent the width, height and number of channels of the output feature map of the lth layer of the feature extraction network, respectively, l (x) u,v,w represents the pixel at the coordinate (u, v, w) of the output feature map of the lth layer when the defect image is input into the feature extraction network, φ l (y) u,v,w represents the pixel at coordinate (u, v, w) of the output feature map of layer l when the defect composite image is input into the feature extraction network; The total loss function is: L=L rec +L GAN (G,D)+L con 。 10. A defect image generation system based on attribute semantic separation, characterized in that: include: A data acquisition module, configured to acquire a defect image and an image to be generated, as well as defect annotation data of one or more attributes of the defect image; A mask module, configured to perform a mask operation on the defect image using the defect annotation data of each attribute to obtain a defect source image of each attribute; A skeleton information acquisition module, used to acquire the skeleton information of the image to be generated; A defect image synthesis module, configured to input the defect source image and the skeleton information of the image to be generated into a trained defect image generation model to obtain a defect synthesis image; The defect image generation model includes a first branch and a second branch, the first branch is a feature extraction network, the second branch is an attribute feature image generation network, and the defect image synthesis module includes an attribute feature extraction submodule and a synthesis submodule; The attribute feature extraction submodule is used to input the defect source image of each attribute into the first branch respectively, perform feature extraction through the feature extraction network, obtain the corresponding defect attribute feature map, and fuse the defect attribute feature maps to obtain a multi-attribute defect feature map; The synthesis submodule is used to input the skeleton information of the image to be generated and the multi-attribute defect feature map into the second branch, perform position feature encoding on the skeleton information through the attribute feature image generation network to obtain a skeleton feature map, perform feature fusion based on attribute feature migration on the skeleton feature map and the multi-attribute defect feature map to obtain a fused feature map, and use the fused feature map to reconstruct the image to obtain a defect composite image.

11. A computer-readable storage medium, characterized in that The medium stores a program, which can be executed by a processor to implement the defect image generation method according to any one of claims 1 to 9.

Images