Image generation method based on heterogeneous feature fusion, readable storage medium

Abstract

Image generation method based on heterogeneous feature fusion, readable storage medium, the method comprises: acquiring a source image and a plurality of first feature maps of different resolutions of each target image; selecting the first feature maps of the source image and each target image with the same resolution as the first to-be-fused feature maps, performing a first feature fusion step to obtain a fused feature map; selecting the first feature maps of the source image and each target image with the same resolution as the second to-be-fused feature maps in a predetermined order, and performing a second feature fusion step; in the first feature fusion step, selecting pixel points from each first to-be-fused feature map to form the fused feature map according to a preset mask map, and in the second feature fusion step, performing the first feature fusion step again by taking the feature fusion of each second to-be-fused feature map and the fused feature map as the first to-be-fused feature map. The method can fuse features from different images at multiple scales, and can synthesize a more realistic image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image generation method based on heterogeneous feature fusion and a readable storage medium. Background Technology

[0002] In recent years, deep learning has become a focus of attention in various fields both domestically and internationally. Deep learning includes two categories: supervised learning and unsupervised learning. In the field of computer vision, supervised learning refers to training neural networks using images and corresponding annotations to perform tasks such as classification, object detection, and semantic segmentation. Unsupervised learning only requires images and does not require additional annotations for training.

[0003] In most scenarios, supervised learning performs better than unsupervised learning. However, supervised learning requires a large number of images and annotation information, which greatly limits the verification and implementation of deep learning algorithms. This is because: on the one hand, annotating images is time-consuming and labor-intensive; on the other hand, for industrial scenarios such as defect detection, samples are usually scarce, especially defect samples. Therefore, the number of defect images that can be obtained is small, making it difficult to collect a sufficient number of samples.

[0004] To address the shortage of defect image samples in industrial defect detection, current methods primarily involve fusing various defects using defect cropping and stitching techniques to generate new defect image samples. However, the defects fused using this method often differ significantly from the real samples, exhibiting noticeable edge discrepancies. For instance, existing image synthesis techniques based on matting algorithms suffer from inconsistencies between image patches from different images, resulting in synthesized images with pixel discontinuities, sharp edges, and inconsistent lighting. Summary of the Invention

[0005] The main technical problem solved by this invention is that the images generated by existing image synthesis methods differ significantly from the real images.

[0006] According to the first aspect, one embodiment provides an image generation method based on heterogeneous feature fusion, comprising:

[0007] Acquire the source image, the target image, and the mask image, wherein the target image can be one or more;

[0008] Feature extraction is performed on the source image and each of the target images respectively to obtain a first feature map group of the source image and a first feature map group of each of the target images, wherein the first feature map group includes multiple first feature maps of different resolutions;

[0009] Select the first feature maps with the same resolution from the first feature map group of the source image and each of the target images as the first feature maps to be fused, and perform the first feature fusion step to obtain the fused feature map;

[0010] In the first feature fusion step, for each pixel point corresponding to a position on a first feature map to be fused, a pixel point of one of the first feature maps to be fused is selected according to the mask image as the pixel point at the corresponding position in the fused feature map. The fused feature map is composed of the pixel points selected at each position, wherein the mask image is used to identify the source of each pixel point in the fused feature map.

[0011] The first feature maps with the same resolution in the first feature map group of the source image and each target image are selected in a predetermined order as the second feature maps to be fused, and the second feature fusion step is performed until the first feature maps in the first feature map group of the source image and each target image have all undergone the second feature fusion step.

[0012] The second feature fusion step involves fusing each second feature map to be fused with the fused feature map, using the image obtained by feature fusion as the first feature map to be fused, and then performing the first feature fusion step. A new fused feature map is obtained from the result of the first feature fusion step.

[0013] The final fused feature map is then processed through convolution and a nonlinear activation function to obtain the synthesized image.

[0014] According to the second aspect, one embodiment provides an image generation method based on heterogeneous feature fusion, comprising:

[0015] Acquire the source image, the target image, and the mask image, wherein the target image can be one or more;

[0016] Feature extraction is performed on the source image and each of the target images respectively to obtain a first feature map group of the source image and a first feature map group of each of the target images, wherein the first feature map group includes multiple first feature maps of different resolutions;

[0017] First feature maps with the same resolution are selected from the first feature map group of the source image and each target image. The first feature map of the selected target image and the mask image are subjected to the same geometric transformation. Then, the first feature maps of the selected source image and the target image are used as the first feature maps to be fused, and the first feature fusion step is performed to obtain the fused feature map.

[0018] In the first feature fusion step, for each pixel point corresponding to a position on a first feature map to be fused, a pixel point of one of the first feature maps to be fused is selected according to the mask image as the pixel point at the corresponding position in the fused feature map. The fused feature map is composed of the pixel points selected at each position, wherein the mask image is used to identify the source of each pixel point in the fused feature map.

[0019] In a predetermined order, the first feature maps with the same resolution in the first feature map group of the source image and each of the target images are used as the second feature maps to be fused, and the second feature fusion step is performed until the first feature maps in the first feature map group of the source image and each of the target images have all undergone the second feature fusion step.

[0020] The second feature fusion step involves fusing each second feature map to be fused with the fused feature map, using the image obtained by feature fusion as the first feature map to be fused, and then performing the first feature fusion step. A new fused feature map is obtained from the result of the first feature fusion step.

[0021] The final fused feature map is then processed through convolution and a nonlinear activation function to obtain the synthesized image.

[0022] In one embodiment, the step of extracting features from the source image and each of the target images to obtain a first feature map group for the source image and a first feature map group for each of the target images includes:

[0023] The source image and each target image are respectively input into the feature extraction network to obtain N feature maps of the source image and each target image with decreasing resolution. These feature maps are used as the first feature maps to form the first feature map group.

[0024] The feature extraction network includes N downsampling modules, each of which generates a feature map, where N is a positive integer and N≥2.

[0025] In one embodiment, each of the downsampling modules includes a downsampling layer and a plurality of convolutional-activation layers.

[0026] In one embodiment, the step of selecting the first feature maps with the same resolution from the first feature map group of the source image and each target image as the first feature map to be fused, and performing the first feature fusion step to obtain the fused feature map, specifically involves: selecting the Nth first feature map of the source image and each target image as the first feature map to be fused, and performing the first feature fusion step to obtain the fused feature map;

[0027] The step of sequentially using first feature maps of the same resolution in the first feature map group of the source image and each target image as second feature maps to be fused, and performing the second feature fusion step, specifically involves: starting from the Nth first feature map in the first feature map group of the source image and each target image up to the 1st first feature map, sequentially using the first feature maps in the first feature map group of the source image and each target image as second feature maps to be fused, and performing the second feature fusion step.

[0028] Specifically, obtaining a new fused feature map from the result of the first feature fusion step involves inputting the result of the first feature fusion step into the upsampling module of the decoder network to obtain a new fused feature map.

[0029] The encoder network corresponding to the decoder network is the feature extraction network. The decoder network includes N upsampling modules. The input of the nth upsampling module is the result of the first feature fusion step when the N+1-nth first feature map performs the second feature fusion step. The output is a feature map with the same resolution as the Nnth first feature map. The output of the Nth upsampling module is a feature map with the same resolution as the source image and the target image, where n = 1, 2, ... N.

[0030] In one embodiment, each of the upsampling modules includes an upsampling layer and several convolutional-activation layers.

[0031] In one embodiment, the encoder network and the decoder network are trained in the following manner:

[0032] A generator is composed of the encoder network and the decoder network, and a generative adversarial network is formed with a pre-built discriminator. The generator and the discriminator are trained alternately to obtain the parameters of the generator and the discriminator.

[0033] The loss function for training the generator is:

[0034]

[0035] p real Let G(p) represent the training image, l represent the mask image, and G(p) represent the training image. real (l) represents the synthesized image obtained by inputting the source image, the geometrically transformed training image, and the mask image into the generator. o (G(p real ,l)) represents the output of the discriminator at layer o when the synthesized image is input into the discriminator, D o (p real) represents the output of the discriminator's layer o when the training image is input into the discriminator, where o represents the set of network layers of the selected discriminator;

[0036] The loss function for training the discriminator is:

[0037] L2=-log(D(p real ))-log(1-D(G(p real ,l))),

[0038] D(p real ) represents the result obtained by inputting the training image into the discriminator, D(G(p) real ,l)) represents the result obtained by inputting the synthesized image into the discriminator.

[0039] In one embodiment, the step of fusing each second feature map to be fused with the fused feature map includes:

[0040] The fused feature map is instance-normalized according to the following formula:

[0041]

[0042] Where a i,j,k To fuse the pixel value of the i-th row, j-th column, k-th channel in the feature map, m k To calculate the mean of the k-th channel of the fused feature map, s k The standard deviation of the k-th channel of the fused feature map is given by ε, where ε is a preset error value and b is the standard deviation of the k-th channel. i,j,k The pixel value of the i-th row, j-th column, and k-th channel in the fused feature map after instance normalization;

[0043] Obtain the mean and standard deviation of each channel of each second feature map to be fused. Multiply each channel of the fused feature map after instance standardization by the standard deviation of the corresponding channel of each second feature map to be fused, and then add it to the mean of the corresponding channel.

[0044] In one embodiment, the pixel value of each pixel in the mask image is used to identify the source of the pixel at the corresponding position in the fused feature map; the step of selecting one pixel of the first feature map to be fused as the pixel at the corresponding position in the fused feature map based on the mask image for each pixel at the position in each first feature map to be fused, and forming the fused feature map from the selected pixels at each position, includes: scaling the mask image to the same resolution as the first feature map to be fused; for each pixel in the scaled mask image, selecting one pixel at the corresponding position in the first feature map to be fused based on its pixel value, and forming the fused feature map from the selected pixels.

[0045] According to a third aspect, one embodiment provides a computer-readable storage medium storing a program that can be executed by a processor to implement the image generation method as described in the first or second aspect above.

[0046] According to the image generation method based on heterogeneous feature fusion in the above embodiments, firstly, features are extracted from the source image and each target image to obtain multiple first feature maps of different resolutions for the source image and each target image, thereby obtaining multi-scale feature maps for the source image and each target image. Then, first feature maps of the same resolution as the source image and each target image are selected as first feature maps to be fused, and a first feature fusion step is performed to obtain a fused feature map. Then, first feature maps of the same resolution as the source image and each target image are selected in a predetermined order as second feature maps to be fused, and a second feature fusion step is performed. In the first feature fusion step, pixels are selected from each of the first feature maps to be fused according to a preset mask to form a fused feature map. In the second feature fusion step, each of the second feature maps to be fused is fused with the fused feature map respectively. The image obtained by feature fusion is used as the first feature map to be fused and the first feature fusion step is performed. Thus, features from different images can be fully fused, and multi-scale feature fusion can be achieved. Compared with traditional cropping and splicing techniques, the generated image is more realistic and can synthesize images with more continuous pixels, smoother edges, and more continuous illumination. Attached Figure Description

[0047] Figure 1 This is a real defect image of one embodiment;

[0048] Figure 2 To utilize traditional image matting algorithms Figure 1 The generated defect image;

[0049] Figure 3 To utilize the image generation method of the present invention Figure 1 The generated defect image;

[0050] Figure 4 A flowchart illustrating an image generation method based on heterogeneous feature fusion according to one embodiment;

[0051] Figure 5 for Figure 4 A schematic diagram illustrating the execution process of the method shown;

[0052] Figure 6 This refers to the source image, target image, and mask image in one embodiment;

[0053] Figure 7 This is a schematic diagram of the structure of a downsampling module according to one embodiment;

[0054] Figure 8 This is a schematic diagram illustrating the process of feature fusion between a first feature map and a fused feature map in one embodiment.

[0055] Figure 9 This is a schematic diagram of the structure of an upsampling module according to one embodiment;

[0056] Figure 10 This is a flowchart illustrating the training process of a GAN network according to one embodiment.

[0057] Figure 11 A schematic diagram for constructing the loss function;

[0058] Figure 12 This is a schematic diagram illustrating the effect of synthesizing damaged defects using the image generation method of the present invention;

[0059] Figure 13 This is a schematic diagram illustrating the effect of synthesizing hole defects using the image generation method of the present invention;

[0060] Figure 14 This is a schematic diagram illustrating the effect of synthesizing printing defects using the image generation method of the present invention;

[0061] Figure 15 This is a schematic diagram illustrating the effect of synthesizing scratch defects using the image generation method of the present invention. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0063] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0064] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0065] Traditional image matting algorithms stitch pixels together, which can easily result in discontinuous pixels, sharp edges, and other distortions. To address this, this invention proposes an image generation method based on heterogeneous feature fusion, capable of synthesizing images with more continuous pixels, smoother edges, and more continuous illumination. This image generation method is flexible and pre-trainable, allowing target features to be transferred to any image while simultaneously generating corresponding annotation information. This information can be used to synthesize training images and corresponding annotations for supervised learning, and can be applied to augmenting deep learning datasets.

[0066] The effectiveness of the image generation method based on heterogeneous feature fusion provided by this invention can be seen in [reference needed]. Figures 1-3 . Figure 1 This is a realistic image of a defect, showing that there are no sharp edges in the defective area. Figure 2 To utilize traditional image matting algorithms Figure 1 The generated defect image shows that the defective part usually has sharp edges, which leads to distortion. Figure 3 To utilize the image generation method of the present invention Figure 1 The generated defect image shows that the synthesized image has no sharp edges in the defect area, making it closer to the real image.

[0067] The image generation method based on heterogeneous feature fusion of the present invention will be described in detail below. Please refer to [link / reference]. Figure 4 In one embodiment, the method includes steps 110 to 170, which are described in detail below.

[0068] Step 110: Obtain the source image, the target image, and the mask image, wherein the target image is one or more.

[0069] The source image and target image can be any images. Dividing images into source and target images is merely for differentiation within specific image synthesis tasks. In some image synthesis tasks, the target image typically refers to an image containing features of interest; these features are transferred to the source image and fused into a new image. For example, in the synthesis of defective images, a real defective image containing defective features can be used as the target image, and a defect-free image as the source image. After processing by the image generation method of this invention, the defective features are transferred to the source image, thus generating a new defective image, such as... Figures 12-15 As shown.

[0070] The mask is preset by the user and determines which images and which pixel locations are selected from the two or more images to be fused to form the fused image. Generally, the mask should be set so that the target features in the target image are reflected in the final synthesized image.

[0071] Step 120: Perform feature extraction on the source image and each target image respectively to obtain the first feature map group of the source image and the first feature map group of each target image, wherein the first feature map group includes multiple first feature maps of different resolutions.

[0072] The number of first feature maps can be determined according to actual needs and is not limited here. For feature extraction, please refer to [reference needed]. Figure 5 This can be achieved using a pre-constructed feature extraction network comprising N downsampling modules, where N is a positive integer and N≥2. The source image and each target image are input into this feature extraction network, and after N consecutive downsampling operations by the N downsampling modules, N feature maps of decreasing resolution are obtained for both the source image and each target image. Each downsampling module generates one feature map, and these feature maps are used as the first feature map group to form the first feature map group.

[0073] Step 130: Select the first feature maps with the same resolution from the first feature map group of the source image and each target image, and execute step 140 to obtain the fused feature map.

[0074] In one embodiment, the selected first feature map can be directly used as the first feature map to be fused to perform step 140. In another embodiment, such as... Figure 5 As shown, a geometric transformation T (referred to as "T-transform") can be preset to perform the T-transform on the first feature map and mask map of the selected target image. Then, the first feature map of the selected source image and the first feature map of the target image after the T-transform are used as the first feature map to be fused to execute step 140. By performing different T-transforms on the first feature map of the target image, various synthetic images of different forms can be generated, further enriching the content of the image set.

[0075] T-transform can include any one or more of the following: angular rotation, translation, flipping, scaling, and cropping, and the resolution of the image remains unchanged before and after the transformation. For example, if the T-transform is to reduce the image to 0.5 times its original size, then the image is reduced to 0.5 times its original size, centered, and padded with 0 pixels around it to make it have the same resolution as the original image.

[0076] Step 140: First feature fusion step, specifically, for each pixel point corresponding to a position on each first feature map to be fused, select one of the pixels in the first feature map to be fused according to the mask image, and use it as the pixel point at the corresponding position in the fused feature map. The fused feature map is composed of the pixels selected at each position, wherein the mask image is used to identify the source of each pixel point in the fused feature map.

[0077] In one embodiment, the pixel value of each pixel in the mask image is used to identify the source of the pixel at the corresponding position in the fused feature map. The number of channels in the mask image is related to the number of images, that is, the number of channels is the same as the number of the first feature maps to be fused, and each channel corresponds to one first feature map to be fused; the pixel value of the mask image is an array composed of 0 and 1, where the pixel value of each channel is 0 or 1, and only one channel has a pixel value of 1, representing that the first feature map to be fused corresponding to that channel is selected.

[0078] In this embodiment, the first feature fusion step specifically involves: scaling the mask image to the same resolution as the first feature image to be fused; for each pixel in the scaled mask image, selecting a pixel at the corresponding position in the first feature image to be fused based on its pixel value, and using this pixel as the pixel at the corresponding position in the fused feature image; and forming the fused feature image from the selected pixels.

[0079] Taking two first feature maps to be fused as an example, the mask image has 2 channels and pixel values ​​of [1,0] or [0,1]. During the first feature fusion step, the mask image is first scaled to the same resolution as the first feature map to be fused. Then, for each pixel (i,j) in the mask image, if its pixel value is [1,0], the pixel at the corresponding position (i,j) in the first first feature map to be fused is selected as the pixel at the corresponding position (i,j) in the fused feature map; if its pixel value is [0,1], the pixel at the corresponding position (i,j) in the second first feature map to be fused is selected as the pixel at the corresponding position (i,j) in the fused feature map. The selected pixels form the fused feature map, where i is the horizontal coordinate and j is the vertical coordinate. The resolution of the fused feature map is the same as that of the first feature map to be fused. Figure 1 The first feature fusion step can be expressed by the formula: R(l,x1,…,x) m ), where l is an m-channel mask, x1,…,x m Let m be the first feature maps to be fused, where m is a positive integer and m≥2.

[0080] Step 150: Select the first feature maps with the same resolution from the first feature map group of the source image and each target image in a predetermined order as the second feature maps to be fused, and execute step 160 until the first feature maps in the first feature map group of the source image and each target image have all been processed by step 160.

[0081] Step 160: Second feature fusion step, specifically, each second feature map to be fused is fused with the fused feature map, the image obtained by feature fusion is used as the first feature map to be fused and the first feature fusion step is performed, and a new fused feature map is obtained from the result of the first feature fusion step.

[0082] The present invention does not limit the order of the first feature maps for performing the second feature fusion step, and it can be determined according to actual needs and operational convenience in practical applications.

[0083] In one embodiment, the image generation method of the present invention can be implemented based on a neural network with an autoencoder structure, wherein the encoder performs feature extraction, and the decoder performs image reconstruction, restoring the feature map to the resolution of the original image size. The encoder can be the feature extraction network described above, which includes N downsampling modules, and the corresponding decoder can be composed of N upsampling modules. In this embodiment, steps 120 to 160 are as follows:

[0084] Step 120: Input the source image and each target image into the feature extraction network respectively to obtain N feature maps of the source image and each target image with decreasing resolution. Use these feature maps as the first feature map group. Since the source image and target image have the same resolution, the first feature map obtained by each downsampling module of the feature extraction network for the source image and target image also has the same resolution.

[0085] Step 130: Select the Nth first feature map (i.e. the first feature map with the smallest resolution) of the source image and each target image as the first feature map to be fused, and execute step 140 to obtain the fused feature map.

[0086] Step 140: First feature fusion step, specifically, for each pixel point corresponding to a position on each first feature map to be fused, select one of the pixels in the first feature map to be fused according to the mask image, and use it as the pixel point at the corresponding position in the fused feature map. The fused feature map is composed of the pixels selected at each position, wherein the mask image is used to identify the source of each pixel point in the fused feature map.

[0087] Step 150: Starting from the Nth first feature map of the first feature map group of the source image and each target image, and ending at the 1st first feature map, sequentially use the first feature maps in the first feature map group of the source image and each target image as the second feature maps to be fused, and execute step 160. That is, select the first feature maps in ascending order of resolution and execute step 160.

[0088] Step 160: Second feature fusion step, specifically, each second feature map to be fused is fused with the fused feature map, the image obtained by feature fusion is used as the first feature map to be fused to perform the first feature fusion step, and the result of the first feature fusion step is input into the upsampling module of the decoder network to obtain a new fused feature map.

[0089] The input of the nth upsampling module of the decoder network is the result of the first feature fusion step when the second feature fusion step is performed on the (N+1-n)th first feature map, and the output is a feature map with the same resolution as the Nnth first feature map. The output of the Nth upsampling module is a feature map with the same resolution as the source image and the target image, where n = 1, 2, ... N.

[0090] The decoder network alternately performs feature fusion and upsampling, gradually increasing the resolution of the feature maps to restore them to the original image size, resulting in a feature map that fuses features from multiple scales. For example, if the first feature map group contains three first feature maps with decreasing resolution, the decoder includes three upsampling modules. First, the third first feature map undergoes a second feature fusion step, and the result of the first feature fusion step is input into the first upsampling module for upsampling, obtaining a new fused feature map with the same resolution as the second first feature map. Then, the second first feature map undergoes a second feature fusion step, and the result of the first feature fusion step is input into the second upsampling module for upsampling, obtaining a new fused feature map with the same resolution as the first first feature map. Finally, the first first feature map undergoes a second feature fusion step, and the result of the first feature fusion step is input into the third upsampling module for upsampling, obtaining a fused feature map with the same resolution as both the source and target images.

[0091] For feature fusion between the second feature map to be fused and the fused feature map, existing feature fusion methods can be used. In one embodiment of the present invention, the fused feature map is first standardized by instance, then each channel of the standardized fused feature map is multiplied by the standard deviation of the corresponding channel of each second feature map to be fused, and then added to the mean of the corresponding channel to complete the feature fusion between the second feature map to be fused and the fused feature map.

[0092] Specifically, firstly, the mean and standard deviation of each channel of the second feature map to be fused and the fused feature map are calculated. For example, for a feature map with 256 channels and a resolution of 128×128, calculating the mean and standard deviation of each channel will yield a vector of length 256 for both the mean and standard deviation. The calculation method for the mean and standard deviation of the feature map is as follows:

[0093]

[0094]

[0095] Where M k S is the mean of the k-th channel of the feature map. k Let H be the standard deviation of the k-th channel of the feature map, H be the height of the feature map, W be the width of the feature map, and A be the standard deviation of the k-th channel of the feature map. i,j,k Let be the pixel value of the i-th row, j-th column, and k-th channel in the feature map.

[0096] After obtaining the mean and standard deviation of each channel of the second feature map to be fused and the fused feature map using the above calculation method, the fused feature map is instantiated so that the mean of each channel is 0 and the standard deviation is 1, which can be expressed by the formula:

[0097]

[0098] Where a i,j,k To fuse the pixel value of the i-th row, j-th column, k-th channel in the feature map, m k To calculate the mean of the k-th channel of the fused feature map, s k The standard deviation of the k-th channel of the fused feature map is given by ε, which is a preset error value. In one example, ε = 10. -7 b i,j,k The pixel value of the i-th row, j-th column, and k-th channel in the fused feature map after the instance is normalized.

[0099] Next, each channel of the instance-normalized fused feature map is multiplied by the standard deviation of the corresponding channel of each second feature map to be fused, and then added to the mean of the corresponding channel to complete the feature fusion between the second feature map to be fused and the fused feature map. For example, if both the instance-normalized fused feature map and the second feature map to be fused have 256 channels, then each pixel of the first channel of the instance-normalized fused feature map is multiplied by the standard deviation of the first channel of the second feature map to be fused, and then added to the mean of the first channel of the second feature map to be fused; each pixel of the second channel of the instance-normalized fused feature map is multiplied by the standard deviation of the second channel of the second feature map to be fused, and then added to the mean of the second channel of the second feature map to be fused, and so on, to complete the calculation for all 256 channels. The number of images after feature fusion is the same as the number of images of the second feature map to be fused.

[0100] Step 170: The final fused feature map is processed by convolution and nonlinear activation functions to obtain the synthesized image.

[0101] The following is combined Figure 5This paper illustrates the image generation method based on heterogeneous feature fusion provided by this invention through a specific example, implemented using a Generative Adversarial Network (GAN). A GAN is a deep learning model consisting of two parts: a generator and a discriminator. The generator's task is to produce an output that approximates a realistic image, while the discriminator's task is to determine whether the input image is generated by the generator or a real image. During training, the generator and discriminator are trained alternately to obtain a generator whose output image closely approximates a realistic image. When generating an image, only the generator needs to be used to synthesize the image.

[0102] like Figure 5 As shown, the generator can consist of an encoder and a decoder. The encoder includes three downsampling modules, and the decoder includes three upsampling modules. The discriminator is a binary classification network responsible for determining whether the input image is a real image or an image generated by the generator. It can employ classic classification network structures such as ResNet50 or VGG.

[0103] The following example uses a source image and a target image as input to illustrate the entire workflow.

[0104] First, obtain the source image, target image, and mask image. Figure 6 In one embodiment, a source image, a target image, and a mask are used, all with the same resolution. The source image is a defect-free image, the target image is a defective image, and the mask marks the locations of defects in the target image. That is, during feature fusion, the defective portions of the target image are retained, while the remaining portions are selected from pixels in the source image. The goal of image synthesis is to move the portion of the target image marked by the mask to any position in the source image.

[0105] Please refer to Figure 5 The source and target images are respectively processed by the encoder for feature extraction, resulting in three first feature maps of decreasing resolution, forming a first feature map group. The first feature map group of the source image is denoted as (f 1,1 ,f 1,2 ,f 1,3 ), where f 1,1 f 1,2 f 1,3 As the resolution decreases, the first feature set of the target image is denoted as (f 1, ′1,f 1, ′2,f 1, ′3), where f 1, ′1、f 1, ′2、f 1, The resolution decreases by 3'. The mean and standard deviation of each first feature map are calculated separately, forming a first mean group and a first standard deviation group. The first feature map group (f)1,1 ,f 1,2 ,f 1,3 The first mean group corresponding to (μ) is denoted as (μ) 1,1 ,μ 1,2 ,μ 1,3 The first standard deviation group is denoted as (σ). 1,1 ,σ 1,2 ,σ 1,3 ), the first feature map group (f 1, ′1,f 1, ′2,f 1, The first mean group corresponding to (μ1′) is denoted as (μ1′). ,1 ,μ1′ ,2 ,μ1′ ,3 The first standard deviation group is denoted as (σ1′). ,1 ,σ1′ ,2 ,σ1′ ,3 ).

[0106] Each downsampling module of the encoder outputs a first feature map. The downsampling module reduces the resolution of the input feature map. Please refer to [reference needed]. Figure 7 In one embodiment, each downsampling module includes a downsampling layer and several convolutional-activation layers, where convolutional-activation layers refer to connected convolutional and activation layers. Specifically, the downsampling layer can be used for max pooling or average pooling, the convolutional layers are used for convolutional processing, and the activation layers are used for non-linear activation function processing. For example, the downsampling layer here can be a max pooling layer with a pooling window of 2×2 and a stride of 2×2. After each downsampling module, the length and width of the feature map are reduced to 0.5 times their original size.

[0107] For the first feature map f′ 1,3 The feature map is obtained by performing a preset T-transform. Scale the mask image l to match the first feature image f′. 1,3 The same resolution yields l 1,3 , will l 1,3 Obtained through T-transform The first feature map f′ 1,3 and feature map As the first feature map to be fused, the mask image is used. The first feature fusion step yields the fused feature map f2, which can be represented by the following equation:

[0108]

[0109] Select the first feature map f 1,3 and f′ 1,3The second feature map to be fused is used as the second feature map for the second feature fusion step. Please refer to [link / reference] for details. Figure 8 First, the fused feature map f2 is instance-normalized, and then the instance-normalized fused feature map f2 is multiplied by the first feature map f. 1,3 Standard deviation σ 1,3 In addition to the first feature map f 1,3 mean μ 1,3 To obtain a statistically close first feature map f. 1,3 The fourth feature map f 4,3 Similarly, performing the same operation yields a statistically close feature map f′. 1,3 The fourth feature map f′ 4,3 The mask image l is scaled to the same resolution as the fused feature map f2 to obtain l2, and the fourth feature map f is... 4,3 and the fourth feature map f′ 4,3 As the first feature map to be fused, the fifth feature map f is obtained by performing the first feature fusion step using the mask map l2. 5,1 The fifth feature map f 5,1 The first upsampling module of the input decoder obtains the new fused feature map f. 3,1 f 3,1 The resolution and the first feature map f 1,2 f′ 1,2 same.

[0110] Subsequently, the first feature map f is sequentially... 1,2 f′ 1,2 As the second feature map to be fused, the first feature map f 1,1 f′ 1,1 The second feature map to be fused is used as the second feature map to be fused, and the second feature fusion step is performed to obtain the final fused feature map f. 3,3 f 3,3 The resolution is the same as the source and target images. The fused feature map f 3,3 The synthesized image is obtained through convolution and nonlinear activation functions.

[0111] The following formula represents the process of fusing the fused feature map with each of the second feature maps to be fused to obtain the fifth feature map:

[0112] V(x1,x2,μ,σ,μ′,σ′)=R(x2,InstanceNorm(x1)×σ+μ,InstanceNorm(x1)×σ′+μ′),

[0113] Where V(x1,x2,μ,σ,μ′,σ′) represents the fifth feature map, x1 represents the fused feature map, x2 represents the mask map, InstanceNorm() represents instance normalization, σ and μ represent the standard deviation and mean of the first feature map of the source image, respectively, and σ′ and μ′ represent the standard deviation and mean of the first feature map of the target image, respectively.

[0114] but

[0115] f 3,1 =Up(V(f2,l2,μ) 1,3 ,σ 1,3 ,μ′ 1,3 ,σ′ 1,3 )),

[0116] f 3,2 =Up(V(f) 3,1 ,l 3,1 ,μ 1,2 ,σ 1,2 ,μ1′ ,2 ,σ1′ ,2 )),

[0117] f 3,3 =Up(V(f) 3,2 ,l 3,2 ,μ 1,1 ,σ 1,1 ,μ1′ ,1 ,σ1′ ,1 )),

[0118] Where Up() represents upsampling, l 3,1 To scale the mask image l to the fused feature map f 3,1 For the same resolution size, l 3,2 To scale the mask image l to the fused feature map f 3,2 Same resolution size.

[0119] The decoder's upsampling module can amplify the resolution of the feature maps in the input. Please refer to [reference needed]. Figure 9 In one embodiment, each upsampling module includes an upsampling layer and several convolutional-activation layers. For example, the upsampling layer here can enlarge both the length and width of the feature map to twice its original size.

[0120] The training process of the entire GAN network will be described below.

[0121] Please refer to Figure 10 First, obtain a training image p. real As the source image, for the training image p realA preset T-transform is performed to obtain the target image. Both the source and target images are processed by an encoder for feature extraction, yielding the first feature map group (f) of the source image. 1,1 ,f 1,2 ,f 1,3 The first feature map group (f) of the target image 1, ′1,f 1, ′2,f 1, ′3). Calculate the mean and standard deviation of each first feature map, forming a first mean group and a first standard deviation group, and a first feature map group (f 1,1 ,f 1,2 ,f 1,3 The first mean group corresponding to (μ) is denoted as (μ) 1,1 ,μ 1,2 ,μ 1,3 The first standard deviation group is denoted as (σ). 1,1 ,σ 1,2 ,σ 1,3 ), the first feature map group (f 1, ′1,f 1, ′2,f 1, The first mean group corresponding to (μ1′) is denoted as (μ1′). ,1 ,μ1′ ,2 ,μ1′ ,3 The first standard deviation group is denoted as (σ1′). ,1 ,σ1′ ,2 ,σ1′ ,3 ).

[0122] For the first feature map f 1, The feature map f is obtained by performing an inverse transform of T on ′3. 1, ″3, Scale the mask image l to match the first feature image f 1, The same resolution as ′3 yields l 1,3 The first feature map f 1,3 and feature map f 1, "3 is used as the first feature map to be fused, using mask map l" 1,3 The first feature fusion step yields the fused feature map f2, which can be represented by the following equation:

[0123] f2=R(l 1,3 ,f 1,3 ,f 1, ″3).

[0124] The inverse transform of T is the opposite of the transform of T, and the resolution remains unchanged before and after the transform. For example, the transform of T reduces the length and width of the image to 0.5 times its own size and fills the edge pixels, while the inverse transform of T enlarges the length and width of the image to twice its own size and removes the excess edge pixels.

[0125] Then, the final fused feature map f is obtained according to the following equation. 3,3 , fusion feature map f 3,3 The synthesized image is obtained after convolution and nonlinear activation functions:

[0126] f 3,1 =Up(V(f2,l2,μ) 1,3 ,σ 1,3 ,μ1′ ,3 ,σ1′ ,3 )),

[0127] f 3,2 =Up(V(f) 3,1 ,l 3,1 ,μ 1,2 ,σ 1,2 ,μ1′ ,2 ,σ1′ ,2 )),

[0128] f 3,3 =Up(V(f) 3,2 ,l 3,2 ,μ 1,1 ,σ 1,1 ,μ1′ ,1 ,σ1′ ,1 )).

[0129] A loss function is constructed based on the synthesized and training images. The generator and discriminator are trained alternately to obtain their parameters. The construction of the loss function can be found in [reference needed]. Figure 11 The loss function for training the generator is:

[0130]

[0131] G(p real (l) represents the synthesized image obtained by using the training image as the source image, the geometrically transformed training image as the target image, and inputting the source image, target image, and mask image into the generator. o (G(p real ,l)) represents the output of the discriminator's o-th layer when the synthesized image is input into the discriminator, D o (p real The expression G(p) represents the output of the discriminator's layer o when the training image is input into the discriminator, where o represents the set of network layers selected for the discriminator. The training objective of the loss function L1 is: the output G(p) of the generator. real ,l) and real image p real The pixel values ​​are close, and the features of the real image extracted by the discriminator are close to the features of the synthetic image.

[0132] The loss function for training the discriminator is:

[0133] L2=-log(D(p real ))-log(1-D(G(p real ,l))),

[0134] D(p real D(G(p)) represents the result obtained by inputting the training image into the discriminator. real ,l)) represents the result obtained by inputting the synthetic image into the discriminator. The training objective of the loss function L2 is: the discriminator outputs 1 when a real image is input and outputs 0 when a synthetic image is input, so that the discriminator can distinguish between real and synthetic images.

[0135] The generator and discriminator are trained alternately based on the above loss function. The parameters of the generator and discriminator are updated through backpropagation until the discriminator can no longer effectively distinguish between real and synthetic images. That is, when the discriminator outputs a value of about 0.5 when inputting either a real or synthetic image, the training is complete.

[0136] Figures 12-15 The effectiveness of the image generation method based on heterogeneous feature fusion of the present invention in synthesizing defective images is presented, wherein... Figure 12 To achieve the effect of synthesizing damage defects, Figure 13 To achieve the effect of synthesizing pore defects, Figure 14 To achieve the effect of synthesizing printing defects, Figure 15 To achieve the effect of synthesizing scratch defects, the synthesized defect images can be used as defect sample images to expand the deep learning dataset used for defect detection training, compensating for the limited number of defect sample images in reality. Since the mask image marks the defect location, it can be used as the corresponding annotation information. If a T-transform is performed, the T-transformed mask image is used as the annotation information.

[0137] The image generation method based on heterogeneous feature fusion provided by this invention first extracts features from the source image and each target image to obtain multiple first feature maps of different resolutions for the source image and each target image, thus obtaining multi-scale feature maps for the source image and each target image. Then, the first feature maps of the same resolution for the source image and each target image are selected as the first feature maps to be fused, and the first feature fusion step is performed to obtain the fused feature map. Then, the first feature maps of the same resolution for the source image and each target image are selected in a predetermined order as the second feature maps to be fused, and the second feature fusion step is performed. In the first feature fusion step, pixels are selected from each of the first feature maps to be fused according to a preset mask to form the fused feature map. In the second feature fusion step, each of the second feature maps to be fused is fused with the fused feature map respectively. The image obtained by feature fusion is used as the first feature map to be fused and the first feature fusion step is performed. Thus, features from different images can be fully fused, and multi-scale feature fusion can be achieved. Compared with traditional cropping and splicing techniques, the generated image is more realistic and can synthesize images with more continuous pixels, smoother edges, and more continuous illumination.

[0138] The image generation method based on heterogeneous feature fusion of the present invention is a general image fusion technology that is not related to specific data. When the image generation method of the present invention is implemented by constructing a neural network, after training and convergence on a large general dataset, it can be used for the synthesis of other types of data without retraining.

[0139] One embodiment of the present invention provides a lightweight image synthesis network based on GAN, which has fewer network weights, runs fast, and can quickly synthesize deep learning training data.

[0140] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0141] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. An image generation method based on heterogeneous feature fusion, characterized in that, include: Acquire the source image, the target image, and the mask image, wherein the target image can be one or more; Feature extraction is performed on the source image and each of the target images respectively to obtain a first feature map group of the source image and a first feature map group of each of the target images, wherein the first feature map group includes multiple first feature maps of different resolutions; Select the first feature maps with the same resolution from the first feature map group of the source image and each of the target images as the first feature maps to be fused, and perform the first feature fusion step to obtain the fused feature map; In the first feature fusion step, for each pixel point corresponding to a position on a first feature map to be fused, a pixel point of one of the first feature maps to be fused is selected according to the mask image as the pixel point at the corresponding position in the fused feature map. The fused feature map is composed of the pixel points selected at each position, wherein the mask image is used to identify the source of each pixel point in the fused feature map. The first feature maps with the same resolution in the first feature map group of the source image and each target image are selected in a predetermined order as the second feature maps to be fused, and the second feature fusion step is performed until the first feature maps in the first feature map group of the source image and each target image have all undergone the second feature fusion step. The second feature fusion step involves fusing each second feature map to be fused with the fused feature map, using the image obtained by feature fusion as the first feature map to be fused, and then performing the first feature fusion step. A new fused feature map is obtained from the result of the first feature fusion step. The final fused feature map is then processed through convolution and a nonlinear activation function to obtain the synthesized image.

2. An image generation method based on heterogeneous feature fusion, characterized in that, include: Acquire the source image, the target image, and the mask image, wherein the target image can be one or more; Feature extraction is performed on the source image and each of the target images respectively to obtain a first feature map group of the source image and a first feature map group of each of the target images, wherein the first feature map group includes multiple first feature maps of different resolutions; First feature maps with the same resolution are selected from the first feature map group of the source image and each target image. The first feature map of the selected target image and the mask image are subjected to the same geometric transformation. Then, the first feature maps of the selected source image and the target image are used as the first feature maps to be fused, and the first feature fusion step is performed to obtain the fused feature map. In the first feature fusion step, for each pixel point corresponding to a position on a first feature map to be fused, a pixel point of one of the first feature maps to be fused is selected according to the mask image as the pixel point at the corresponding position in the fused feature map. The fused feature map is composed of the pixel points selected at each position, wherein the mask image is used to identify the source of each pixel point in the fused feature map. In a predetermined order, the first feature maps with the same resolution in the first feature map group of the source image and each of the target images are used as the second feature maps to be fused, and the second feature fusion step is performed until the first feature maps in the first feature map group of the source image and each of the target images have all undergone the second feature fusion step. The second feature fusion step involves fusing each second feature map to be fused with the fused feature map, using the image obtained by feature fusion as the first feature map to be fused, and then performing the first feature fusion step. A new fused feature map is obtained from the result of the first feature fusion step. The final fused feature map is then processed through convolution and a nonlinear activation function to obtain the synthesized image.

3. The image generation method as described in claim 1 or 2, characterized in that, The step of extracting features from the source image and each of the target images to obtain a first feature map group for the source image and a first feature map group for each of the target images includes: The source image and each target image are respectively input into the feature extraction network to obtain N feature maps of the source image and each target image with decreasing resolution. These feature maps are used as the first feature maps to form the first feature map group. The feature extraction network includes N downsampling modules, each of which generates a feature map, where N is a positive integer and N≥2.

4. The image generation method as described in claim 3, characterized in that, Each of the downsampling modules includes a downsampling layer and several convolutional-activation layers.

5. The image generation method as described in claim 3, characterized in that, The step of selecting the first feature map with the same resolution from the first feature map group of the source image and each target image as the first feature map to be fused, and performing the first feature fusion step to obtain the fused feature map, specifically involves: selecting the Nth first feature map of the source image and each target image as the first feature map to be fused, and performing the first feature fusion step to obtain the fused feature map; The step of sequentially using first feature maps of the same resolution in the first feature map group of the source image and each target image as second feature maps to be fused, and performing the second feature fusion step, specifically involves: starting from the Nth first feature map in the first feature map group of the source image and each target image up to the 1st first feature map, sequentially using the first feature maps in the first feature map group of the source image and each target image as second feature maps to be fused, and performing the second feature fusion step. Specifically, obtaining a new fused feature map from the result of the first feature fusion step involves inputting the result of the first feature fusion step into the upsampling module of the decoder network to obtain a new fused feature map. The encoder network corresponding to the decoder network is the feature extraction network. The decoder network includes N upsampling modules. The input of the nth upsampling module is the result of the first feature fusion step when the N+1-nth first feature map performs the second feature fusion step. The output is a feature map with the same resolution as the Nnth first feature map. The output of the Nth upsampling module is a feature map with the same resolution as the source image and the target image. n = 1, 2...N.

6. The image generation method as described in claim 5, characterized in that, Each of the upsampling modules includes an upsampling layer and several convolutional-activation layers.

7. The image generation method as described in claim 5 or 6, characterized in that, The encoder network and the decoder network are trained in the following manner: A generator is composed of the encoder network and the decoder network, and a generative adversarial network is formed with a pre-built discriminator. The generator and the discriminator are trained alternately to obtain the parameters of the generator and the discriminator. The loss function for training the generator is: p real Let G(p) represent the training image, l represent the mask image, and G(p) represent the training image. real (l) represents the synthesized image obtained by inputting the source image, the geometrically transformed training image, and the mask image into the generator. o (G(p real ,l)) represents the output of the discriminator at layer o when the synthesized image is input into the discriminator, D o (p real ) represents the output of the discriminator's layer o when the training image is input into the discriminator, where o represents the set of network layers of the selected discriminator; The loss function for training the discriminator is: L2=-log(D(p real ))-log(1-D(G(p real ,l))), D(p real ) represents the result obtained by inputting the training image into the discriminator, D(G(p) real ,l)) represents the result obtained by inputting the synthesized image into the discriminator.

8. The image generation method according to any one of claims 1 to 7, characterized in that, The step of fusing each second feature map to be fused with the fusion feature map includes: The fused feature map is instance-normalized according to the following formula: Where a i,j,k To fuse the pixel value of the i-th row, j-th column, k-th channel in the feature map, m k To calculate the mean of the k-th channel of the fused feature map, s k The standard deviation of the k-th channel of the fused feature map is given by ε, where ε is a preset error value and b is the standard deviation of the k-th channel. i,j,k The pixel value of the i-th row, j-th column, and k-th channel in the fused feature map after instance normalization; Obtain the mean and standard deviation of each channel of each second feature map to be fused. Multiply each channel of the fused feature map after instance standardization by the standard deviation of the corresponding channel of each second feature map to be fused, and then add it to the mean of the corresponding channel.

9. The image generation method as described in claim 1, characterized in that, The pixel value of each pixel in the mask is used to identify the source of the pixel at the corresponding position in the fused feature map. The step of selecting one pixel in the first feature map to be fused as the pixel at the corresponding position in the fused feature map, and forming the fused feature map from the selected pixels at each position, includes: scaling the mask to the same resolution as the first feature map to be fused; for each pixel in the scaled mask, selecting one pixel at the corresponding position in the first feature map to be fused as the pixel at the corresponding position in the fused feature map, and forming the fused feature map from the selected pixels.

10. A computer-readable storage medium, characterized in that, The medium stores a program that can be executed by a processor to implement the image generation method as described in any one of claims 1-9.

Images