Image generation method, apparatus, device, storage medium, and program product
By performing transpose convolution and adaptive affine transformation on the training dataset of generative adversarial networks, and combining it with feature extraction from the discriminator, the problem of insufficient image generation quality and stability in existing technologies is solved, and more efficient image generation and model training are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI INTEGRATED CIRCUIT RESEARCH & DEVELOPMENT CENTER CO LTD
- Filing Date
- 2024-12-20
- Publication Date
- 2026-06-26
Smart Images
Figure CN122289424A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image generation technology, and in particular to an image generation method, apparatus, device, storage medium, and program product. Background Technology
[0002] Currently, when applying image generation technology to deep learning tasks that use images as research objects (such as face detection, lane recognition, etc.), training images are generated through image generation technology.
[0003] Existing techniques, in the process of generating training images for generative adversarial networks, adjust the feature data of each channel in the feature map through batch normalization to enhance the quality of the generated images, thereby improving the training efficiency of generative adversarial networks.
[0004] However, existing technologies struggle to fully extract spatial and detail features from images, which can easily reduce the quality of generated images, thereby decreasing the training efficiency and stability of generative adversarial networks (GANs). Summary of the Invention
[0005] This application provides image generation methods, apparatus, devices, storage media, and program products to enhance the ability to acquire image details, improve image quality and the stability of image generation models, and increase the efficiency of iterative training.
[0006] In a first aspect, embodiments of this application provide an image generation method, including:
[0007] Obtain the training dataset and input it into the generative adversarial network. The training dataset includes multiple training source image data and corresponding label images. The generative adversarial network includes an image generation model and a discriminator.
[0008] An image generation model is used to perform transposed convolution processing on the training source image data to obtain an upsampled feature map; an adaptive affine transformation is performed on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map; and a generated image corresponding to the training source image data is generated based on the pixel-level normalized feature map.
[0009] The discriminator extracts features from both the generated image and the label image to obtain the discrimination result of the generated image. The discrimination result is used to update and iterate the parameters of the image generation model.
[0010] Optionally, an adaptive affine transformation is performed on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map, specifically including:
[0011] Batch normalization is performed on the upsampled feature maps to obtain the normalized feature maps corresponding to the training source image data;
[0012] Multiple potential mask data corresponding to the upsampled feature map are determined based on the normalized feature map;
[0013] Based on the scaling parameter matrix and offset parameter matrix corresponding to multiple potential mask data, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain a pixel-level normalized feature map. The size of the scaling parameter matrix and offset parameter matrix is the same as the matrix size of the upsampled feature map.
[0014] Optionally, multiple potential mask data corresponding to the upsampled feature map are determined based on the normalized feature map, specifically including:
[0015] A nonlinear transformation is performed on the normalized feature map to obtain a nonlinear feature map; the nonlinear feature map is then processed to obtain feature maps corresponding to multiple output channels.
[0016] The feature map corresponding to each output channel is processed to obtain multiple potential mask data corresponding to multiple output channels in the upsampled feature map.
[0017] Optionally, based on the scaling parameter matrix and offset parameter matrix corresponding to multiple latent mask data, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain a pixel-level normalized feature map, specifically including:
[0018] For any output channel's latent mask data, an adaptive affine transformation is performed on multiple pixels in the feature map corresponding to the output channel based on the scaling parameter matrix and offset parameter matrix corresponding to the latent mask data.
[0019] After performing pixel-adaptive affine transformation on each output channel of the upsampled feature map, the pixel-level normalized feature map corresponding to the upsampled feature map is obtained.
[0020] Optionally, for any output channel, anti-latent mask data is determined;
[0021] The first parameter matrix and the second parameter matrix are obtained by performing depthwise separable convolution on the latent mask data; and the third parameter matrix and the fourth parameter matrix are obtained by performing depthwise separable convolution on the inverse latent mask data.
[0022] The scaling parameter matrix corresponding to the potential mask data is determined based on the first parameter matrix and the third parameter matrix; and the offset parameter matrix corresponding to the potential mask data is determined based on the second parameter matrix and the fourth parameter matrix.
[0023] Optionally, a discriminator performs feature extraction on the generated image and the label image to obtain the discrimination result of the generated image, specifically including:
[0024] By performing convolution processing on the generated image, the feature data of the generated image is obtained. After batch normalization processing on the feature data of the generated image, the normalized feature data corresponding to the generated image is obtained. Similarly, by performing convolution processing on the label image, the feature data of the label image is obtained. After batch normalization processing on the feature data of the label image, the normalized feature data corresponding to the label image is obtained.
[0025] The discrimination result of the generated image is determined based on the normalized feature data corresponding to the generated image and the normalized feature data corresponding to the label image.
[0026] Secondly, embodiments of this application provide an image generation apparatus, comprising:
[0027] The acquisition module is used to acquire the training dataset and input the training dataset into the generative adversarial network. The training dataset includes multiple training source image data and corresponding label images. The generative adversarial network includes an image generation model and a discriminator.
[0028] The processing module is used to perform transposed convolution processing on the training source image data using an image generation model to obtain an upsampled feature map; to perform adaptive affine transformation processing on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map; and to generate the generated image corresponding to the training source image data based on the pixel-level normalized feature map.
[0029] The processing module is also used to extract features from the generated image and the label image respectively through a discriminator to obtain the discrimination result of the generated image. The discrimination result is used to update and iterate the parameters of the image generation model.
[0030] Optionally, the processing module is also used to obtain a normalized feature map corresponding to the training source image data by performing batch normalization processing on the upsampled feature map;
[0031] Multiple potential mask data corresponding to the upsampled feature map are determined based on the normalized feature map;
[0032] Based on the scaling parameter matrix and offset parameter matrix corresponding to multiple potential mask data, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain a pixel-level normalized feature map. The size of the scaling parameter matrix and offset parameter matrix is the same as the matrix size of the upsampled feature map.
[0033] Optionally, the processing module is also used to perform a nonlinear transformation on the normalized feature map to obtain a nonlinear feature map; and to process the nonlinear feature map to obtain feature maps corresponding to multiple output channels respectively.
[0034] The feature map corresponding to each output channel is processed to obtain multiple potential mask data corresponding to multiple output channels in the upsampled feature map.
[0035] Optionally, the processing module is also used to perform adaptive affine transformation processing on multiple pixels in the feature map corresponding to the output channel based on the scaling parameter matrix and offset parameter matrix corresponding to the potential mask data of any output channel.
[0036] After performing pixel-adaptive affine transformation on each output channel of the upsampled feature map, the pixel-level normalized feature map corresponding to the upsampled feature map is obtained.
[0037] Optionally, the processing module is also used to determine the anti-latent mask data for the latent mask data corresponding to any output channel;
[0038] The first parameter matrix and the second parameter matrix are obtained by performing depthwise separable convolution on the latent mask data; and the third parameter matrix and the fourth parameter matrix are obtained by performing depthwise separable convolution on the inverse latent mask data.
[0039] The scaling parameter matrix corresponding to the potential mask data is determined based on the first parameter matrix and the third parameter matrix; and the offset parameter matrix corresponding to the potential mask data is determined based on the second parameter matrix and the fourth parameter matrix.
[0040] Optionally, the processing module is further configured to obtain feature data of the generated image by performing convolution processing on the generated image, and obtain normalized feature data corresponding to the generated image after batch normalization processing on the feature data of the generated image; and obtain feature data of the label image by performing convolution processing on the label image, and obtain normalized feature data corresponding to the label image after batch normalization processing on the feature data of the label image.
[0041] The discrimination result of the generated image is determined based on the normalized feature data corresponding to the generated image and the normalized feature data corresponding to the label image.
[0042] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0043] The memory stores the instructions that the computer executes;
[0044] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0046] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0047] The image generation method, apparatus, device, storage medium, and program product provided in this application embodiment acquire a training dataset, input the training dataset into a generative adversarial network (GAN), and use the GAN's image generation model to perform transposed convolution processing on the training source image data to obtain an upsampled feature map. Then, adaptive affine transformation processing is performed on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map. Based on the pixel-level normalized feature map, a generated image corresponding to the training source image data is generated. A discriminator extracts features from both the generated image and the label image corresponding to the training source image data to obtain a discrimination result for the generated image. The parameters of the image generation model are updated and iterated based on the discrimination result. This application enhances the network's ability to capture image details during training, improves the quality of the generated images, enhances the stability of the image generation model, and accelerates the training process. Attached Figure Description
[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0049] Figure 1 A schematic diagram illustrating the application scenarios provided in this application;
[0050] Figure 2 Flowchart of the image generation method provided in this application Figure 1 ;
[0051] Figure 3 Flowchart of the image generation method provided in this application Figure 2 ;
[0052] Figure 4 Flowchart of the image generation method provided in this application Figure 3 ;
[0053] Figure 5 Flowchart of the image generation method provided in this application Figure 4 ;
[0054] Figure 6 Flowchart of the image generation method provided in this application Figure 5 ;
[0055] Figure 7 A schematic diagram of the image generation apparatus provided in this application;
[0056] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.
[0057] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0059] First, let me explain the terms used in this application:
[0060] Batch normalization is a method used to train neural network models to accelerate the training process.
[0061] Currently, image generation techniques are applied to deep learning tasks that use images as research objects (such as face detection and lane recognition), generating training images through these techniques. For example, ... Figure 1 As shown, a generative adversarial network (GAN) consists of an image generation model and a discriminator. Random vectors are input into the image generation model, which transforms training samples into generated images. These generated images, along with real images, are then input into the discriminator. The discriminator determines whether the generated images output by the image generation model are sufficiently close to the real images and generates a discrimination result. Based on this result, the image generation model is optimized. Through iterative training, the image generation model is made capable of generating realistic images.
[0062] Existing technologies, during the iterative training of generative adversarial networks (GANs), process each channel of the feature map corresponding to the input random vector using batch normalization to standardize the feature distribution of each channel in the feature map, thereby enhancing image quality and further improving the training efficiency of GANs and the stability of image generation models.
[0063] However, existing techniques perform batch normalization on each channel of the feature map using the same scaling and translation parameters, which makes it difficult to fully extract the spatial and detailed features of the image, easily reducing the quality of the generated image, thereby reducing the training efficiency of generative adversarial networks and the stability of the image generation model.
[0064] The image generation method provided in this application acquires a training dataset and inputs it into a generative adversarial network (GAN). The image generation model within the GAN performs transposed convolution on the training source image data to obtain an upsampled feature map. This upsampled feature map is then batch-normalized to obtain a normalized feature map. A nonlinear transformation is applied to the normalized feature map to determine a nonlinear feature map, resulting in a feature map for each output channel. Each output channel's feature map is processed to obtain its corresponding latent mask data. Based on this latent mask data, scaling and offset parameters applied to that output channel are calculated. The resulting scaling and offset parameters are then used to process individual pixels in the feature map corresponding to the output channel. This allows for adaptive affine transformation processing using the corresponding parameter matrices for different output channels, resulting in a pixel-level normalized feature map corresponding to the upsampled feature map. The generated image is then output based on this pixel-level normalized feature map, enhancing the ability to capture image details, improving image quality and the stability of the image generation model, and increasing the efficiency of iterative training. Meanwhile, the generated image and the label image are batch normalized separately by the discriminator to obtain the normalized feature maps corresponding to the generated image and the label image respectively. The image generation model is optimized by determining the authenticity probability of the generated image so that the output of the image generation model is closer to the label image, thereby improving the quality of the generated image and accelerating the model training process.
[0065] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0066] Figure 2 Flowchart of the image generation method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:
[0067] S201. Obtain the training dataset and input it into the generative adversarial network.
[0068] More specifically, a training dataset is obtained and input into the generative adversarial network. The training dataset includes multiple training source image data and corresponding labeled images. The generative adversarial network includes an image generation model and a discriminator.
[0069] For example, the image generation model in a generative adversarial network (GAN) is a generator. Additionally, the GAN includes an input layer. The training dataset is input into the generator through the input layer. The generator processes the training source image data and outputs the generated image corresponding to the training source image data. In this embodiment, the generator is an image generation model, and the training source image data is a random vector.
[0070] S202. The training source image data is processed using an image generation model to obtain pixel-level normalized feature maps and generate generated images corresponding to the training source image data.
[0071] More specifically, an image generation model is used to perform transposed convolution processing on the training source image data to obtain an upsampled feature map; an adaptive affine transformation is performed on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map; and a generated image corresponding to the training source image data is generated based on the pixel-level normalized feature map.
[0072] In one possible embodiment, such as Figure 3 As shown, the image generation model includes multiple generation units, each comprising a transposed convolutional layer, a pixel normalization module, and a ReLU function (Rectified Linear Unit). The transposed convolutional layer upsamples the training source image data to obtain an upsampled feature map, which is then input to the pixel normalization module. The pixel normalization module performs an adaptive affine transformation on the upsampled feature map to obtain a pixel-level normalized feature map. Finally, the ReLU function performs non-linear activation on the pixel-level normalized feature map, outputting the generated feature map. The transposed convolutional layer uses a 4x4 kernel with a stride of 2.
[0073] Optionally, the feature data can be processed again by the aforementioned generation unit according to the size of the generated feature map until the size of the generated feature map is consistent with the size of the label image, at which point the generated image is obtained and output.
[0074] Optionally, a pixel-level normalized feature map is obtained by performing adaptive affine transformation on each pixel of the upsampled feature map. Specifically, this includes: batch normalizing the upsampled feature map to obtain a normalized feature map corresponding to the training source image data; determining multiple latent mask data corresponding to the upsampled feature map based on the normalized feature map; and performing adaptive affine transformation on each pixel of the normalized feature map according to the scaling parameter matrix and offset parameter matrix corresponding to the multiple latent mask data, to obtain a pixel-level normalized feature map. The size of the scaling parameter matrix and offset parameter matrix is the same as the matrix size of the upsampled feature map. The latent mask data includes foreground and background features of the feature map corresponding to the output channel. Foreground features refer to features related to the main objects and / or regions in the feature map, and are typically used to identify and segment important objects in the image. Background features refer to features related to non-main objects and / or regions in the feature map, and background features are typically used to understand the overall environment and / or scene of the image.
[0075] This embodiment performs batch normalization on the upsampled feature map to obtain a normalized feature map. Then, based on the multiple latent mask data corresponding to the normalized feature map, it determines the scaling parameter matrix and offset parameter matrix corresponding to each latent mask data. Thus, it processes each pixel of the normalized feature map according to the multiple scaling parameter matrices and multiple offset parameter matrices, solving the technical problem that the normalized feature map is difficult to capture spatial and detailed features. This enhances the flexibility of image generation and image transformation, improves image generation efficiency and image quality, and further accelerates the network training process.
[0076] Optionally, determining multiple potential mask data corresponding to the upsampled feature map based on the normalized feature map specifically includes: performing a nonlinear transformation on the normalized feature map to obtain a nonlinear feature map; processing the nonlinear feature map to obtain feature maps corresponding to multiple output channels respectively; and processing the feature map corresponding to each output channel to obtain multiple potential mask data corresponding to multiple output channels in the upsampled feature map.
[0077] Optionally, based on the scaling parameter matrix and offset parameter matrix corresponding to the multiple latent mask data respectively, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain a pixel-level normalized feature map. Specifically, this includes: for the latent mask data of any output channel, performing an adaptive affine transformation on multiple pixels in the feature map corresponding to the output channel based on the scaling parameter matrix and offset parameter matrix corresponding to the latent mask data; after the adaptive affine transformation of the pixels of each output channel in the upsampled feature map is completed, a pixel-level normalized feature map corresponding to the upsampled feature map is obtained.
[0078] In one possible embodiment, such as Figure 4 As shown above, Figure 3 The pixel normalization module in this embodiment includes a batch normalization module, a ReLU function, a 1x1 convolutional layer, and a Sigmoid function. When the transposed convolutional layer inputs the upsampled feature map corresponding to the training source image data into the batch normalization module of the pixel normalization module, the batch normalization module performs batch normalization processing on the upsampled feature map to obtain a normalized feature map, which is then copied. Simultaneously, the copied normalized feature map is sent to the ReLU function, which performs a nonlinear transformation on the normalized feature map to obtain a nonlinear feature map, which is then sent to the 1x1 convolutional layer. The 1x1 convolutional layer processes the nonlinear feature map to obtain each output channel of the nonlinear feature map and the feature map corresponding to each output channel.
[0079] In one possible embodiment, the upsampled feature map is batch-normalized using a batch normalization module to obtain a normalized feature map, which is then stored. Simultaneously, the upsampled feature map is batch-normalized again using the same module to obtain a normalized feature map, which is then sent to the ReLU function.
[0080] The method for batch normalization of feature maps in this embodiment is similar in principle and effect to the existing technology. Both aim to improve the stability of feature distribution in each channel of the feature map and accelerate the network training process. This embodiment will not elaborate further here.
[0081] For example, if the dimensions of the nonlinear feature map are: length h, width w, and number of channels c, then when processing the nonlinear feature map through a 1x1 convolutional layer, the nonlinear feature map of length h and width w is mapped onto c output channels, resulting in a feature map corresponding to each output channel. Each output channel's feature map includes multiple elements, each element being one pixel.
[0082] In one possible embodiment, for any output channel's feature map, the Sigmoid function is used to process the feature map, mapping the values in the feature map to between 0 and 1, thus obtaining the latent mask data corresponding to the feature map of that output channel. Based on this latent mask data, the corresponding scaling parameter matrix and offset parameter matrix are calculated, and the obtained scaling parameter matrix is used... and offset parameter matrix Normalized feature map Each element belonging to the output channel is subjected to an adaptive affine transformation to obtain the pixel-level normalized feature map corresponding to the feature map of the output channel. The adaptive affine transformation formula is as follows: y is the pixel-level normalized feature map corresponding to the feature map of the output channel.
[0083] In one possible embodiment, such as Figure 4As shown, the feature maps of each output channel are processed by the Sigmoid function to obtain the latent mask data corresponding to each output channel. The scaling parameter matrix and offset parameter matrix corresponding to each output channel are calculated. Based on the obtained scaling parameter matrix and offset parameter matrix of different output channels, the normalized feature map is processed pixel by pixel to determine the pixel-level normalized feature map corresponding to the feature map of each output channel, thereby obtaining the pixel-level normalized feature map corresponding to the upsampled feature map.
[0084] This embodiment introduces nonlinearity into the upsampled feature map to obtain a nonlinear feature map, enhancing the network's ability to learn complex features. For each output channel of the nonlinear feature map, corresponding latent mask data is calculated. Then, each pixel in the feature map of that output channel is processed using the scaling and offset parameter matrices corresponding to that output channel, thereby fully extracting the local features of each pixel. This enhances the generative adversarial network's ability to capture spatial and detailed features of feature maps from different output channels during deep learning, thus improving the quality of the generated image and further increasing image generation efficiency.
[0085] Optionally, for any output channel, anti-latent mask data is determined; a first parameter matrix and a second parameter matrix are obtained by performing depthwise separable convolution on the latent mask data; a third parameter matrix and a fourth parameter matrix are obtained by performing depthwise separable convolution on the anti-latent mask data; a scaling parameter matrix corresponding to the latent mask data is determined based on the first parameter matrix and the third parameter matrix; and an offset parameter matrix corresponding to the latent mask data is determined based on the second parameter matrix and the fourth parameter matrix.
[0086] In one possible embodiment, for any given output channel corresponding to the latent mask data m, the corresponding anti-latent mask data is determined. ,like Figure 5 As shown, by performing depthwise separable convolution on the latent mask data m, the first parameter matrix corresponding to the latent mask data m is obtained. Second parameter matrix And by analyzing the anti-latent mask data The anti-latent mask data is obtained by performing depthwise separable convolution. The corresponding first parameter matrix Second parameter matrix .according to The scaling parameter matrix corresponding to the output channel is calculated, based on... The offset parameter matrix corresponding to the output channel is calculated. The depthwise separable convolution processing method in this embodiment is similar in implementation principle and technical effect to the prior art, and will not be described in detail here.
[0087] This embodiment calculates anti-latent mask data for the latent mask data corresponding to each output channel, thereby calculating the scaling parameter matrix and offset parameter matrix of the corresponding output channel for the latent mask data and anti-latent mask data of each output channel. This improves the accuracy of the parameters on which the adaptive affine transformation depends, enhances the ability and accuracy of the generative adversarial network to capture spatial and detail features of images during deep learning, improves image quality, increases image generation efficiency, and further accelerates the network training process.
[0088] S203. The discriminator extracts features from the generated image and the label image respectively to obtain the discrimination result of the generated image.
[0089] More specifically, the discriminator extracts features from the generated image and the label image respectively to obtain the discrimination result of the generated image. The discrimination result is used to update and iterate the parameters of the image generation model.
[0090] Optionally, a discriminator extracts features from the generated image and the label to obtain the discrimination result of the generated image. Specifically, this includes: performing convolution processing on the generated image to obtain feature data of the generated image, and performing batch normalization processing on the feature data of the generated image to obtain normalized feature data corresponding to the generated image; performing convolution processing on the label image to obtain feature data of the label image, and performing batch normalization processing on the feature data of the label image to obtain normalized feature data corresponding to the label image; and determining the discrimination result of the generated image based on the normalized feature data corresponding to the generated image and the normalized feature data corresponding to the label image.
[0091] In one possible embodiment, such as Figure 6 As shown, the discriminator includes convolutional layers, a ReLU function, a sigmoid function, and multiple discriminant units. Each discriminant unit includes a convolutional layer, a batch normalization module, and a ReLU function. The generated image and the corresponding label image are input into the convolutional layer. The convolutional layer processes the generated image and the label image separately to obtain feature data for the generated image and the label image, respectively. The ReLU function is then used to perform non-linear activation processing on the feature data of the generated image and the label image, respectively, to obtain non-linear feature data for the generated image and the label image, which are then input into the discriminant unit. The convolutional layer described above has a kernel size of 4x4 and a stride of 2.
[0092] In one possible embodiment, image feature data of the generated image and the label image are extracted by the convolutional layer of the discriminant unit. The distribution of image feature data on different output channels is processed by a batch normalization model. Then, the feature data on each output channel after processing is nonlinearly transformed by the ReLU function to obtain the enhanced feature data of the generated image and the enhanced feature data (or enhanced feature map) of the label image.
[0093] Optionally, such as Figure 6 As shown, in order to align the size of the enhanced feature map of the generated image and the enhanced feature map of the label image with the target size, multiple discriminant units are used to process the generated image and the label image until the size of the corresponding enhanced feature map is consistent with the target size. Then, features are extracted from the enhanced feature map through a convolutional layer. After that, the output of the convolutional layer is converted into a probability value using the Sigmoid function. This probability value is used to represent the probability that the generated image is the label image (or the real image).
[0094] Optionally, a binary cross-entropy loss function is used to measure the difference between the generated image and the labeled image. Backpropagation and optimization algorithms (such as the Adam optimizer) are used to continuously adjust the parameters of the generative adversarial network or the image generation model to minimize the difference between the generated and labeled images. When the probability value of the generated image exceeds a preset threshold (e.g., 0.95), the generated image is considered sufficiently realistic and can be recognized as a real image by the discriminator; at this point, training stops. The loss function in this embodiment is similar in principle and technical effect to existing technologies, and will not be elaborated upon here.
[0095] This embodiment performs batch normalization processing on the generated image and the labeled image by using multiple discriminator units, which improves the discriminator's feature extraction capability, processing speed, and network training efficiency, thereby further enhancing the quality of the generated image.
[0096] The image generation method provided in this embodiment obtains pixel-level normalized feature maps by performing adaptive affine transformation on the upsampled feature maps, thereby enhancing the image quality generated by the image generation model. Furthermore, it enhances the network's ability to learn complex features by performing nonlinear transformation on the pixel-level normalized feature maps, thereby improving the overall performance of the generative adversarial network and accelerating the training process.
[0097] Figure 7 A schematic diagram of the image generation apparatus provided in this application is shown below. Figure 7 As shown, the image generation apparatus 70 provided in this embodiment includes:
[0098] The acquisition module 701 is used to acquire the training dataset and input the training dataset into the generative adversarial network. The training dataset includes multiple training source image data and corresponding label images. The generative adversarial network includes an image generation model and a discriminator.
[0099] The processing module 702 is used to perform transposed convolution processing on the training source image data using an image generation model to obtain an upsampled feature map; to perform adaptive affine transformation processing on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map; and to generate a generated image corresponding to the training source image data based on the pixel-level normalized feature map.
[0100] The processing module 702 is also used to extract features from the generated image and the label image respectively through a discriminator to obtain the discrimination result of the generated image. The discrimination result is used to update and iterate the parameters of the image generation model.
[0101] Optionally, the processing module 702 is further configured to obtain a normalized feature map corresponding to the training source image data by performing batch normalization processing on the upsampled feature map;
[0102] Multiple potential mask data corresponding to the upsampled feature map are determined based on the normalized feature map;
[0103] Based on the scaling parameter matrix and offset parameter matrix corresponding to multiple potential mask data, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain a pixel-level normalized feature map. The size of the scaling parameter matrix and offset parameter matrix is the same as the matrix size of the upsampled feature map.
[0104] Optionally, the processing module 702 is further configured to perform a nonlinear transformation on the normalized feature map to obtain a nonlinear feature map; and to process the nonlinear feature map to obtain feature maps corresponding to multiple output channels respectively.
[0105] The feature map corresponding to each output channel is processed to obtain multiple potential mask data corresponding to multiple output channels in the upsampled feature map.
[0106] Optionally, the processing module 702 is further configured to perform adaptive affine transformation processing on multiple pixels in the feature map corresponding to the output channel based on the scaling parameter matrix and offset parameter matrix corresponding to the potential mask data of any output channel.
[0107] After performing pixel-adaptive affine transformation on each output channel of the upsampled feature map, the pixel-level normalized feature map corresponding to the upsampled feature map is obtained.
[0108] Optionally, the processing module 702 is also used to determine the anti-latent mask data for the latent mask data corresponding to any output channel;
[0109] The first parameter matrix and the second parameter matrix are obtained by performing depthwise separable convolution on the latent mask data; and the third parameter matrix and the fourth parameter matrix are obtained by performing depthwise separable convolution on the inverse latent mask data.
[0110] The scaling parameter matrix corresponding to the potential mask data is determined based on the first parameter matrix and the third parameter matrix; and the offset parameter matrix corresponding to the potential mask data is determined based on the second parameter matrix and the fourth parameter matrix.
[0111] Optionally, the processing module 702 is further configured to obtain feature data of the generated image by performing convolution processing on the generated image, and obtain normalized feature data corresponding to the generated image after batch normalization processing on the feature data of the generated image; and obtain feature data of the label image by performing convolution processing on the label image, and obtain normalized feature data corresponding to the label image after batch normalization processing on the feature data of the label image.
[0112] The discrimination result of the generated image is determined based on the normalized feature data corresponding to the generated image and the normalized feature data corresponding to the label image.
[0113] The image generation apparatus provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0114] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.
[0115] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.
[0116] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0117] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0118] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0119] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0120] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0121] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), and flash memory. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0122] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0123] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0125] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0126] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM) and random access memory (RAM).
[0127] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM and RAM.
[0128] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. An image generation method characterized by, include: Obtain a training dataset and input the training dataset into a generative adversarial network. The training dataset includes multiple training source image data and corresponding label images. The generative adversarial network includes an image generation model and a discriminator. The image generation model is used to perform transposed convolution processing on the training source image data to obtain an upsampled feature map; an adaptive affine transformation is performed on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map; and a generated image corresponding to the training source image data is generated based on the pixel-level normalized feature map. The generated image and the labeled image are subjected to feature extraction by a discriminator to obtain the discrimination result of the generated image. The discrimination result is used to update and iterate the parameters of the image generation model.
2. The method of claim 1, wherein, By performing adaptive affine transformation on each pixel of the upsampled feature map, a pixel-level normalized feature map is obtained, specifically including: Batch normalization is performed on the upsampled feature map to obtain the normalized feature map corresponding to the training source image data; Based on the normalized feature map, determine multiple potential mask data corresponding to the upsampled feature map; Based on the scaling parameter matrix and offset parameter matrix corresponding to the multiple potential mask data, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain the pixel-level normalized feature map. The size of the scaling parameter matrix and the offset parameter matrix is the same as the matrix size of the upsampled feature map.
3. The method of claim 2, wherein, Based on the normalized feature map, multiple potential mask data corresponding to the upsampled feature map are determined, specifically including: The normalized feature map is subjected to a nonlinear transformation to obtain a nonlinear feature map; the nonlinear feature map is then processed to obtain feature maps corresponding to multiple output channels. The feature map corresponding to each output channel is processed to obtain multiple potential mask data corresponding to multiple output channels in the upsampled feature map.
4. The method of claim 3, wherein, Based on the scaling parameter matrix and offset parameter matrix corresponding to the multiple potential mask data, an adaptive affine transformation is performed on each pixel in the normalized feature map to obtain the pixel-level normalized feature map, specifically including: For any output channel's potential mask data, an adaptive affine transformation is performed on multiple pixels in the feature map corresponding to the output channel based on the scaling parameter matrix and the offset parameter matrix corresponding to the potential mask data. After performing pixel-adaptive affine transformation on each output channel of the upsampled feature map, a pixel-level normalized feature map corresponding to the upsampled feature map is obtained.
5. The method of claim 4, wherein, Also includes: For any output channel, determine the inverse latent mask data; By performing depthwise separable convolution on the potential mask data, the first parameter matrix and the second parameter matrix are obtained. The third parameter matrix and the fourth parameter matrix are obtained by performing depthwise separable convolution on the anti-latent mask data. The scaling parameter matrix corresponding to the potential mask data is determined based on the first parameter matrix and the third parameter matrix; The offset parameter matrix corresponding to the potential mask data is determined based on the second parameter matrix and the fourth parameter matrix.
6. The method of claim 1, wherein, The generated image and the labeled image are subjected to feature extraction by a discriminator to obtain the discrimination result of the generated image, specifically including: By performing convolution processing on the generated image, feature data of the generated image is obtained, and after batch normalization processing on the feature data of the generated image, normalized feature data corresponding to the generated image is obtained; by performing convolution processing on the label image, feature data of the label image is obtained, and after batch normalization processing on the feature data of the label image, normalized feature data corresponding to the label image is obtained. The discrimination result of the generated image is determined based on the normalized feature data corresponding to the generated image and the normalized feature data corresponding to the label image.
7. An image generation apparatus characterized by comprising: include: The acquisition module is used to acquire a training dataset and input the training dataset into the generative adversarial network. The training dataset includes multiple training source image data and corresponding label images. The generative adversarial network includes an image generation model and a discriminator. The processing module is used to perform transposed convolution processing on the training source image data using the image generation model to obtain an upsampled feature map; to perform adaptive affine transformation processing on each pixel of the upsampled feature map to obtain a pixel-level normalized feature map; and to generate a generated image corresponding to the training source image data based on the pixel-level normalized feature map. The processing module is further configured to extract features from the generated image and the label image respectively using a discriminator to obtain a discrimination result of the generated image, and the discrimination result is used to update and iterate the parameters of the image generation model.
8. An electronic device, comprising: include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.
10. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.