Image generation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining target posture information and a source image; performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; performing random noise simulation on the source image to obtain a noise-added simulation image; performing denoising guide analysis on the multi-scale appearance feature data through a target denoising guide sub-model of the target image generation model to obtain denoising guide data; and through a target convolution denoising sub-model of the target image generation model, performing denoising processing on the noise-added simulation image according to the denoising guide data and the target posture information to obtain a target generation image. Therefore, by adopting the method, the quality and generalization ability of the attitude guidance image generation technology can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of image synthesis technology, pose-guided person image synthesis technology (Pose-Guided Person Image Generation, PGPIG) has emerged. This technology can be applied to many application scenarios, such as film production, clothing display, virtual reality, advertising production, etc. These application scenarios demonstrate the diversity and flexibility of PGPIG technology, which can be customized and applied according to different needs and environments. However, the current PGPIG technology has some technical problems with limitations in specific applications. For example, a generative adversarial network (Conditional Generative Adversarial Networks, CGANs) applied to PGPIG technology will face the problems of unstable training and limited sample diversity, and a generative model (Variational Autoencoders VAEs) applied to PGPIG technology will be difficult to ensure detail alignment and pose consistency. Therefore, there is an urgent need for an image generation method that can improve the quality and generalization ability of PGPIG technology and achieve effective person pose guided images without complex text prompts or annotations. Summary of the invention

[0003] Based on this, it is necessary to provide an image generation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the quality and generalization ability of PGPIG technology and realize effective character posture guidance images without complex text prompts or annotations to address the above-mentioned technical problems.

[0004] In a first aspect, the present application provides an image generation method, comprising:

[0005] Get target pose information and source image;

[0006] Performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data;

[0007] Performing random noise simulation on the source image to obtain a noisy simulated image;

[0008] Performing denoising guidance analysis on the multi-scale appearance feature data through a target denoising guidance sub-model of the target image generation model to obtain denoising guidance data;

[0009] The target convolution denoising sub-model of the target image generation model is used to denoise the noisy simulated image according to the denoising guide data and the target posture information to obtain a target generated image.

[0010] In one embodiment, the target denoising guidance sub-model includes: a target perception refinement sub-model and a target attention sub-model, and the denoising guidance data includes: coarse-grained feature data and fine-grained texture data; the target denoising guidance sub-model of the target image generation model performs denoising guidance analysis on the multi-scale appearance feature data to obtain denoising guidance data, including:

[0011] Perceptually refining the multi-scale appearance feature data according to the target perception refinement sub-model to obtain the coarse-grained structural features;

[0012] Attention analysis is performed on the multi-scale appearance feature data according to the target attention sub-model to obtain the fine-grained texture data.

[0013] In one embodiment, the target convolution denoising sub-model includes: a target convolution neural model and a target posture adaptation model; the target convolution denoising sub-model generated by the target image generation model performs denoising on the noisy simulated image according to the denoising guidance data and the target posture information to obtain a target generated image, including:

[0014] Performing posture structure analysis on the target posture information according to the target posture adaptation model to obtain posture structure guidance data;

[0015] According to the target convolutional neural model, the noisy simulated image is gradually denoised based on the denoising guidance data and the posture structure guidance data to obtain the target generated image.

[0016] In one of the embodiments, before performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance sub-model of the target image generation model to obtain denoising guidance data, the method further includes:

[0017] Acquire training set data, wherein the training set data includes: training posture information and training images corresponding to the training posture information;

[0018] Preprocessing the training image to obtain multi-scale training feature data and a noisy training image;

[0019] According to the original denoising guidance sub-model of the original image generation model, performing denoising guidance analysis on the multi-scale training feature data to obtain training guidance data;

[0020] According to the original convolution denoising sub-model of the original image generation model, the noisy training image is denoised according to the training guidance data and the training posture information to obtain an original generated image;

[0021] The parameters of the original image generation model are loss-optimized according to the original generated image and the preset optimization model to obtain the target image generation model.

[0022] In one embodiment, the preprocessing of the training image to obtain multi-scale training feature data and a noisy training image includes:

[0023] Performing multi-scale feature mapping on the training image to obtain the multi-scale training feature data;

[0024] Perform random noise simulation on the training image to obtain the noisy training image.

[0025] In one embodiment, the method further comprises:

[0026] Performing loss optimization on the parameters of the original image generation model according to the original generated image and the preset optimization model to obtain a candidate image generation model;

[0027] Performing a performance evaluation on the candidate image generation model to obtain a performance evaluation result;

[0028] If the performance evaluation result meets the preset conditions, the candidate image generation model is used as the target image generation model.

[0029] In a second aspect, the present application further provides an image generating device, comprising:

[0030] An acquisition module, used to acquire target posture information and source images;

[0031] A mapping module, used for performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data;

[0032] A noise simulation module, used for performing random noise simulation on the source image to obtain a noisy simulated image;

[0033] A guidance analysis module, configured to perform denoising guidance analysis on the multi-scale appearance feature data according to a target denoising guidance sub-model of a target image generation model to obtain denoising guidance data;

[0034] An image generation module is used to perform denoising processing on the noisy simulated image according to the target convolution denoising sub-model of the target image generation model, according to the denoising guidance data and the target posture information, to obtain a target generated image.

[0035] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Get target pose information and source image;

[0037] Performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data;

[0038] Performing random noise simulation on the source image to obtain a noisy simulated image;

[0039] Performing denoising guidance analysis on the multi-scale appearance feature data through a target denoising guidance sub-model of the target image generation model to obtain denoising guidance data;

[0040] The target convolution denoising sub-model of the target image generation model is used to denoise the noisy simulated image according to the denoising guide data and the target posture information to obtain a target generated image.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] Get target pose information and source image;

[0043] Performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data;

[0044] Performing random noise simulation on the source image to obtain a noisy simulated image;

[0045] Performing denoising guidance analysis on the multi-scale appearance feature data through a target denoising guidance sub-model of the target image generation model to obtain denoising guidance data;

[0046] The target convolution denoising sub-model of the target image generation model is used to denoise the noisy simulated image according to the denoising guide data and the target posture information to obtain a target generated image.

[0047] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0048] Get target pose information and source image;

[0049] Performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data;

[0050] Performing random noise simulation on the source image to obtain a noisy simulated image;

[0051] Performing denoising guidance analysis on the multi-scale appearance feature data through a target denoising guidance sub-model of the target image generation model to obtain denoising guidance data;

[0052] The target convolution denoising sub-model of the target image generation model is used to denoise the noisy simulated image according to the denoising guide data and the target posture information to obtain a target generated image.

[0053] The above-mentioned image generation method, device, computer equipment, computer-readable storage medium and computer program product obtain target posture information and source image; perform multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; perform random noise simulation on the source image to obtain a noisy simulated image; perform denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel of the target image generation model to obtain denoising guidance data; perform denoising processing on the noisy simulated image according to the denoising guidance data and target posture information through the target convolution denoising submodel of the target image generation model to obtain the target generated image. Therefore, by performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel, denoising guidance data with denoising guidance function can be analyzed and obtained, thereby guiding the subsequent denoising process to generate high-quality target generated images, that is, without the need for complex text prompts or annotations, effective character posture guidance image generation can be achieved, thereby ensuring the generalization performance of the target image generation model. In addition, the target convolution denoising submodel and target posture information can be used to gradually remove noise while ensuring the correctness of the posture in the target generated image, further improving the image generation quality of the PGPIG technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] FIG1 is a diagram showing an application environment of an image generation method according to an embodiment;

[0056] FIG2 is a schematic diagram of a flow chart of an image generating method in one embodiment;

[0057] FIG3 is a schematic diagram of a flow chart of an image generating method in one embodiment;

[0058] FIG4 is a schematic diagram of the overall flow of an image generating method in another embodiment;

[0059] FIG5 is a diagram showing the generation effect of a target generated image of an image generation method according to an embodiment;

[0060] FIG6 is a structural block diagram of an image generating device in one embodiment;

[0061] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The image generation method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1. The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the target posture information and the source image; performs multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; performs random noise simulation on the source image to obtain a noisy simulated image; performs denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel of the target image generation model to obtain denoising guidance data; performs denoising processing on the noisy simulated image according to the denoising guidance data and the target posture information through the target convolution denoising submodel of the target image generation model to obtain the target generated image. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0064] In an exemplary embodiment, as shown in FIG2 , an image generation method is provided, which is described by taking the method applied to the server 104 in FIG1 as an example, and includes the following steps 202 to 212. Among them:

[0065] Step 202, obtaining target posture information and source image.

[0066] Among them, the target posture information is the posture information used to determine the posture form of the character in the subsequent target generated image, the source image is the original image to be processed containing character information and background information, and the source image is used to provide appearance feature data for the subsequent image generation process.

[0067] In some embodiments, posture information can be extracted from posture images to obtain target posture information, or the target posture information can be directly obtained from a relevant posture information database. The posture image can be an image obtained in real time, or a posture image obtained from a posture image material set containing different types of postures, but is not limited to this.

[0068] In some embodiments, the source image can be directly acquired by real-scene shooting using a shooting device, or frame image information can be extracted from a long video as the source image, but is not limited thereto.

[0069] Step 204: perform multi-scale feature mapping on the source image to obtain multi-scale appearance feature data.

[0070] Among them, multi-scale feature mapping is a processing method for extracting feature data of the source image, and multi-scale features mean feature data containing different granularities.

[0071] In some embodiments, a source image encoder performs multi-scale feature mapping on a source image to obtain multi-scale appearance feature data, wherein the source image encoder is used to receive a source image and extract the multi-scale feature mapping to provide input for a subsequent image generation process.

[0072] Step 206: Perform random noise simulation on the source image to obtain a noisy simulated image.

[0073] Among them, random noise simulation is achieved by injecting random noise into the latent space corresponding to the source image, which is used to simulate the process of gradually losing details of the source image.

[0074] In some embodiments, random noise is injected into the latent space corresponding to the source image to obtain a noisy simulated image, so as to simulate the process of the source image gradually losing details.

[0075] Step 208 , performing denoising guidance analysis on the multi-scale appearance feature data according to the target denoising guidance sub-model of the target image generation model to obtain denoising guidance data.

[0076] Among them, the target image generation model is a pre-trained image generation model, which is used to generate a clear and realistic person image, which is visually similar to the source image, but uses a target posture corresponding to the target posture information.

[0077] It should be noted that the target denoising guidance sub-model is used to analyze the multi-scale appearance feature data to obtain denoising guidance data that can be used to guide the denoising operation, and the target convolution denoising sub-model is used to denoise the noisy simulated image.

[0078] In some embodiments, multi-scale appearance feature data can be input into the target denoising guidance sub-model, and the target posture information can be input into the target convolution denoising sub-model. After the target denoising guidance sub-model completes the denoising guidance analysis, the obtained denoising guidance data will be input into the target convolution denoising sub-model to guide the target convolution denoising sub-model to perform image denoising processing.

[0079] Among them, the multi-scale appearance feature data includes appearance feature data of different granularities. The denoising guidance analysis can perform a reasonable attention allocation analysis based on the appearance feature data of different granularities to obtain denoising guidance data that can guide the denoising operation to generate appearance features similar to the source image.

[0080] In some embodiments, the target denoising guidance submodel includes: a target perception refinement submodel and a target attention submodel, and the denoising guidance data includes: coarse-grained feature data and fine-grained texture data; the target denoising guidance submodel of the target image generation model is used to perform denoising guidance analysis on the multi-scale appearance feature data to obtain denoising guidance data, including: performing perceptual refinement on the multi-scale appearance feature data according to the target perception refinement submodel to obtain coarse-grained structural features; performing attention analysis on the multi-scale appearance feature data according to the target attention submodel to obtain fine-grained texture data.

[0081] The target perception refinement sub-model can use a perception refinement decoder, and the target attention sub-model can use a hybrid granularity attention module. The perception refinement decoder is a Perception-Refined Decoder, PRD. The hybrid granularity attention module is a Hybrid-Granularity Attention module, HGA.

[0082] In some embodiments, the target perceptual refinement sub-model includes a learnable query and a Transformer decoder, and a learnable query is performed on the multi-scale appearance feature data through the Transformer decoder to achieve perceptual refinement, so as to gradually refine the query to obtain coarse-grained image feature representations. These coarse-grained image feature representations are used to capture high-level semantic features of the source image, thereby obtaining coarse-grained structural features.

[0083] Among them, the coarse-grained image feature representation is coarse-grained prompt. Learnable queries are Learnable Queries.

[0084] In some embodiments, the hybrid granularity attention module adopted by the target attention sub-model is embedded in the upsampling block of the target denoising guidance sub-model, and the hybrid granularity attention module includes a component for encoding multi-scale fine-grained appearance features. The hybrid granularity attention module can encode fine-grained appearance features as bias terms, thereby realizing attention analysis of multi-scale appearance feature data and obtaining fine-grained texture data to enhance the coarse-grained image feature representation and improve the texture details of the generated image.

[0085] Step 210, according to the target convolution denoising sub-model of the target image generation model, denoising is performed on the noisy simulated image according to the denoising guidance data and the target posture information to obtain the target generated image.

[0086] The target convolution denoising sub-model includes a pre-trained convolutional neural network, which includes downsampling blocks, intermediate blocks, and upsampling blocks, as well as skip connections and partially frozen fine-tuning. The convolutional neural network is used as a basic network for image denoising and detail restoration. The convolutional neural network is UNet (Hn), the skip connection is SkipConnection, and the partially frozen fine-tuning is Frozen Partial Fine-tuning.

[0087] It should be noted that the process of denoising a noisy simulated image can be regarded as a means of image generation. By removing unnecessary noise points from an image full of noise points according to requirements, a target generated image that meets the requirements can be obtained.

[0088] In some embodiments, the target convolution denoising sub-model includes: a target convolution neural model and a target posture adaptation model; the target convolution denoising sub-model of the target image generation model performs denoising on the noisy simulated image according to denoising guidance data and target posture information to obtain a target generated image, including: performing posture structure analysis on the target posture information according to the target posture adaptation model to obtain posture structure guidance data; according to the target convolution neural model, the noisy simulated image is gradually denoised according to the denoising guidance data and the posture structure guidance data to obtain a target generated image.

[0089] In some embodiments, the target pose adaptation model uses a trained pose adapter, and performs pose structure analysis on the target pose information according to the pose adapter to obtain pose structure guidance data to ensure the correctness of the pose of the person in the generated image during the process of gradually removing noise. The pose adapter is a pose encoder (Hp).

[0090] In some embodiments, the target convolutional neural model adopts a trained convolutional neural network, and the noisy simulated image is gradually denoised through the convolutional neural network. During the step-by-step denoising process, the denoising process is guided by denoising guidance data and posture structure guidance data to ensure the correctness of the posture while guiding the denoising process to generate a high-quality target generated image.

[0091] In the above-mentioned image generation method, by obtaining the target posture information and the source image; performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; performing random noise simulation on the source image to obtain a noisy simulated image; performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel of the target image generation model to obtain denoising guidance data; performing denoising processing on the noisy simulated image according to the denoising guidance data and the target posture information through the target convolution denoising submodel of the target image generation model to obtain the target generated image. Therefore, by performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel, denoising guidance data with denoising guidance function can be analyzed and obtained, thereby guiding the subsequent denoising process to generate high-quality target generated images, that is, without the need for complex text prompts or annotations, effective character posture guidance image generation can be achieved, thereby ensuring the generalization performance of the target image generation model. In addition, the use of the target convolution denoising submodel and the target posture information can gradually remove noise while ensuring the correctness of the posture in the target generated image, further improving the image generation quality of the PGPIG technology.

[0092] In an exemplary embodiment, as shown in FIG3 , before performing denoising guidance analysis on the multi-scale appearance feature data by the target denoising guidance sub-model of the target image generation model to obtain denoising guidance data, the image generation method further includes: steps 302 to 310. Among them:

[0093] Step 302: Acquire training set data, wherein the training set data includes: training posture information and training images corresponding to the training posture information.

[0094] Among them, the training set data uses paired data, each training posture information corresponds to a unique training image, the training posture information is used to determine the posture information of the character posture in the subsequent original generated image, the training image corresponding to the training posture information is the original image to be processed containing character information and background information, and the training image is used to provide appearance feature data for the subsequent image generation process.

[0095] In some embodiments, posture information can be extracted from a large number of posture images to obtain training set data, or the training set data can be directly obtained from a relevant posture information database. The posture images can be images acquired in real time, or they can be posture images acquired from a posture image material set containing different types of postures, but are not limited to this.

[0096] Step 304: preprocess the training image to obtain multi-scale training feature data and a noisy training image.

[0097] In some embodiments, the training image is preprocessed to obtain multi-scale training feature data and a noisy training image, including: performing multi-scale feature mapping on the training image to obtain multi-scale training feature data; performing random noise simulation on the training image to obtain a noisy training image.

[0098] Among them, multi-scale feature mapping is a processing method for extracting feature data of the source image, and multi-scale features mean feature data containing different granularities.

[0099] In some embodiments, a source image encoder is used to perform multi-scale feature mapping on a training image to obtain multi-scale training feature data, wherein the source image encoder is used to receive the training image and extract the multi-scale feature mapping to provide input for a subsequent image generation process.

[0100] In some embodiments, random noise is injected into the latent space corresponding to the training image to obtain a noisy training image, so as to simulate the process of gradually losing details of the training image.

[0101] Step 306 , performing denoising guidance analysis on the multi-scale training feature data according to the original denoising guidance sub-model of the original image generation model to obtain training guidance data.

[0102] Among them, the original image generation model is an untrained image generation model, which is used to generate a clear and realistic person image, which is visually similar to the training image, but adopts the target posture corresponding to the target posture information.

[0103] It should be noted that the original denoising guidance sub-model is used to analyze the multi-scale training feature data to obtain training guidance data that can be used to guide the denoising operation, and the original convolution denoising sub-model is used to denoise the noisy training image.

[0104] In some embodiments, the multi-scale training feature data can be input into the original denoising guidance sub-model, and the training posture information can be input into the original convolution denoising sub-model. After the original denoising guidance sub-model completes the denoising guidance analysis, the obtained training guidance data will be input into the original convolution denoising sub-model to guide the original convolution denoising sub-model to perform image denoising processing.

[0105] Among them, the multi-scale training feature data includes appearance training feature data of different granularities. The denoising guidance analysis can perform a reasonable attention allocation analysis based on the appearance training feature data of different granularities to obtain training guidance data that can guide the denoising operation to generate appearance features similar to those of the training image.

[0106] In some embodiments, the original denoising guidance sub-model includes: an original perceptual refinement sub-model and an original attention sub-model, and the training guidance data includes: coarse-grained training structure data and fine-grained training texture data; the multi-scale training feature data can be perceptually refined according to the original perceptual refinement sub-model to obtain coarse-grained training structure data; and the multi-scale training feature data can be attention analyzed according to the original attention sub-model to obtain fine-grained training texture data.

[0107] The original perception-refined sub-model can use a perception-refined decoder, and the original attention sub-model can use a hybrid granularity attention module. The perception-refined decoder is a Perception-Refined Decoder, PRD. The hybrid granularity attention module is a Hybrid-Granularity Attention module, HGA.

[0108] In some embodiments, the raw-perceptual refinement sub-model includes a learnable query and a Transformer decoder, and the raw-perceptual refinement sub-model uses multi-scale training feature data and training pose information to gradually generate coarse-grained semantic cues through the Transformer decoder. At the same time, the hybrid granularity attention module (HA) encodes fine-grained appearance features as bias terms to enhance the details of the generation process and obtain fine-grained training texture data.

[0109] Among them, coarse-grained semantic prompts are coarse-grained prompts and learnable queries are LearnableQueries.

[0110] Step 308, according to the original convolution denoising sub-model of the original image generation model, denoising is performed on the noisy training image according to the training guidance data and the training posture information to obtain the original generated image.

[0111] The original convolutional denoising sub-model includes a pre-trained convolutional neural network for learning how to recover a clear image from noise. The original convolutional denoising sub-model also includes a pose adapter, which can provide structural guidance of the pose for the original convolutional denoising sub-model. The convolutional neural network is UNet (Hunet) and the pose adapter is Hp.

[0112] In some embodiments, the noisy training image is gradually denoised by the convolutional neural network of the original convolutional denoising sub-model, and the denoising process is guided by training guidance data and training posture information during the gradual denoising process to ensure the correctness of the posture while guiding the denoising process to generate a higher quality original generated image.

[0113] Step 310, performing loss optimization on the parameters of the original image generation model according to the original generated image and the preset optimization model to obtain the target image generation model.

[0114] Among them, during the training process, the preset optimization model adopts the minimization of reconstruction loss model and mean square error loss optimization model. The image quality of the original generated image is calculated by minimizing the reconstruction loss model and the mean square error loss optimization model to achieve parameter optimization of the parameters of the original image generation model, thereby ensuring that the image generated by the original image generation model not only conforms to the target posture but also maintains the appearance characteristics of the source image.

[0115] In some embodiments, the method also includes: performing loss optimization on the parameters of the original image generation model according to the original generated image and the preset optimization model to obtain a candidate image generation model; performing performance evaluation on the candidate image generation model to obtain a performance evaluation result; if the performance evaluation result meets the preset conditions, the candidate image generation model is used as the target image generation model.

[0116] Among them, when evaluating the performance of candidate image generation models, quantitative indicators such as structural similarity index (SSIM) and perceptual hash similarity (LPIPS) are used to measure image quality, and the preset conditions can be set according to actual application requirements.

[0117] In this embodiment, by performing performance evaluation on the candidate image generation model after performing loss optimization on the parameters of the original image generation model, it can be further ensured that the performance of the target image generation model meets user requirements.

[0118] In order to more clearly understand the solution of the present application, it is described here with reference to FIG. 4 and FIG. 5 , as follows:

[0119] Please refer to FIG. 4, which specifically shows the specific generation process of the target generated image, wherein the target perception refinement sub-model is H in FIG. D , the target attention sub-model is H A , the source image encoder is H S , the posture adapter is H P , the target convolutional neural model is H UNET .

[0120] Please refer to FIG5 , which is a diagram showing the specific effect of generating a target generated image based on a source image and target posture information.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides an image generation device for implementing the above-mentioned image generation method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image generation device embodiments provided below can refer to the limitations on the image generation method above, and will not be repeated here.

[0123] In an exemplary embodiment, as shown in FIG6 , an image generation device is provided, including: an acquisition module 602, a mapping module 604, a noise simulation module 606, a guidance analysis module 608 and an image generation module 610, wherein:

[0124] An acquisition module 602 is used to acquire target posture information and source images;

[0125] A mapping module 604 is used to perform multi-scale feature mapping on the source image to obtain multi-scale appearance feature data;

[0126] The noise simulation module 606 is used to perform random noise simulation on the source image to obtain a noisy simulated image;

[0127] A guidance analysis module 608 is used to perform denoising guidance analysis on the multi-scale appearance feature data according to the target denoising guidance sub-model of the target image generation model to obtain denoising guidance data;

[0128] The image generation module 610 is used to perform denoising processing on the noisy simulated image according to the target convolution denoising sub-model of the target image generation model, according to the denoising guidance data and the target posture information, to obtain the target generated image.

[0129] In some embodiments, the target denoising guidance sub-model includes: a target perception refinement sub-model and a target attention sub-model, and the denoising guidance data includes: coarse-grained feature data and fine-grained texture data; the guidance analysis module 608 is also used to perform perceptual refinement on the multi-scale appearance feature data according to the target perception refinement sub-model to obtain coarse-grained structural features; and perform attention analysis on the multi-scale appearance feature data according to the target attention sub-model to obtain fine-grained texture data.

[0130] In some embodiments, the target convolution denoising sub-model includes: a target convolution neural model and a target posture adaptation model; the image generation module 610 is also used to perform posture structure analysis on the target posture information according to the target posture adaptation model to obtain posture structure guidance data; according to the target convolution neural model, the noisy simulated image is gradually denoised according to the denoising guidance data and the posture structure guidance data to obtain a target generated image.

[0131] In some embodiments, before performing denoising guidance analysis on multi-scale appearance feature data through a target denoising guidance sub-model of a target image generation model to obtain denoising guidance data, the device also includes: a model training module, used to obtain training set data, wherein the training set data includes: training posture information and training images corresponding to the training posture information; preprocessing the training images to obtain multi-scale training feature data and noisy training images; performing denoising guidance analysis on the multi-scale training feature data according to the original denoising guidance sub-model of the original image generation model to obtain training guidance data; performing denoising processing on the noisy training image according to the original convolution denoising sub-model of the original image generation model based on the training guidance data and the training posture information to obtain the original generated image; performing loss optimization on the parameters of the original image generation model according to the original generated image and a preset optimization model to obtain a target image generation model.

[0132] In some embodiments, the model training module is further used to perform multi-scale feature mapping on the training image to obtain multi-scale training feature data; and perform random noise simulation on the training image to obtain a noisy training image.

[0133] In some embodiments, the model training module is also used to perform loss optimization on the parameters of the original image generation model based on the original generated image and the preset optimization model to obtain a candidate image generation model; perform performance evaluation on the candidate image generation model to obtain a performance evaluation result; if the performance evaluation result meets the preset conditions, the candidate image generation model is used as the target image generation model.

[0134] In the above-mentioned image generation device, by acquiring the target posture information and the source image; performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; performing random noise simulation on the source image to obtain a noisy simulated image; performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel of the target image generation model to obtain denoising guidance data; performing denoising processing on the noisy simulated image according to the denoising guidance data and the target posture information through the target convolution denoising submodel of the target image generation model to obtain the target generated image. Therefore, by performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance submodel, denoising guidance data with denoising guidance function can be analyzed and obtained, thereby guiding the subsequent denoising process to generate high-quality target generated images, that is, without the need for complex text prompts or annotations, effective character posture guidance image generation can be achieved, thereby ensuring the generalization performance of the target image generation model. In addition, the use of the target convolution denoising submodel and the target posture information can gradually remove noise while ensuring the correctness of the posture in the target generated image, further improving the image generation quality of the PGPIG technology.

[0135] Each module in the above-mentioned image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0136] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store target image generation model data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, a near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image generation method is implemented. The display unit of the computer device is used to form a visually visible picture, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball, or touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0137] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0140] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0142] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0144] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image generation method, characterized in that: The method comprises: Get target pose information and source image; Performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; Performing random noise simulation on the source image to obtain a noisy simulated image; Performing denoising guidance analysis on the multi-scale appearance feature data through a target denoising guidance sub-model of the target image generation model to obtain denoising guidance data; The target convolution denoising sub-model of the target image generation model is used to denoise the noisy simulated image according to the denoising guide data and the target posture information to obtain a target generated image.

2. The method according to claim 1, characterized in that: The target denoising guidance sub-model includes: a target perception refinement sub-model and a target attention sub-model, and the denoising guidance data includes: coarse-grained feature data and fine-grained texture data; the target denoising guidance sub-model of the target image generation model performs denoising guidance analysis on the multi-scale appearance feature data to obtain denoising guidance data, including: Perceptually refining the multi-scale appearance feature data according to the target perception refinement sub-model to obtain the coarse-grained structural features; Attention analysis is performed on the multi-scale appearance feature data according to the target attention sub-model to obtain the fine-grained texture data.

3. The method according to claim 1, characterized in that: The target convolution denoising sub-model includes: a target convolution neural model and a target posture adaptation model; the target convolution denoising sub-model generated by the target image generation model performs denoising processing on the noisy simulated image according to the denoising guidance data and the target posture information to obtain a target generated image, including: Performing posture structure analysis on the target posture information according to the target posture adaptation model to obtain posture structure guidance data; According to the target convolutional neural model, the noisy simulated image is gradually denoised based on the denoising guidance data and the posture structure guidance data to obtain the target generated image.

4. The method according to claim 1, characterized in that: Before performing denoising guidance analysis on the multi-scale appearance feature data through the target denoising guidance sub-model of the target image generation model to obtain denoising guidance data, the method further includes: Acquire training set data, wherein the training set data includes: training posture information and training images corresponding to the training posture information; Preprocessing the training image to obtain multi-scale training feature data and a noisy training image; According to the original denoising guidance sub-model of the original image generation model, performing denoising guidance analysis on the multi-scale training feature data to obtain training guidance data; According to the original convolution denoising sub-model of the original image generation model, the noisy training image is denoised according to the training guidance data and the training posture information to obtain an original generated image; The parameters of the original image generation model are loss-optimized according to the original generated image and the preset optimization model to obtain the target image generation model.

5. The method according to claim 4, characterized in that The preprocessing of the training image to obtain multi-scale training feature data and a noisy training image includes: Performing multi-scale feature mapping on the training image to obtain the multi-scale training feature data; Perform random noise simulation on the training image to obtain the noisy training image.

6. The method according to claim 4, characterized in that The method further comprises: Performing loss optimization on the parameters of the original image generation model according to the original generated image and the preset optimization model to obtain a candidate image generation model; Performing a performance evaluation on the candidate image generation model to obtain a performance evaluation result; If the performance evaluation result meets the preset conditions, the candidate image generation model is used as the target image generation model.

7. An image generating device, characterized in that: The device comprises: An acquisition module, used to acquire target posture information and source images; A mapping module, used for performing multi-scale feature mapping on the source image to obtain multi-scale appearance feature data; A noise simulation module, used for performing random noise simulation on the source image to obtain a noisy simulated image; A guidance analysis module, configured to perform denoising guidance analysis on the multi-scale appearance feature data according to a target denoising guidance sub-model of a target image generation model to obtain denoising guidance data; An image generation module is used to perform denoising processing on the noisy simulated image according to the target convolution denoising sub-model of the target image generation model, according to the denoising guidance data and the target posture information, to obtain a target generated image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images