Posture guide image synthesis method and device, electronic equipment and storage medium

Abstract

The invention relates to a posture guide image synthesis method and device, electronic equipment and a storage medium. The posture guide image synthesis method comprises the steps of extracting a posture feature map to generate a first condition code and embedding the first condition code into each down-sampling module; extracting a multi-scale feature map to generate a second condition code, and embedding the second condition code into each up-sampling module; obtaining multi-scale refined appearance features to generate third condition codes, and embedding the third condition codes into each down-sampling module and each up-sampling module; and inputting the noise image into the Unet model, and synthesizing a target image based on the first, second and third condition codes. A first conditional code is embedded into a down-sampling module, a second conditional code is embedded into an up-sampling module, and a third conditional code is embedded into the up-sampling module and the down-sampling module, so that a model gradually introduces richer detail information in different stages of image reconstruction, and a high-quality image with a specific posture is generated. Meanwhile, based on the diffusion model, the model ensures the image quality, reduces the use of computing resources, and is easier to be used for products and landing.

Description

Technical Field

[0001] The present invention relates to the field of image synthesis, and more specifically, to a gesture-guided image synthesis method, device, electronic device and storage medium. Background Art

[0002] Pose-guided character image synthesis is a computer vision and deep learning technology that aims to generate new portrait images that match the target pose based on preset character poses and conditional images. In this process, the model not only accurately adjusts the character's body posture, but also retains and integrates visual features such as clothing details and facial features to ensure that the character's identity and other specific attributes remain unchanged while changing the pose. Compared with traditional image capture and post-editing processing, pose-guided character image synthesis can quickly generate a large number of images in different poses, greatly improving work efficiency. At the same time, users can customize target poses and scenes to generate unique images as needed to meet the needs of personalized image generation.

[0003] Although posture-guided character image synthesis technology has significant advantages, it faces many technical challenges. For example, although adversarial networks can generate high-quality images, the training cost is high and they are easily affected by data bias; variational autoencoders are stable in training, but the generated images are not realistic enough and may lack details; progressive diffusion models can gradually optimize image quality, but they have high requirements for computing resources and poor generalization for rare postures. These image synthesis methods all have the problems of large model parameters, slow inference speed, and high hardware requirements, which make it difficult to implement products. Summary of the invention

[0004] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a posture-guided image synthesis method to solve the problems existing in the current technology of insufficient realism of synthesized images and lack of in-depth understanding of the high-level semantics of character images.

[0005] According to a first aspect of the present application, a posture-guided image synthesis method is provided, the method is based on a diffusion model, the diffusion model includes a Unet module, the Unet module includes an encoder and a decoder, the encoder includes a plurality of downsampling modules, the decoder includes upsampling modules with the same number as the downsampling modules, the method includes: Acquire a source image, extract a posture feature map according to the source image, generate a first conditional code based on the posture feature map, and embed the first conditional code into each of the downsampling modules; Extracting a multi-scale feature map according to the source image, generating a second conditional code based on the multi-scale feature map, and embedding the second conditional code into each of the upsampling modules; Extracting a multi-scale feature map according to the source image, generating a second conditional code based on the multi-scale feature map, and embedding the second conditional code into each of the upsampling modules; Acquire a multi-scale refined appearance feature based on the multi-scale feature map, generate a third conditional code based on the multi-scale refined appearance feature, and embed the third conditional code into each of the down-sampling modules and each of the up-sampling modules; A noise image of the source image is obtained, and the noise image is input into the Unet model, and the Unet model synthesizes a target image based on the first conditional coding, the second conditional coding, and the third conditional coding.

[0006] By embedding the posture feature map as the first conditional coding into each downsampling module of the encoder, the model can generate image content that matches the given posture; by embedding the second conditional coding generated by the multi-scale feature map into each upsampling module of the decoder, richer detail information can be gradually introduced at different stages of image reconstruction; by embedding the third conditional coding generated by multi-scale refined appearance features into all up- and down-sampling modules of the encoder and decoder, the model can continuously consider the coherence and consistency of the appearance information during the entire image synthesis process, thereby reducing artifacts, color inconsistency and other problems in the synthesized image, and improving the overall quality of the generated image.

[0007] Optionally, the diffusion model also includes a posture feature extraction module for extracting a posture feature map based on the source image, and the posture feature extraction module includes a PixelUnshuffle module, a first convolutional layer, a first residual module, a second convolutional layer, a second residual module, a third convolutional layer, and a third residual module connected in sequence.

[0008] The PixelUnshuffle module can reduce the resolution of the image and thus reduce the computational effort of subsequent processing. It can also indirectly enhance the local receptive field of the network, enabling the model to better capture the local structure and details in the image.

[0009] Optionally, the first residual module, the second residual module and the third residual module each include two branches, one branch passes through the fourth convolutional layer, and the other branch passes through two fifth convolutional layers connected in series.

[0010] Optionally, the diffusion model also includes a multi-scale feature extraction module for extracting a multi-scale feature map based on the source image, the multi-scale feature extraction module includes a StemBlock module and three StageBlock modules; the StemBlock module includes a number of sixth convolutional layers; the StageBlock module evenly divides the number of input channels of the diffusion model into two parts to obtain a short branch Shortout and a main branch Mainout.

[0011] Optionally, the diffusion model further includes a perceptual refinement decoder for acquiring multi-scale refined appearance features based on the multi-scale feature map, the perceptual refinement decoder includes a plurality of decoding modules connected in series, each of the decoding modules includes a CrossAttention module and a GAU module; the acquiring multi-scale refined appearance features based on the multi-scale feature map includes the following steps: Perform a Flatten operation on the multi-scale feature map to obtain a one-dimensional sequence ; Randomly initialized learnable labels , for the learnable marker Perform Flatten operation to obtain a one-dimensional sequence ; The one-dimensional sequence and one-dimensional sequence The input perceptual refinement decoder is processed in sequence by a number of CrossAttention modules and GAU modules of the decoding modules connected in series to obtain multi-scale refined appearance features.

[0012] Optionally, the attention calculation formula of the GAU module is as follows: Among them, s is the preset step size, By respectively Input GAU module and pass through the learnable projection matrix , , Mapping obtained, is the activation function, is the global attention function of the GAU module.

[0013] Optionally, the acquiring a noise image of the source image and inputting the noise image into the Unet model includes: With random Gaussian noise As the initial potential noise , and the initial potential noise Mapping to different time steps Potential noise Input the Unet model, the potential noise The calculation formula is as follows: in, is the time step The noise level coefficient at yes The potential noise of the step, is random Gaussian noise sampled from a standard normal distribution.

[0014] Using random Gaussian noise as the initial potential noise, the distribution characteristics of Gaussian noise can flexibly control the blur degree and detail richness of the generated image.

[0015] According to a second aspect of the present application, a gesture-guided image synthesis device is provided, the device comprising: The posture feature encoding and embedding module is used to obtain a source image, extract a posture feature map according to the source image, generate a first conditional encoding based on the posture feature map, and embed the first conditional encoding into each of the downsampling modules.

[0016] A multi-scale feature coding and embedding module is used to extract a multi-scale feature map according to the source image, generate a second conditional code based on the multi-scale feature map, and embed the second conditional code into each of the upsampling modules.

[0017] A multi-scale refined appearance feature encoding and embedding module is used to obtain multi-scale refined appearance features based on the multi-scale feature map, generate a third conditional code based on the multi-scale refined appearance features, and embed the third conditional code into each of the downsampling modules and each of the upsampling modules.

[0018] An image synthesis module is used to obtain a noise image of the source image, input the noise image into the Unet model, and the Unet model synthesizes a target image based on the first conditional coding, the second conditional coding and the third conditional coding.

[0019] According to a third aspect of the present application, an electronic device is provided, including: a memory for storing one or more computer programs; The processor implements the gesture-guided image synthesis method described in the first aspect when the one or more computer programs are executed by the processor.

[0020] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the posture-guided image synthesis method described in the first aspect when executed.

[0021] Based on any of the above aspects, the posture-guided image synthesis method, device, electronic device and computer storage medium provided in the embodiments of the present application, by embedding the posture feature map as the first conditional coding into each downsampling module of the encoder, enables the model to generate high-quality images of specific postures; by embedding the second conditional coding generated by the multi-scale feature map into each upsampling module of the decoder, the model can gradually introduce richer detail information at different stages of image reconstruction. By embedding the third conditional coding generated by multi-scale refined appearance features into all up- and down-sampling modules of the encoder and decoder, the model can continuously consider the coherence and consistency of the appearance information throughout the image synthesis process, thereby reducing artifacts, color inconsistencies and other problems in the synthesized image, and further improving the quality of the generated image. The image synthesis method based on the diffusion model can reduce the use of computing resources while ensuring image quality, making the model easier to use in products and implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of an application scenario of the posture-guided image synthesis method provided in this embodiment.

[0024] Figure 2 This is a flowchart of the gesture-guided image synthesis method provided in this embodiment.

[0025] Figure 3 A schematic diagram of the overall structure of the diffusion model provided in this embodiment.

[0026] Figure 4 A schematic diagram of the structure of a human posture feature extraction module provided in this embodiment.

[0027] Figure 5 A schematic diagram of the structure of the residual module provided in this embodiment.

[0028] Figure 6 A schematic diagram of the structure of a multi-scale feature extraction module provided in this embodiment.

[0029] Figure 7A schematic diagram of the structure of the StemBlock module provided in this embodiment.

[0030] Figure 8 A schematic diagram of the structure of the CSPLayer module provided in this embodiment.

[0031] Fig. 9 A schematic diagram of the structure of a posture-guided image synthesis device provided in this embodiment.

[0032] Fig.10 A schematic diagram of the structure of an electronic device provided by this embodiment. DETAILED DESCRIPTION

[0033] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] The inventors have found that in the existing posture-guided character image generation technology, the conditional generative adversarial network requires a lot of computing resources and is easily affected by the bias of training data, resulting in unstable image quality; the variational autoencoder generates images with low fidelity and lacks details; although the progressive conditional diffusion model can generate high-quality images, the network structure is complex, the computing resource requirements are high, and the generalization ability for rare postures is poor, and it is easy to overfit. Therefore, how to balance image quality, computing efficiency and model generalization ability, reduce the number of model parameters, improve the inference speed, and enhance the adaptability to various postures, so that the model can be applied and implemented in resource-constrained environments is a problem that needs to be solved in current research.

[0037] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0038] For example, a schematic diagram of an application scenario of a posture-guided image synthesis method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has an image processing function and can also have a data transmission function for video streams and audio streams; the terminal device 200 has a streaming media playback function and can also have an image processing function.

[0039] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0040] In one practicable manner, the server 100 and the terminal 200 may respectively execute the posture-guided image synthesis method provided in the embodiment of the present application, or, optionally, the posture-guided image synthesis method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0041] This embodiment provides a posture-guided image synthesis method based on a diffusion model, wherein the diffusion model includes a Unet module. Figure 3 As shown, the Unet module includes an encoder and a decoder, the encoder includes several downsampling modules, and the decoder includes upsampling modules corresponding to the number of the downsampling modules.

[0042] In this embodiment, the encoder includes three downsampling layers, down1, down2, and down3, which gradually reduce the spatial dimensions of the feature map through the three downsampling layers, and correspondingly increase the number of feature channels, so that the model can capture more abstract and deep image features. The goal of the decoder is to gradually reconstruct the image from the high-level features extracted by the encoder. The decoder includes three upsampling layers, up1, up2, and up3, which gradually restore the spatial dimensions of the feature map through the three upsampling layers, so that the final output image is both accurate and clear. In the Unet model, by embedding additional information in the encoder and decoder so that the model can understand and respond to these specific conditions, for example, by encoding specific conditions in the form of text descriptions, the model can achieve conditional image generation, greatly enhancing the flexibility and breadth of the model.

[0043] like Figure 2 As shown, this embodiment provides a posture-guided image synthesis method, which may include the following steps: S1. Acquire a source image, extract a posture feature map according to the source image, generate a first conditional code based on the posture feature map, and embed the first conditional code into each of the downsampling modules.

[0044] like Figure 3 As shown, the diffusion model includes a posture feature extraction module for extracting a posture feature map according to the source image. Figure 4 As shown, the human posture feature extraction module includes a PixelUnshuffle module, a first convolutional layer, a first residual module, a second convolutional layer, a second residual module, a third convolutional layer, and a third residual module, which are connected in sequence.

[0045] Preferably, the downsampling factor of the PixelUnshuffle module is 4, the convolution kernel size of the first convolution layer is 1, and the convolution kernel sizes of the second and third convolution layers are both 3 and the step sizes are both 2.

[0046] In the specific implementation process, the 2D skeleton information is input into the human posture feature extraction module, passes through the PixelUnshuffle module, and the first convolution layer, and then passes through the first residual module to obtain the first feature Feature1. After passing through the first residual module, the first feature Feature1 is downsampled through the second convolution layer, and then passes through the second residual module to obtain the second feature Feature2. After passing through the second residual module, the second feature Feature2 is downsampled through the third convolution layer, and then passes through the third residual module to obtain the third feature Feature3. The first feature Feature1, the second feature Feature2, and the third feature Feature3 correspond to feature information of different scales of human posture, respectively.

[0047] like Figure 5 As shown, the residual module of this embodiment includes two branches, one of which passes through the fourth convolutional layer, and the other branch passes through two fifth convolutional layers connected in series. In specific implementation, the output of the residual module is obtained by adding the processed outputs of the two branches.

[0048] Preferably, the convolution kernel size of the fourth convolution layer is 1, and the convolution kernel size of the fifth convolution layer is 3.

[0049] S2. Extract a multi-scale feature map according to the source image, generate a second conditional code based on the multi-scale feature map, and embed the second conditional code into each of the upsampling modules.

[0050] In this embodiment, the diffusion model also includes a multi-scale feature extraction module for extracting a multi-scale feature map according to the source image. The multi-scale feature extraction module adopts a CSPNeXt structure, including a StemBlock module and three StageBlock modules. Among them, the StemBlock module includes a number of sixth convolutional layers; the StageBlock module evenly divides the number of input channels of the diffusion model into two parts to obtain a short branch Shortout and a main branch Mainout. The number of sixth convolutional layers is preferably 3, and the convolution kernel size of each sixth convolutional layer is 3.

[0051] When implementing it, Figure 7 As described above, the output of the main branch Mainout is input into the CSPLayer module for processing, and the output of the CSPLayer module is Cat-joined with the output of the short branch Shortout, and then processed by the channel attention mechanism and a convolution layer with a convolution kernel of 1x1 to obtain the output value. Through the above process, the multi-scale feature extraction module extracts a multi-scale feature map Fs = [f1, f2, f3, f4] containing multi-level details from the source image, providing a basis for coarse-to-fine appearance control for subsequent image generation.

[0052] like Figure 8 As shown, in an optional implementation, the CSPLayer module includes a convolution layer and a depth-separable convolution layer connected in sequence. In this embodiment, the convolution kernel size of the convolution layer of the CSPLayer module is 3x3, and the convolution kernel size of the depth-separable convolution layer is 5x5. The input value of the CSPLayer module is processed by the convolution layer and the depth-separable convolution layer in sequence to obtain the corresponding eigenvalue, and the obtained eigenvalue is added to the input value to obtain the output value of the CSPLayer module.

[0053] In order to improve the efficiency of image feature extraction while maintaining the advantage of model inference speed and making the model easy to deploy, this embodiment uses the ReLU6 function as the activation function of the StageBlock module. The specific formula of the ReLU6 function is as follows: The ReLU6 function formula indicates that if the input value x is a negative number, the output is 0; if the input value x is between 0 and 6, the output is x directly; if the input value x is greater than 6, the output is 6. By limiting the maximum output value of the function to 6, the gradient explosion problem that may occur in deep networks can be avoided, thereby maintaining the numerical stability of the model. In addition, since the ReLU6 function is simple to calculate, it can speed up the training and reasoning of the model, allowing the module to be more efficiently deployed in embedded devices.

[0054] S3. Acquire a multi-scale refined appearance feature based on the multi-scale feature map, generate a third conditional code based on the multi-scale refined appearance feature, and embed the third conditional code into each of the down-sampling modules and each of the up-sampling modules.

[0055] In this embodiment, the diffusion model further includes a perceptual refinement decoder for obtaining multi-scale refined appearance features based on the multi-scale feature map. The perceptual refinement decoder includes a plurality of decoding modules connected in series, each of which includes a CrossAttention module and a GAU module; like Figure 6 As shown, in this embodiment, the perceptual refinement decoder includes four decoding modules, each of which includes a CrossAttention module and a GAU module. Specifically, the perceptual refinement decoder adopts a cascade architecture, and the four decoding modules are connected in series in sequence. Each decoding module is responsible for further refining the output of the previous decoding module to gradually improve the decoding quality, accuracy and detail performance. Acquiring multi-scale refined appearance features based on the multi-scale feature map includes the following steps: Perform a Flatten operation on the multi-scale feature map F4 to obtain a one-dimensional sequence ; Randomly initialized learnable labels , for the learnable marker Perform Flatten operation to obtain a one-dimensional sequence ; The one-dimensional sequence and one-dimensional sequence The input-aware refinement decoder obtains multi-scale refined appearance features.

[0056] For each of the decoding modules, the input feature Z is input into the CrossAttention module to obtain a one-dimensional sequence Specifically, the input features Z are respectively transformed through the learnable projection matrix , , (C is the dimension of the input feature Z) to obtain the corresponding query vector Q, key vector K and value vector V. The calculation formulas for the query vector Q, key vector K and value vector V are as follows: The attention calculation formula of the CrossAttention module is as follows: The one-dimensional sequence output by the CrossAttention module Input GAU module, through the new learnable projection matrix , , , get the corresponding Q, K, V, and take the step size s as 128. The attention calculation formula of the GAU module is as follows: In this embodiment, the learnable projection matrix , , Random initialization can be performed before model training, and optimization and update can be performed through the back propagation algorithm during model training.

[0057] S4. Obtain a noise image of the source image, input the noise image into the Unet model, and synthesize a target image based on the first conditional coding, the second conditional coding, and the third conditional coding by the Unet model.

[0058] This embodiment generates a noise image by gradually adding Gaussian noise to the source image. Specifically, by setting the step size The forward diffusion process is based on random Gaussian noise. As the initial potential noise , the initial potential noise Mapping to different time steps Get the potential noise , the potential noise The corresponding noise image is obtained by conversion. Potential noise The calculation formula is as follows: in, is the time step The noise level coefficient at yes The potential noise of the step, is random Gaussian noise sampled from a standard normal distribution.

[0059] The potential noise The corresponding noise image captures the key information and features in the data through three consecutive downsampling layers, and finally outputs a highly generalized multidimensional tensor through layer-by-layer compression and abstraction; after the multidimensional tensor is input into the decoder, it passes through three consecutive upsampling layers and finally outputs a multidimensional feature tensor, which is a prediction and reconstruction of the information in the original potential noise. After obtaining the multidimensional feature tensor, the predicted noise is subjected to inverse noise mapping processing to finally obtain a synthesized image.

[0060] like Fig. 9 As shown, the embodiment of the present application further provides a posture guidance image synthesis device 610. Optionally, the posture guidance image synthesis device 610 may include: The posture feature coding and embedding module 611 is used to obtain a source image, extract a posture feature map according to the source image, generate a first conditional coding based on the posture feature map, and embed the first conditional coding into each of the downsampling modules.

[0061] The multi-scale feature coding and embedding module 612 is used to extract a multi-scale feature map according to the source image, generate a second conditional code based on the multi-scale feature map, and embed the second conditional code into each of the upsampling modules.

[0062] The multi-scale refined appearance feature encoding and embedding module 613 is used to obtain multi-scale refined appearance features based on the multi-scale feature map, generate a third conditional code based on the multi-scale refined appearance features, and embed the third conditional code into each of the downsampling modules and each of the upsampling modules.

[0063] The image synthesis module 614 is used to obtain a noise image of the source image, input the noise image into the Unet model, and synthesize the target image based on the first conditional coding, the second conditional coding and the third conditional coding by the Unet model.

[0064] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A posture-guided image synthesis device provided in an embodiment of the present application can execute a posture-guided image synthesis method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the posture-guided image synthesis device can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software modules.

[0065] Specifically, each step of the method embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor. The steps of the posture-guided image synthesis method in the embodiment of the present application can be directly embodied as a hardware encoding processor, or a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0066] The present application embodiment provides an electronic device 710, whose structure is as follows: Fig.10 The electronic device 710 may be Figure 1 The server 100 or the terminal 200 is shown.

[0067] like Fig.10 As shown, the electronic device 710 includes a memory 711, a processor 712, a communication module 713 and an input / output interface 714, etc. Optionally, the memory 711, the processor 712, the communication module 713 and the input / output interface 714 can be connected and communicated through a bus 715.

[0068] The memory 711 is used to store one or more computer programs and transmit the code of the computer program to the processor 712; when the one or more computer programs are executed by the processor 711, the posture-guided image synthesis method in the embodiment of the present application is implemented.

[0069] Optionally, the electronic device 710 can be connected to a network via a communication module 713 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 710 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 710 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0070] Optionally, the electronic device 710 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 714. The electronic device 710 itself can have a display device, and can also be connected to other display devices through the input / output interface 714. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 714, so that data in the electronic device 710 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 711. It can be understood that the input / output interface 714 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 714 can be a component of the electronic device 710, or it can be an external device connected to the electronic device 710 when needed.

[0071] Optionally, the memory 711 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0072] Optionally, the computer program stored in the processor 711 may be divided into one or more modules, which are stored in the memory 711 and executed by the processor 712 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 710.

[0073] Optionally, the processor 712 may be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 712 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 712 executes various methods and processes of this embodiment, illustratively, such as a gesture-guided image synthesis method in an embodiment of the present application.

[0074] Optionally, the bus 715 may include a path for transmitting information. The bus 715 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 715 may be divided into an address bus, a data bus, a control bus, etc.

[0075] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 711 of the electronic device 710. When the computer program is executed by the processor 712, one or more steps of a posture-guided image synthesis method of the embodiment of the present application can be performed.

[0076] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0077] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.

[0078] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A posture-guided image synthesis method, the method is based on a diffusion model, the diffusion model includes a Unet module, the Unet module includes an encoder and a decoder, the encoder includes a number of downsampling modules, the decoder includes upsampling modules with the same number as the downsampling modules, characterized in that: The method comprises: Acquire a source image, extract a posture feature map according to the source image, generate a first conditional code based on the posture feature map, and embed the first conditional code into each of the downsampling modules; Extracting a multi-scale feature map according to the source image, generating a second conditional code based on the multi-scale feature map, and embedding the second conditional code into each of the upsampling modules; Acquire a multi-scale refined appearance feature based on the multi-scale feature map, generate a third conditional code based on the multi-scale refined appearance feature, and embed the third conditional code into each of the down-sampling modules and each of the up-sampling modules; A noise image of the source image is obtained, and the noise image is input into the Unet module, and the Unet module synthesizes a target image based on the first conditional coding, the second conditional coding, and the third conditional coding.

2. A posture-guided image synthesis method according to claim 1, characterized in that: The diffusion model also includes a posture feature extraction module for extracting a posture feature map according to the source image, and the posture feature extraction module includes a PixelUnshuffle module, a first convolutional layer, a first residual module, a second convolutional layer, a second residual module, a third convolutional layer, and a third residual module connected in sequence.

3. A posture-guided image synthesis method according to claim 2, characterized in that: The first residual module, the second residual module and the third residual module each include two branches, one of which passes through the fourth convolutional layer, and the other branch passes through two fifth convolutional layers connected in series.

4. A posture-guided image synthesis method according to claim 1, characterized in that: The diffusion model also includes a multi-scale feature extraction module for extracting a multi-scale feature map according to the source image, and the multi-scale feature extraction module includes a StemBlock module and three StageBlock modules; the StemBlock module includes a plurality of sixth convolutional layers; the StageBlock module evenly divides the number of input channels of the diffusion model into two parts to obtain a short branch Shortout and a main branch Mainout.

5. The method for synthesizing a posture-guided image according to claim 1, characterized in that: The diffusion model further includes a perceptual refinement decoder for obtaining multi-scale refined appearance features based on the multi-scale feature map, wherein the perceptual refinement decoder includes a plurality of decoding modules connected in series, each of which includes a CrossAttention module and a GAU module; and obtaining multi-scale refined appearance features based on the multi-scale feature map includes the following steps: Perform a Flatten operation on the multi-scale feature map to obtain a one-dimensional sequence ; Randomly initialized learnable labels , for the learnable marker Perform Flatten operation to obtain a one-dimensional sequence ; The one-dimensional sequence and one-dimensional sequence The input perceptual refinement decoder is processed in sequence by a number of CrossAttention modules and GAU modules of the decoding modules connected in series to obtain multi-scale refined appearance features.

6. A posture-guided image synthesis method according to claim 5, characterized in that: The attention calculation formula of the GAU module is as follows: in, is the preset step size, By transforming the one-dimensional sequence Input GAU module and pass through the learnable projection matrix , , Mapping obtained, is the activation function, is the global attention function of the GAU module.

7. A posture-guided image synthesis method according to claim 1, characterized in that: The step of obtaining the noise image of the source image comprises the following steps: With random Gaussian noise As the initial potential noise , the initial potential noise Mapping to different time steps Get the potential noise , the potential noise Convert to obtain the corresponding noise image; The potential noise The calculation formula is as follows: in, is the time step The noise level coefficient at yes The potential noise of the step, is random Gaussian noise sampled from a standard normal distribution.

8. A posture-guided image synthesis device, characterized in that: The device comprises: A posture feature coding and embedding module, used for acquiring a source image, extracting a posture feature map according to the source image, generating a first conditional coding based on the posture feature map, and embedding the first conditional coding into each of the downsampling modules; A multi-scale feature coding and embedding module, used for extracting a multi-scale feature map according to the source image, generating a second conditional code based on the multi-scale feature map, and embedding the second conditional code into each of the upsampling modules; A multi-scale refined appearance feature encoding and embedding module, used for acquiring multi-scale refined appearance features based on the multi-scale feature map, generating a third conditional code based on the multi-scale refined appearance features, and embedding the third conditional code into each of the down-sampling modules and each of the up-sampling modules; An image synthesis module is used to obtain a noise image of the source image, input the noise image into the Unet model, and the Unet model synthesizes a target image based on the first conditional coding, the second conditional coding and the third conditional coding.

9. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the gesture-guided image synthesis method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the gesture-guided image synthesis method according to any one of claims 1 to 7 when executed.

Images