Image generation method based on style content adaptive normalization pose guidance
By employing a style-content adaptive normalization pose-guided image generation method, which utilizes human keypoint detection and edge mapping information, combined with a multi-scale content transfer network and a style-adaptive normalization generator, the problem of unrealistic texture details in human image pose transfer is solved, achieving fast and efficient image generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DALIAN NATIONALITIES UNIVERSITY
- Filing Date
- 2022-12-12
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies struggle to preserve the source style and spatial context during pose transfer of human images, resulting in unrealistic texture details in the generated images, long training times, and slow convergence speeds.
An image generation method based on style-content adaptive normalization pose guidance is adopted. Through human keypoint detection, edge mapping information extraction, multi-scale content transfer network, optical flow estimation and style adaptive normalization generator, a target person image with the same style and target pose as the source image is generated.
It improves the accuracy of pose transfer and the realism of character appearance, reduces training time, enhances the generation of texture details, and improves the convergence speed of the network.
Smart Images
Figure HDA0003994247610000011 
Figure HDA0003994247610000021
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image synthesis technology, specifically an image generation method based on style content adaptive normalization pose guidance. Background Technology
[0002] Pose-guided human image transformation is an image generation task that synthesizes arbitrary target poses using source images of people as conditions. This topic has many potential applications, such as video generation and virtual try-on. Furthermore, as research into human behavior using deep learning deepens, the demand for human image data has surged. Therefore, human pose transfer provides the necessary data for these studies, offering a wealth of data for further research on human behavior.
[0003] In recent years, significant progress has been made in converting source images into target poses using conditional GANs. These methods are all based on conditional GANs, inserting multiple repeating modules to learn the correspondence between poses through neural networks, and recombine the source image features into an image with the target pose. However, these methods cannot preserve the relationship between the source style and spatial context, making it difficult to predict clear and reasonable target images. To address this issue, flow-based methods predict the offset between the source and target positions, guiding the distortion of source features into a reasonable target pose, resulting in more accurate and realistic texture images. However, the large changes in pose between the source and target produce noticeable artifacts. To alleviate the misalignment problem caused by large pose changes, some methods introduce human body analytic mapping to provide semantic relationships corresponding to the target pose to synthesize target images that approximate the source style. While these methods synthesize relatively satisfactory human images, they still do not generate realistic texture details. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention provides an image generation method based on style content adaptive normalization pose guidance, which aims to improve the accuracy of pose transfer and the realism of human appearance. It effectively synthesizes realistic human appearance images, reduces training time, and accelerates convergence speed while ensuring image quality.
[0005] The technical solution adopted by this invention to solve its technical problem is: an image generation method based on style content adaptive normalization posture guidance, which involves inputting a person image, selecting a source image and a target image from the person image, and generating a target person image with the same style as the source image and the same posture as the target image. Specifically, it includes the following steps:
[0006] S1: Perform human key point detection on the human image to obtain a pose heatmap;
[0007] S2: Extract the edge mapping information of the human body in the image to obtain the edge map;
[0008] S3: Randomly select two images from the person image as the source image and the target image respectively. Based on the obtained edge map and pose heatmap, predict the edge map of the target image through an aligned multi-scale content transfer network.
[0009] S4: Input the pose heatmaps of the source image and the target image into the optical flow estimation model to obtain the optical flow map and occlusion mask information between the source image and the target image;
[0010] S5: Input the optical flow map, occlusion mask information, target image pose heatmap, and source image into the local attention model to obtain a rough target image;
[0011] S6: Input the above rough target image, target image edge map and source image into the style adaptive normalization generator to obtain the final pose transfer target image.
[0012] Furthermore, in step S1, the pose heatmap of the person image in 18 channels is estimated using the OpenPose method, which includes 18 key points: nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, left knee, left ankle, right hip, right knee, right ankle, left eye, right eye, left ear, and right ear. Each key point is represented by one channel, and the key points are interconnected to form the skeletal structure of the human body.
[0013] Furthermore, in step S2, the extended Gaussian difference edge detection method is used to extract the edge mapping information of the person image to obtain the black and white grayscale source edge map of the human body in the person image.
[0014] Furthermore, the aligned multi-scale content transfer network described in step S3 consists of an aligned multi-scale transfer decoder and three encoders; each encoder consists of a downsampling layer, an instance normalization layer, an activation layer, and a residual block; the aligned multi-scale transfer decoder consists of a deconvolution layer, an instance normalization layer, an activation function, and a residual block. The deconvolution layer uses a 4x4 convolution kernel with a stride of 2 and a margin of 1; the source image edge map, the source image pose heatmap, and the target image pose heatmap are input into the encoder to obtain a feature map with 256 channels and a size of 32x32. After attention calculation, the target edge map is decoded.
[0015] Furthermore, the encoder uses a 4x4 convolutional kernel for its downsampling layer with a stride of 2 and a margin of 1; the residual block consists of two convolutional layers, two instance normalization layers, and one activation layer. The convolutional layers use a 3x3 convolutional kernel with a stride of 1 and a margin of 1; each convolutional layer is followed by an instance normalization layer, and a ReLU activation function is added after the first instance normalization layer.
[0016] Furthermore, the optical flow estimation model described in step S4 consists of an encoder and a decoder. The encoder consists of an upsampling layer and a convolutional layer. Each layer is preceded by an instance normalization layer and an activation function layer. The upsampling layer uses a 4x4 convolutional kernel with a stride of 2, and the convolutional layer uses a 3x3 convolutional kernel with a stride of 1. The source image, the source image pose heatmap, and the target image pose heatmap are fused in the channel dimension and then passed through the encoder to obtain a feature map with 256 channels and a size of 32x32. The decoder outputs a two-dimensional flow field optical flow map with 2 channels and an occlusion mask information with 1 channel.
[0017] Furthermore, in step S5, local feature patch pairs are extracted from the source and target images based on the flow field. A context-aware sampling kernel is calculated using a kernel prediction network, and finally, the source features are sampled to obtain the distortion result at the sampling location. A 3x3 convolutional kernel is used to extract the local feature patch pairs. The kernel prediction network consists of convolutional layers, activation layers, and softmax, obtaining the local correlation between the source and target, and guiding the deformation of the source's local features.
[0018] Furthermore, the style adaptive normalization generator described in step S6 consists of a pose encoder, a style encoder, a residual block, a style adaptive normalization module, and a residual decoder. The pose encoder consists of a 4x4 convolutional kernel, an upsampling layer with a stride of 2, and a 3x3 convolutional kernel with a stride of 1. Each layer is preceded by an instance normalization layer and an activation function layer. The style encoder consists of a 4x4 convolutional kernel, an upsampling layer with a stride of 2, and a 3x3 convolutional kernel with a stride of 1. The layers combine convolution and self-attention, each preceded by an instance normalization layer and an activation function layer; the residual block consists of two convolutional layers, each consisting of an activation layer, an instance normalization layer, and a convolutional layer with a 3x3 kernel; the residual decoder consists of a convolutional layer with a 3x3 kernel and a transposed convolutional layer with a 4x4 kernel; the style adaptive normalization module consists of three region adaptive normalization layers, each modulating the input feature parameters with two spatially adaptive normalization parameters.
[0019] The beneficial effects of the present invention include: 1) using edge mapping as an additional constraint on the pose heatmap to solve the problem of insufficient content information, thereby guiding the network to enhance texture details and generate more realistic human images.
[0020] 2) The style adaptive normalization generator explicitly distributes the style features of the source image to the target pose and injects the source style style layer by layer, thus preserving the real texture information.
[0021] 3) A novel aligned multi-scale content transfer network is proposed, which can distort and rationally recombine input data at the feature level, not only generating new content but also enhancing the convergence speed of the network. Attached Figure Description
[0022] Figure 1 This is a flowchart of the overall method of the present invention;
[0023] Figure 2 This is a structural diagram of the overall model of the present invention. Detailed Implementation
[0024] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] This invention provides an image generation method based on style-content adaptive normalization pose guidance. First, by using an aligned multi-scale content transfer network to predict the target edge map, pose information is pre-transferred, preserving texture content and mitigating spatial misalignment. Second, a style-texture transfer network is used to progressively transfer source style features to the target pose and achieve a reasonable arrangement. This is implemented by a style-adaptive normalization generator, which maps source style features, target pose, and edges into the same latent space. By adaptively adjusting the source style and target pose, the consistency of style texture and content is enhanced, thereby preserving source style features and enhancing the texture details of the generated target.
[0026] Example 1
[0027] An image generation method based on style-content adaptive normalized pose guidance takes a person image as input, selects a source image and a target image from the person image, and generates a target person image with the same style as the source image and the same pose as the target image.
[0028] like Figure 1 As shown, firstly, the key point coordinates of the person in the image are extracted, i.e., the pose heatmap, and then the edge map of the image is extracted. A source image and a target image are selected from the image. A new target image edge map is generated based on the pose heatmap and edge map of the source image and the pose heatmap of the target image. The correspondence between poses is calculated based on the key point coordinates of the source and target images, and the corresponding optical flow map and occlusion mask information are output. A coarse target person image is generated based on the optical flow map, occlusion mask information, the pose heatmap of the source image and the target image. Finally, the coarse target person image is refined using the target image pose heatmap, the generated target image edge map, and the source image to obtain a realistic target person image. Specifically:
[0029] 1) In the training set of human images, extract pose heatmaps and edge maps for all human images. Use the OpenPose method to extract 18 keypoints from the human images, each keypoint representing a joint of the human body. Use the Extended Difference of Gaussian (ADG) edge detection method to extract edge information from the human images, utilizing the contrast between black and white to highlight the texture details of the image. Select a pair of images from the training set as the source image and the target image.
[0030] 2) The key point information of the obtained human image is rendered with different colors to obtain a posture heatmap. The nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, left knee, left ankle, right hip, right knee, right ankle, left eye, right eye, left ear and right ear are all different colors and connected with corresponding lines to form a skeletal structure diagram that resembles the human body.
[0031] 3) such as Figure 2 As shown, feature information from the pose heatmap, edge map, and target image pose heatmap of the source image (256x256 resolution) is extracted. After passing through the input layer, a feature map with 64 channels and a resolution of 128x128 is obtained. This is then passed through an upsampling layer with a kernel of 4 and a stride of 2, and a convolutional layer with a kernel of 3 and a stride of 1, resulting in a feature map with 128 channels and a resolution of 64x64. Similarly, after passing through another upsampling layer and a convolutional layer, a feature map with 256 channels and a resolution of 32x32 is obtained. The 32x32 feature maps of the source and target images are weighted and summed to obtain a relation matrix. This matrix is then weighted and summed with the edge feature map of the source image of the same size to obtain a coarse target image edge feature map. This coarse target image edge feature map is then added to the pixel values of the source image edge feature map. The calculated result is input into a deconvolutional layer with a kernel size of 4 and a stride of 2, resulting in a target image edge feature map with 128 channels and a resolution of 64x64. The same operation is performed on the feature maps of the source image pose and the target image pose of the same size (64x64) to obtain the target image edge feature map with 64 channels and a resolution of 128x128. Then, it is passed through one deconvolution layer, 5 residual convolution layers with 3 kernels and a stride of 1, and an output layer with 1 kernel and a stride of 1. Finally, the target image edge map is generated by outputting the Tanh() function.
[0032] 4) The source image, source image pose heatmap, and target image pose heatmap are fused in the channel dimension to form a feature map with 39 channels and a resolution of 256x256. This feature map is then upsampled using a 4x4 convolutional kernel with a stride of 2, followed by a convolutional layer using a 3x3 convolutional kernel with a stride of 1, resulting in a feature map with 32 channels and a resolution of 128x128. This process is repeated to obtain feature maps with 64 channels and a resolution of 64x64, 128 channels and a resolution of 32x32, 256 channels and a resolution of 16x16, and 256 channels and a resolution of 8x8.
[0033] The feature map with 256 channels and an 8x8 resolution is passed through a residual convolutional layer with a kernel of 3 and a stride of 1. This is then added to a deconvolutional layer with 256 channels and a 16x16 resolution, also with a kernel of 3 and a stride of 2. These layers are then combined, and a flow field information layer with 2 channels and a size of 16x16 is output, along with a feature map with 1 channel. The sigmoid() function is then executed to obtain the occlusion mask. The feature map with 256 channels and a 16x16 resolution is passed through a deconvolutional layer with a kernel of 3 and a stride of 2, resulting in a feature map with 128 channels and a resolution of 32x32. This is added to a downsampled feature map of the same size, and deconvolution is performed again to output a flow field and occlusion mask of size 64x64.
[0034] 5) Use a 4x4 convolution kernel and an upsampling layer with a stride of 2 to extract pose features from the source and target images, respectively, to obtain feature maps of size 64x64 and 32x32. Use bilinear interpolation to warp the source feature map. After fusing the warped source image feature map and the target image pose feature channel dimensions, pass them through a convolutional layer with a kernel of 3 and a stride of 1 and the Softmax() function to obtain the attention matrix. Then, perform average pooling after weighted summation of the warped source feature map and the attention matrix.
[0035] 6) Using a coarse target image as input, the image passes through three residual convolutional layers. Then, the appearance and content are modulated and demodulated using the source image, target image, and target image edge map in the region adaptive normalization layer. The source image (32x32), target image edge map (64x64), and target image (128x128) are sequentially modulated using convolutional layers with a kernel size of 3x3 and a stride of 1. Similarly, the pose heatmaps of the source and target images are used to modulate the parameters. The coarse target image features are multiplied and added to the modulation parameters to obtain the modulated target features. Finally, the image passes through three transposed convolutional layers with a kernel size of 3x3, an output layer with a kernel size of 1x1 and a stride of 1, and the Tanh() function to obtain the final target image.
[0036] In summary, this invention discloses an image generation method based on style-content adaptive normalization pose guidance. 1) It proposes a novel two-stage network to decouple style and content, aiming to improve the accuracy of pose transfer and the realism of the character's appearance. 2) By using an aligned multi-scale content transfer network to predict the edge map of the target image, pose information is transferred in advance, preserving texture content and alleviating spatial misalignment. 3) A style-texture transfer network is used to progressively transfer source style features to the target pose and achieve a reasonable arrangement, implemented by a style adaptive normalization generator. This maps source style features, target pose, and edges to the same latent space, and enhances the consistency of style texture and content by adaptively adjusting the source style and target pose, thereby preserving source style features and enhancing the texture details of the generated target image. This invention generates character images with consistent target pose and preserved source image style texture, reducing training difficulty and accelerating model convergence.
[0037] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. An image generation method based on style-content adaptive normalized pose guidance, characterized in that, Input a person image, select a source image and a target image from the person image, and generate a target person image with the same style as the source image and the same pose as the target image. The specific steps include the following: S1: Perform human key point detection on the human image to obtain a pose heatmap; S2: Extract the edge mapping information of the human body in the image to obtain the edge map; S3: Randomly select two images from the person image as the source image and the target image respectively. Based on the pose heatmap and edge map of the source image and the pose heatmap of the target image, predict the edge map of the target image through an aligned multi-scale content transfer network. S4: Input the source image, source image pose heatmap and target image pose heatmap into the optical flow estimation model to obtain the optical flow map and occlusion mask information between the source image and the target image; S5: Input the optical flow map, occlusion mask information, target image pose heatmap, and source image into the local attention model to obtain a rough target image; S6: Input the above rough target image, target image edge map and source image into the style adaptive normalization generator to obtain the final pose transfer target image.
2. The image generation method based on style content adaptive normalization pose guidance according to claim 1, characterized in that, In step S1, the pose heatmap of the person image in 18 channels is estimated using the OpenPose method, which includes 18 key points: nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, left knee, left ankle, right hip, right knee, right ankle, left eye, right eye, left ear, and right ear. Each key point is represented by one channel, and the key points are interconnected to form the skeletal structure of the human body.
3. The image generation method based on style content adaptive normalization pose guidance according to claim 1, characterized in that, In step S2, the extended Gaussian difference edge detection method is used to extract the edge mapping information of the person image, and the black and white grayscale source edge map of the human body in the person image is obtained.
4. The image generation method based on style content adaptive normalization pose guidance according to claim 1, characterized in that, The aligned multi-scale content transfer network described in step S3 consists of an aligned multi-scale transfer decoder and three encoders; each encoder consists of a downsampling layer, an instance normalization layer, an activation layer and a residual block. The aligned multi-scale transfer decoder consists of a deconvolutional layer, an instance normalization layer, an activation function, and residual blocks. The deconvolutional layer uses a 4x4 convolutional kernel with a stride of 2 and a margin of 1. The source image edge map, source image pose heatmap, and target image pose heatmap are input into the encoder to obtain a feature map with 256 channels and a size of 32x32. After attention calculation, the target image edge map is decoded.
5. The image generation method based on style content adaptive normalization pose guidance according to claim 4, characterized in that, The encoder uses a 4x4 convolutional kernel for its downsampling layer with a stride of 2 and a margin of 1. The residual block consists of two convolutional layers, two instance normalization layers, and one activation layer. The convolutional layers use a 3x3 convolutional kernel with a stride of 1 and a margin of 1. Each convolutional layer is followed by an instance normalization layer, and a ReLU activation function is added after the first instance normalization layer.
6. The image generation method based on style content adaptive normalization pose guidance according to claim 1, characterized in that, The optical flow estimation model described in step S4 consists of an encoder and a decoder. The encoder consists of a downsampling layer and a convolutional layer. Each layer is preceded by an instance normalization layer and an activation function layer. The downsampling layer uses a 4x4 convolutional kernel with a stride of 2, and the convolutional layer uses a 3x3 convolutional kernel with a stride of 1. The source image, the source image pose heatmap, and the target image pose heatmap are fused in the channel dimension and then passed through the encoder to obtain a feature map with 256 channels and a size of 32x32. The decoder outputs a two-dimensional flow field optical flow map with 2 channels and an occlusion mask information with 1 channel.
7. The image generation method based on style content adaptive normalization pose guidance according to claim 6, characterized in that, In step S5, local feature patch pairs are extracted from the source image and the target image based on the flow field. The context-aware sampling kernel is calculated using a kernel prediction network. Finally, the source features are sampled to obtain the distortion result of the sampling position.
8. The image generation method based on style content adaptive normalization pose guidance according to claim 7, characterized in that, The kernel prediction network uses 3x3 convolutional kernels to extract local feature patch pairs. It consists of convolutional layers, activation layers, and softmax.
9. The image generation method based on style content adaptive normalization pose guidance according to claim 1, characterized in that, The style adaptive normalization generator described in step S6 consists of a pose encoder, a style encoder, a residual block, a style adaptive normalization module, and a residual decoder. The pose encoder consists of a 4x4 convolutional kernel, a downsampling layer with a stride of 2, and a 3x3 convolutional kernel with a stride of 1. Each layer is preceded by an instance normalization layer and an activation function layer. The style encoder consists of a 4x4 convolutional kernel, a downsampling layer with a stride of 2, and a 3x3 convolutional kernel with a stride of 1. Each layer, combined with self-attention, is preceded by an instance normalization layer and an activation function layer. The residual block consists of two convolutional layers, each consisting of an activation layer, an instance normalization layer, and a convolutional layer with a 3x3 kernel. The residual decoder consists of a convolutional layer with a 3x3 kernel and a transposed convolutional layer with a 4x4 kernel. The style adaptive normalization module consists of three region adaptive normalization layers, each modulating the input feature parameters with two spatially adaptive normalization parameters.