Attitude-driven figure image synthesis method based on attention mechanism

Abstract

The invention discloses a posture-driven figure image synthesis method based on an attention mechanism. Firstly, a target skeleton is converted into feature mapping through a skeleton encoder, and an original image is converted into feature mapping through an appearance encoder; performing multi-scale image refinement on the extracted features by using a multi-scale image refinement network to obtain a final target figure image; and finally, training the model in an end-to-end mode, and learning feature fusion and target image generation at the same time. According to the method, an attitude-driven attention mechanism is provided, appearance features and attitude features can be effectively extracted, and feature maps with different attention degrees are obtained through adaptive weights of the channel attention mechanism. And finally adding and outputting according to the weight, thereby being beneficial to keeping the information of the original image to the greatest extent without losing the original features in the attitude transformation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a method for character synthesis, and in particular to a method for character image generation using an attention mechanism. Background Art

[0002] Pose-based character image generation technology has attracted widespread attention in academia and industry in recent years. It has a wide range of applications, such as e-commerce, movie special effects, and character re-identification. The technology aims to convert the posture of a given character into a target posture to generate a corresponding image. In the field of virtual reality, character synthesis technology plays an important role. It can generate realistic character models, thereby enhancing the user's interactive experience with the virtual environment. In the task of virtual character reconstruction, this technology helps to restore or reconstruct damaged virtual character images. In simulated clothing changing and clothing texture generation applications, character synthesis technology can create more realistic clothing effects and texture effects, providing users with a more realistic clothing changing and clothing design experience.

[0003] Although this technology has made some progress in maintaining high-fidelity texture and body structure, the spatial misalignment and occlusion of local body parts caused by posture changes remain a challenge. In addition, existing datasets usually contain limited texture styles, which limits the diversity of generated character appearance. The pose-guided character image generation task aims to generate character images of any given pose. Some early models were built on ordinary CNN neural networks. CNN lacks the ability to perform complex geometric transformations and is not suitable for character pose transformation tasks with high spatial complexity. With the continuous development of models such as generative adversarial networks and variational autoencoders, pose-guided character synthesis technology has made great progress. The method is committed to introducing additional priors, such as character parsing semantic images and character pose key points, or using attention mechanisms to improve the network's spatial transformation capabilities. The challenges faced by character synthesis technology mainly focus on two key areas: one is how to generate character images with diverse and high-quality features, and the other is how to achieve full control over the generated character pose and appearance. To solve the first problem, researchers can explore the use of more advanced neural network architectures and combine them with large-scale training datasets to improve the generalization ability and generation quality of the model. In addition, the generation diversity of the model can be enhanced by introducing random noise and diverse stylized sampling techniques during the training process. For the second challenge, how to accurately control the pose and appearance of the generated characters, the research can be achieved by introducing additional prior knowledge and constraints. This includes using pose-guided prior knowledge to guide the generation process of character images, and using the semantic information corresponding to the character images as a priori conditions for the generation task, thereby enhancing the controllability of the generation process. These methods are all based on a deep understanding of character structure, clothing design principles, and pose control mechanisms, and aim to improve the overall performance and practicality of character synthesis technology.

[0004] Human synthesis technology has broad application prospects in many fields, but the challenges it faces cannot be ignored. Future research needs to further explore how to solve the spatial dislocation and occlusion problems caused by posture changes, and how to expand the diversity of texture styles in the dataset to improve the quality of generated human images.

[0005] The problems existing in the existing pose-guided human image generation methods include:

[0006] The first is the uncertainty of the translation of human body structure from source to target. Existing methods that introduce priors cannot accurately guide the generation of human images and lack scalability for large-scale applications. But without additional priors, the introduced human poses do not correspond spatially to the target images, resulting in texture misalignment of some body parts.

[0007] The second is the problem of insufficient quality in character appearance synthesis. Traditional synthesis methods focus too much on the generalization ability of the model in pose transfer, and often lack attention to the texture details in the source image, resulting in excellent performance of synthetic image learning in pose transfer but lack of fine-grained texture in character appearance. Summary of the invention

[0008] In order to solve the problems of texture misalignment and insufficient appearance quality in character image synthesis, the present invention proposes a posture-driven character image synthesis method based on an attention mechanism. In the process of realizing complex posture changes, the original features of the character's appearance can be effectively retained.

[0009] The present invention is dedicated to solving the problem of misalignment between the pose of the synthesized character image and the texture of the original image and insufficient details of the synthesized texture in the field of character image synthesis. We introduce a generalized linear attention mechanism synthesis method. Taking the character image and the key points of the character skeleton as input, the image features and the spatial distribution of the character pose are extracted using feature extraction filters and normalized matrices. The appearance features of each part of the character are distributed to the character pose through multi-channel matrix multiplication. By using the attention mechanism, a more specific and realistic character appearance texture can be obtained, while avoiding the problem of misalignment between appearance and pose due to excessive changes in pose.

[0010] The present invention can effectively reduce training costs and accelerate the model training process in actual training. At the same time, it can effectively control the appearance and posture of the synthesized character. Compared with the traditional method that introduces a single prior or no prior content, the present invention introduces the skeletal key points corresponding to the character image as a prior during training, and uses the appearance encoder and the skeleton encoder to learn the character appearance and character posture, and the introduced features are fused and output through matrix multiplication. In addition, the present invention proposes a progressive sampling method that can generate images at various scales and obtain the final image through a step-by-step generation method. In the inference stage, our model is able to generate character features in different postures.

[0011] A posture-driven character image synthesis method based on an attention mechanism comprises the following steps:

[0012] Step 1: Convert the target skeleton into feature maps through a skeleton encoder.

[0013] Step 2: Convert the original image into a feature map through the appearance encoder.

[0014] Step 3: After obtaining the features extracted by the skeleton encoder and the appearance encoder, a multi-scale image refinement network is used to perform multi-scale image refinement on the extracted features.

[0015] The multi-scale image refinement network takes the feature maps generated by the skeleton encoder and the appearance encoder as input. For feature maps of different scales, pose-driven attention is used to generate target features. Finally, character images at different resolutions are generated step by step by upsampling and "ToRGB" the feature output of each scale to obtain the final target character image.

[0016] Step 4: Train the model in an end-to-end manner to learn feature fusion and target image generation simultaneously. The model is trained using attention reconstruction loss, perceptual loss, style loss, and adversarial loss:

[0017] The overall losses are as follows:

[0018] L all =λ atten L atten +λ style L style +λ perc L perc +λ adv L adv

[0019] where λ atten ,λ style ,λ perc ,λ adv Represent the attention reconstruction loss L atten , Perceptual loss L perc , style loss L style and adversarial loss L adv The weight of .

[0020] Furthermore, the specific method of step 1 is as follows:

[0021] First, the original character image is processed using the character pose estimation method to obtain the target skeleton key points, and the target skeleton key points are constructed into 18-channel feature heat maps. These feature heat maps represent the connection relationship between human joints. The 18-channel feature heat maps are input into the skeleton encoder, and the biased downsampling convolutional neural network is used to encode the feature heat maps into a high-dimensional space to guide image generation. The final output resolution of the skeleton encoder is 16×16.

[0022] Furthermore, the specific method of step 2 is as follows:

[0023] The appearance encoder also uses a biased downsampling convolutional neural network. The original person image is encoded into a high-dimensional space to guide the generation of the final image. The appearance encoder divides the output resolution into 5 levels, and outputs in 5 resolution levels in order: 16×16, 32×32, 64×64, 128×128, and 256×256.

[0024] Furthermore, the specific method of step 3 is as follows:

[0025] The multi-scale image refinement network takes the feature maps generated by the skeleton encoder and the appearance encoder as input. For feature maps of different scales, pose-driven attention is used to generate target features. After obtaining the target features at the current scale, it is injected into the pose-driven attention of the next layer as input together with the features output by the appearance encoder. In general, for each layer, the network uses the target features output by the previous layer as reference features to predict the spatial distribution of the target person image. For the feature generation of the first level, the output of the appearance encoder and the skeleton encoder is directly used for target feature generation. For the target features of other levels, the output of the appearance encoder and the target features output by the pose-driven attention of the previous level are used for feature generation to obtain the target features at the current level. Finally, the feature output of each scale is upsampled and "ToRGB" to generate character images at different resolutions step by step to obtain the final target person image.

[0026] Furthermore, posture-driven attention is implemented as follows:

[0027] For the pose-driven attention at the 16×16 level, the features generated by the appearance encoder and the skeleton encoder are used as input, while for other levels, the features obtained by the pose-driven attention at the previous level and the features output by the appearance encoder are used as input. It uses convolution filtering and normalization functions to extract appearance and structural features. Then the appearance information is broadcast to the character pose through multi-channel matrix multiplication.

[0028] First, extract the appearance features:

[0029] Let F r represents the features extracted from the appearance of the person by the appearance encoder. r Input pixel analysis filter, calculate pixel analysis filter W b and F r The attention correlation matrix C between b :

[0030] C b =W b ×F r

[0031] Among them, C b Each row of contains the contribution of each appearance encoder’s encoded features to the output appearance features of this level. The convolution filter is used as the pixel analysis filter W b , a two-dimensional convolutional filter with spatial size is used to obtain accurate texture features.

[0032] The obtained attention correlation matrix is ​​normalized by the softmax function to obtain the normalized matrix In addition, the obtained eigenvalues ​​are mapped to the latent space through the mapping function f, and multiplied with the normalized matrix to obtain the processed appearance feature F e , the formula is as follows:

[0033]

[0034] Then the structural features are extracted using F t Represents the structural information extracted from the character skeleton by the skeleton encoder (first level) or the features output by the posture-driven attention (except the first level). t Input the attitude estimation filter and calculate the attitude estimation filter W d and F t The attention correlation matrix C between d :

[0035] C d =W d ×F t

[0036] A two-dimensional convolution filter with spatial size W is also used as the pose estimation filter d Perform feature extraction. Normalize through the softmax function to obtain a normalized matrix in Each column of contains the contribution of each feature extracted by the skeleton encoder or the features output by the pose-driven attention to the synthesis of the output structural features of this level.

[0037] Finally, the obtained appearance features are assigned to the target person's posture through a multi-channel matrix multiplication (MCM) including channel attention. The formula is as follows:

[0038]

[0039] Where MCM stands for multi-channel matrix multiplication, Represents the feature map of the Nth output with different attention regions.

[0040] MCM utilizes channel attention to generate feature maps with different focus areas, and through channel attention, they will be assigned different weight values.

[0041]

[0042] Among them, Linear means that N feature maps with different attention levels are generated and added according to different weights to obtain a feature map and then output, Fo Represents the final target feature.

[0043] Furthermore, attention reconstruction loss: Attention reconstruction loss is used to constrain feature fusion operations to obtain accurate spatial distribution results. This loss promotes the distribution results of each layer to be closer to the real image.

[0044]

[0045] in It is composed of real person images I t And the target feature map F generated from the original person image o It is obtained by adjusting to the lth resolution level, where l∈[1,2,3,4,5].

[0046] Furthermore, perceptual loss: the target image can be restricted to a higher feature level through perceptual loss. The pre-trained VGG19 model is used to calculate the difference between the generated target person image and the real image.

[0047]

[0048] in Represents the generated target person image, I t Represents a real person image, Represents the i-th activation layer in the VGG19 network, which is calculated using the 5 layers [relu1_1,relu2_1,relu3_1,relu4_1,relu5_1].

[0049] Furthermore, style loss: In order to further enhance the spatial consistency of the target person image and the source image in terms of appearance attributes, style loss is introduced. The specific formula for calculating the error between the generated target person image and the real image using the Gram matrix is ​​as follows;

[0050]

[0051] in Represents the generated target person image, I t Represents a real person image, represents the ath activation layer in the VGG19 network, using the 4 layers [relu2_2,relu3_4,relu4_4,relu5_2] for calculation. G is the Gram matrix.

[0052] Furthermore, adversarial loss: adversarial loss is used to constrain the generated image so that it approximates the distribution of the real image, and is also suitable for capturing the high-frequency details of the image. The specific formula is as follows;

[0053] Ladv =E[log(1-D(G(P i ,I i )))]+E[log(D(I i ))]

[0054] Where D stands for discriminator and G stands for generator.

[0055] The beneficial effects of the present invention are as follows:

[0056] The present invention proposes a posture-driven attention mechanism, which can effectively extract appearance features and posture features, and obtain feature maps of different attention levels through the adaptive weights of the channel attention mechanism. Finally, the weighted sum is output, which is conducive to not losing the original features in the posture change and retaining the information of the original image to the greatest extent. At the same time, the present invention proposes a progressive generation network that can generate character images step by step. Not only is it conducive to the final generation of the target image to obtain a finer texture, but the progressive generation process allows the model to perform posture migration at different resolutions, which is conducive to the model capturing the image texture under complex posture changes, and can achieve the correspondence between the posture and the original appearance of the character when synthesizing the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the overall training framework of an embodiment of the present invention;

[0058] Figure 2 This is a specific structural diagram of the posture-driven attention embodiment of the present invention;

[0059] Figure 3 This is an example diagram of the experimental results of the present invention;

[0060] Figure 4 This is the specific structure diagram of the appearance encoder and skeleton encoder. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0062] like Figure 1 Figure 2 As shown, a posture-driven character image synthesis method based on an attention mechanism includes the following steps:

[0063] Step 1: Convert the target skeleton into feature maps through a skeleton encoder.

[0064] In the posture path, the original character image is first processed using the character posture estimation method to obtain the target skeleton key points, and the target skeleton key points are constructed as 18-channel feature heat maps. These feature heat maps represent the connection relationship between human joints, such as the connection relationship between arms and torsos. The 18-channel feature heat maps are input into the skeleton encoder, and the feature heat maps are encoded into a high-dimensional space using a biased downsampling convolutional neural network to guide image generation. In order to adapt to our task, the final output resolution of the skeleton encoder of the present invention is 16×16.

[0065] Step 2: Convert the original image into a feature map through the appearance encoder:

[0066] In the appearance path, the appearance encoder adopts a structure similar to the skeleton encoder, using a biased downsampling convolutional neural network. The original character image is encoded into a high-dimensional space to guide the generation of the final image. In order to adapt to our task, the appearance encoder of the present invention divides the output resolution into 5 levels, and outputs in 5 resolution levels of 16×16, 32×32, 64×64, 128×128, and 256×256 in sequence.

[0067] Step 3: After obtaining the features extracted by the skeleton encoder and the appearance encoder, a multi-scale image refinement network is used to perform multi-scale image refinement on the extracted features.

[0068] The multi-scale image refinement network takes the feature maps generated by the skeleton encoder and the appearance encoder as input. For feature maps of different scales, posture-driven attention is used to generate target features. After obtaining the target features at the current scale, it is injected into the posture-driven attention of the next layer as input together with the features output by the appearance encoder. In general, for each layer, the network uses the target features output by the previous layer as reference features to predict the spatial distribution of the target person image. For the feature generation of the first level, the output of the appearance encoder and the skeleton encoder is directly used for target feature generation. For target features of other levels, the present invention uses the output of the appearance encoder and the target features output by the posture-driven attention of the previous level for feature generation to obtain the target features at the current level, which helps to predict a more accurate spatial distribution. This process can generate more realistic images in subsequent layers. Finally, we generate character images at different resolutions step by step by upsampling and "ToRGB" the feature output of each scale to obtain the final target person image.

[0069] The specific implementation of posture-driven attention is as follows:

[0070] For the pose-driven attention at the 16×16 level, the present invention uses the features generated by the appearance encoder and the skeleton encoder as input, while for other levels, the present invention uses the features obtained by the pose-driven attention at the previous level and the features output by the appearance encoder as input. It uses convolution filtering and normalization functions to extract appearance and structural features. The appearance information is then broadcast to the character pose through multi-channel matrix multiplication. This allows the model to pay attention to information at different levels.

[0071] First, extract the appearance features:

[0072] Let F r represents the features extracted from the appearance of the person by the appearance encoder. r Input pixel analysis filter, calculate pixel analysis filter W b and F r The attention correlation matrix C between b :

[0073] C b =W b ×F r

[0074] Among them, C b Each row of contains the contribution of each appearance encoder’s encoded features to the output appearance features of this level. The convolution filter is used as the pixel analysis filter W b , instead of using 1×1 convolution filters, we use 2D convolution filters with spatial size to obtain accurate texture features. This configuration helps the model automatically learn the correlation of matching features. In addition, the features extracted by a specific filter consistently represent the same composition elements, which helps to decompose the appearance of different poses.

[0075] The obtained attention correlation matrix is ​​normalized by the softmax function to obtain the normalized matrix In addition, the obtained eigenvalues ​​are mapped to the latent space through the mapping function f, and multiplied with the normalized matrix to obtain the processed appearance feature F e , the formula is as follows:

[0076]

[0077] Then the structural features are extracted using F t Represents the structural information extracted from the character skeleton by the skeleton encoder (first level) or the features output by the posture-driven attention (except the first level). In essence, the structural feature extraction operation builds a spatial feature model of the appearance features. t Input the attitude estimation filter and calculate the attitude estimation filter W d and F tThe attention correlation matrix C between d :

[0078] C d =W d ×F t

[0079] A two-dimensional convolution filter with spatial size W is also used as the pose estimation filter d Perform feature extraction. Normalize through the softmax function to obtain a normalized matrix in Each column of contains the contribution of each feature extracted by the skeleton encoder or the features output by the pose-driven attention to the synthesis of the output structural features of this level.

[0080] Finally, the obtained appearance features are assigned to the target person's posture through a multi-channel matrix multiplication (MCM) including channel attention. The formula is as follows:

[0081]

[0082] Where MCM stands for multi-channel matrix multiplication, Represents the feature map of the Nth output with different attention regions.

[0083] MCM utilizes channel attention to generate feature maps with different focus areas, and through channel attention, they will be assigned different weight values.

[0084]

[0085] Among them, Linear means that N feature maps with different attention levels are generated and added according to different weights to obtain a feature map and then output, F o Represents the final target feature.

[0086] Step 4: The present invention defines some loss functions for model training. Our model aims to generate realistic individual images with fine textures and accurate character structures. To achieve this, we train the model in an end-to-end manner, learning feature fusion and target image generation at the same time. We use attention reconstruction loss, perceptual loss, style loss and adversarial loss to train the model:

[0087] Attention reconstruction loss: Use attention reconstruction loss to constrain feature fusion operations to obtain accurate spatial distribution results. This loss encourages the distribution results of each layer to be closer to the real image.

[0088]

[0089] in It is composed of real person images I t And the target feature map F generated from the original person image o It is obtained by adjusting to the lth resolution level, where l∈[1,2,3,4,5].

[0090] Perceptual loss: The perceptual loss can constrain the target image to a higher level of features. We use the pre-trained VGG19 model to calculate the difference between the generated target person image and the real image.

[0091]

[0092] in Represents the generated target person image, I t Represents a real person image, Represents the i-th activation layer in the VGG19 network. In actual operation, we use the five layers [relu1_1, relu2_1, relu3_1, relu4_1, relu5_1] for calculation.

[0093] Style loss: In order to further enhance the spatial consistency of the target person image and the source image in terms of appearance attributes such as texture features, we introduced style loss. The specific formula for calculating the error between the generated target person image and the real image using the Gram matrix is ​​as follows;

[0094]

[0095] in Represents the generated target person image, I t Represents a real person image, Represents the ath activation layer in the VGG19 network. In actual operation, we use the 4 layers [relu2_2, relu3_4, relu4_4, relu5_2] for calculation. G is the Gram matrix.

[0096] Adversarial loss: We use adversarial loss to constrain the generated image to approximate the distribution of the real image, and also to capture the high-frequency details of the image. The specific formula is as follows;

[0097] L adv =E[log(1-D(G(P i ,I i )))]+E[log(D(I i ))]

[0098] Where D stands for discriminator and G stands for generator.

[0099] The overall losses are as follows:

[0100] L all =λ atten L atten +λ style L style +λ perc L perc +λ adv L adv

[0101] where λ atten ,λ style ,λ perc ,λ adv Represents the weight of the corresponding loss function.

[0102] Example

[0103] First, the person image I is encoded by the appearance encoder r Processing is performed to map the input into high-dimensional features; at the same time, a skeleton encoder is used to encode the corresponding target skeleton key points P t Processing is performed to learn the distribution of character postures while performing feature mapping, and introducing features into high-dimensional space. Then, the features mapped by the appearance encoder and the skeleton encoder are operated with a posture-guided attention mechanism at different resolutions. The task image synthesis of each scale is achieved, and the final synthetic image is generated step by step by performing "ToRGB" and "Upsample" on the target features of different scales. The "ToRGB" operation refers to the 1x1 convolution of converting high-dimensional features into RGB images, while "Upsmp" refers to the bilinear upsampling operation. The generated target character image will be input into the discriminator to determine the authenticity.

[0104] The input of the skeleton encoder is a tensor composed of an 18-channel heat map. This part of the data is processed using pose estimation technology. The specific technology used is the character pose estimation method mentioned in the Openpose paper. During training, the 18-channel heat map is processed by the skeleton encoder and input into the first level of pose-driven attention. The heat map represents the relationship between the character's poses, similar to the connection between the various bone joints in the body, and can guide image generation in subsequent synthesis. The skeleton encoder is composed of a convolutional neural network with downsampling function, such as Figure 4 As shown in the figure, we marked the corresponding resolution at the top. The skeleton encoder contains 5 3×3 convolutions, 4 of which are downsampled by a factor of 2. The convolution layers are connected by a Blur layer, which blurs the output between convolutions to make the final output of the skeleton encoder smoother. The skeleton encoder can map the input to a high-dimensional space and mine the correlation between character images. The final output resolution is 16×16.

[0105] The input of the appearance encoder is a real person image. The structure of the appearance encoder is similar to that of the skeleton encoder. Figure 4 , we marked the corresponding resolution at the top of the figure. It consists of a convolutional neural network with downsampling, and the 5 convolutional layers are also connected by Blur layers. We extract the output features of each convolutional layer as the output of the appearance encoder. The resolutions are 16×16, 32×32, 64×64, 128×128, and 256×256. It will learn the appearance distribution of human images and map the learned features to a high-dimensional space.

[0106] The pose-driven attention operation is a generalized attention mechanism paradigm. It first extracts the input appearance information F through an appearance feature extraction operation. r Through a pixel analysis filter (PixelAnalysis Filters) W b , and then calculate the pixel analysis filter and F r The attention correlation matrix C between b , use the softmax function to normalize and get the normalized matrix At the same time, the obtained eigenvalues ​​are mapped to the latent space using the mapping function f, and multiplied with the normalized matrix to obtain the extracted appearance features. Similarly, we use a pose estimation filter (Pose Estimate Filters) W d Extract features and extract structural information F t Attention correlation matrix C d Calculation of F t Represents the output of the skeleton encoder, or the features of the posture-driven attention output of the previous level. Then the normalized matrix is ​​calculated by the softmax function Through multi-channel matrix multiplication (Multi-Channel Matmual) and obtaining feature maps of different attention levels, the generated N feature maps of different attention levels are finally added according to different weights and output as the final target feature.

[0107] After obtaining the target features of the gesture-driven attention output, the present invention converts the target features into a character image of the current level resolution through "Upsmp" (upsampling) and "ToRGB" (to color). We convert the features into RGB images by performing "ToRGB" on the target features, and then obtain a new RGB image by upsampling and adding it to the "ToRGB" result of the next layer, completing the process from low resolution to high resolution. The new RGB image will continue to be upsampled and added to the RGB image generated by the next layer until the final target character image with a resolution of 256×256 is generated.

[0108] All the operations above are the components of the generator, and there is a dynamic game process between it and the discriminator. The generator strives to generate fake data that can deceive the discriminator, while the discriminator strives to improve its ability to distinguish between real data and fake data. This game process allows the generator and the discriminator to gradually reach a balance during model training, thereby obtaining high-quality generation results.

[0109] Through the above training process, the present invention can achieve high-fidelity and high-resolution image generation, and can effectively control the diversity of generated images.

[0110] The proposed method is trained on the deepfashion dataset. This dataset contains 52,712 high-resolution images, including a large number of model person images with different poses and appearances, and is widely used in person image generation methods. We selected 101,966 pairs of images for training and 8,750 pairs of images for testing. There is no overlap between the training set and the test set.

[0111] In the experiment, we use 256 × 176 to train the proposed model. All experiments are based on the PyTorch framework, and the proposed method is trained for 200 epochs using the Adam optimizer (β1 = 0, β2 = 0.99) with a batch size of 8. The learning rate of the generator and the discriminator during training is set to 2 × 10 -3 In addition, during training we set λ atten =15,λ style =2,λ perc =2,λ adv =1.5.

[0112] The experimental results of the present invention are exemplified as follows Figure 3 As shown in the figure, the first three columns are the original image, the target skeleton and the generated image, and the last column is the result after posture transfer. From the visualization effect, it can be seen that the framework proposed by the present invention generates the appearance posture of the human body very well, and retains the detailed texture of the original image. Moreover, the texture of the synthesized human image is not misaligned with the posture.

[0113] The comparison of the present invention with other technologies is shown in Table 1. The indicators are evaluated using structural similarity (SSIM) and FID scores. SSIM (structural similarity index) is an indicator for measuring the similarity between two images, which takes into account the structure and texture information of the images. Unlike simple pixel comparison, SSIM not only considers the three aspects of brightness, contrast and structure, but also performs a weighted average of these three aspects to more comprehensively evaluate the similarity of images. FID is an indicator for evaluating the similarity of the distribution of two groups of images. It is based on the feature extractor of the deep learning model InceptionV3 to calculate the average Frechet distance of two groups of images. It can be seen that the present invention has reached the optimal level in both indicators.

[0114] Table 1: Quantitative comparison of other technologies

[0115]

[0116]

[0117] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0118] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A posture-driven character image synthesis method based on attention mechanism, characterized in that: The steps include: Step 1: Convert the target skeleton into feature maps through the skeleton encoder; Step 2: Convert the original image into a feature map through the appearance encoder; Step 3: After obtaining the features extracted by the skeleton encoder and the appearance encoder, a multi-scale image refinement network is used to perform multi-scale image refinement on the extracted features; The multi-scale image refinement network takes the feature maps generated by the skeleton encoder and the appearance encoder as input; pose-driven attention is used to generate target features for feature maps of different scales; finally, character images of different resolutions are generated step by step by upsampling and "ToRGB" the feature output of each scale to obtain the final target character image; Step 4: Train the model in an end-to-end manner to learn feature fusion and target image generation simultaneously; attention reconstruction loss, perceptual loss, style loss, and adversarial loss are used to train the model: The overall losses are as follows: L all =λ atten L atten +λ style L style +λ perc L perc +λ adv L adv where λ atten ,λ style ,λ perc ,λ adv Represent the attention reconstruction loss L atten , Perceptual loss L perc , style loss L style and adversarial loss L adv The weight of .

2. According to claim 1, a posture-driven character image synthesis method based on attention mechanism is characterized in that: Step 1: Firstly, the original character image is processed by the character pose estimation method to obtain the target skeleton key points, and the target skeleton key points are constructed into 18-channel feature heat maps; these feature heat maps represent the connection relationship between human joints; the 18-channel feature heat maps are input into the skeleton encoder, and the biased downsampling convolutional neural network is used to encode the feature heat maps into a high-dimensional space to guide image generation; the final output resolution of the skeleton encoder is 16×16.

3. The method of character image synthesis based on a posture-driven attention mechanism according to claim 1, characterized in that: Step 2: The appearance encoder also uses a biased downsampling convolutional neural network. It guides the generation of the final image by encoding the original character image into a high-dimensional space. The appearance encoder divides the output resolution into 5 levels and outputs them in 5 resolution levels of 16×16, 32×32, 64×64, 128×128, and 256×256 in sequence.

4. A posture-driven attention-based character image synthesis method according to any one of claims 1 to 3, characterized in that: Step 3: The multi-scale image refinement network takes the feature maps generated by the skeleton encoder and the appearance encoder as input; pose-driven attention is used to generate target features for feature maps of different scales; After obtaining the target features at the current scale, they are injected into the posture-driven attention of the next layer together with the features output by the appearance coder as input; In general, for each layer, the network uses the target features output by the previous layer as reference features to predict the spatial distribution of the target person image. For the feature generation of the first level, the output of the appearance encoder and the skeleton encoder is directly used to generate the target features; for the target features of other levels, the output of the appearance encoder and the target features output by the posture-driven attention of the previous level are used for feature generation to obtain the target features of the current level; finally, through upsampling and the feature output of each scale of "ToRGB", character images of different resolutions are generated step by step to obtain the final target person image.

5. The method of character image synthesis based on a posture-driven attention mechanism according to claim 4, characterized in that: The specific implementation of posture-driven attention is as follows: For the pose-driven attention at the 16×16 level, the features generated by the appearance encoder and the skeleton encoder are used as input, while for other levels, the features obtained by the pose-driven attention at the previous level and the features output by the appearance encoder are used as input; it uses convolution filtering and normalization functions to extract appearance and structural features; then the appearance information is broadcast to the character pose through multi-channel matrix multiplication; First, extract the appearance features: Let F r represents the features extracted from the appearance of the person by the appearance encoder. r Input pixel analysis filter, calculate pixel analysis filter W b and F r The attention correlation matrix C between b : C b =W b ×F r Among them, C b Each row of contains the contribution of the features encoded by each appearance encoder to the output appearance features of this level; the convolution filter is used as the pixel analysis filter W b ,Using 2D convolution filters with spatial size to obtain accurate texture features; The obtained attention correlation matrix is ​​normalized by the softmax function to obtain the normalized matrix In addition, the obtained eigenvalues ​​are mapped to the latent space through the mapping function f, and multiplied with the normalized matrix to obtain the processed appearance feature F e , the formula is as follows: Then the structural features are extracted using F t Represents the structural information extracted by the skeleton encoder from the character skeleton or the features of the posture-driven attention output; F t Input the attitude estimation filter and calculate the attitude estimation filter W d and F t The attention correlation matrix C between d : C d =W d ×F t A two-dimensional convolution filter with spatial size W is also used as the pose estimation filter d Perform feature extraction; normalize through the softmax function to obtain a normalized matrix in Each column of contains the contribution of each skeleton encoder extracted feature or pose driven attention output feature to the synthesis of the output structure feature of this level; Finally, the obtained appearance features are assigned to the target person's posture through a multi-channel matrix multiplication including channel attention. The formula is as follows: Where MCM stands for multi-channel matrix multiplication, Represents the feature map of the Nth output with different focus regions; MCM uses channel attention to generate feature maps with different focus areas, and through channel attention, they will be assigned different weight values; Among them, Linear means that N generated feature maps with different attention levels are added according to different weights to obtain a feature map and then output, F o Represents the final target feature.

6. The method of character image synthesis based on a posture-driven attention mechanism according to claim 1, characterized in that: The attention reconstruction loss is as follows: Use attention reconstruction loss to constrain feature fusion operations to obtain accurate spatial distribution results. This loss makes the distribution results of each layer closer to the real image. in It is composed of real person images I t And the target feature map F generated from the original person image o It is obtained by adjusting to the lth resolution level, where l∈[1,2,3,4,5].

7. The method of character image synthesis based on a posture-driven attention mechanism according to claim 1, characterized in that: The perceptual loss is as follows: The perceptual loss can limit the target image to a higher feature level; the pre-trained VGG19 model is used to calculate the difference between the generated target person image and the real image; in Represents the generated target person image, I t Represents a real person image, Represents the i-th activation layer in the VGG19 network, which is calculated using the 5 layers [relu1_1,relu2_1,relu3_1,relu4_1,relu5_1].

8. The method of character image synthesis based on a posture-driven attention mechanism according to claim 1, characterized in that: The style loss is as follows: In order to further enhance the spatial consistency of the target person image and the source image in terms of appearance attributes, style loss is introduced; the specific formula for calculating the error between the generated target person image and the real image using the Gram matrix is ​​as follows; in, Represents the generated target person image, I t Represents a real person image, represents the ath activation layer in the VGG19 network, using the four layers [relu2_2,relu3_4,relu4_4,relu5_2] for calculation; G is the Gram matrix matrix.

9. The method of character image synthesis based on a posture-driven attention mechanism according to claim 1, characterized in that: The adversarial loss is as follows: Use adversarial loss to constrain the generated image so that it approximates the distribution of the real image, and is also suitable for capturing the high-frequency details of the image; the specific formula is as follows; L adv =E[log(1-D(G(P i ,I i )))]+E[log(D(I i ))] Where D stands for discriminator and G stands for generator.

10. A posture-driven attention-based character image synthesis method according to claim 2 or 3, characterized in that: The biased downsampling convolutional neural network includes five 3×3 convolutions, four of which are downsampled by a factor of 2; the convolutional layers are connected by a Blur layer.

Images