Human pose transfer method based on image feature alignment

Abstract

Human pose transfer based on image feature alignment belongs to the field of image processing. To address the challenges of utilizing global receptive domains, preserving detailed features, preventing style feature loss, and improving network convergence speed, the key is that the feature alignment network module includes a first input, a second input, and an output, with a style mapping F... p The first input to the first feature alignment network module is the pose feature map F. s The second input of each feature alignment network module is input in parallel. The output of the previous feature alignment network module is input as the first input of the next feature alignment network module. The output of the nth feature alignment network module is the output effect of the feature alignment network, which can generate realistic human images with ideal poses.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to a method for human pose transfer based on image feature alignment. Background Technology

[0002] Pose-guided human image generation aims to generate realistic human images using human images and several desired poses, and has wide applications in human re-identification, image processing, and video generation. However, it presents a significant challenge due to large spatial distortions and text occlusion.

[0003] Recently, Generative Adversarial Networks (GANs) have been successfully applied to human pose transfer. State-of-the-art human pose transfer methods primarily utilize convolutional structures. They can be categorized into two types: direct deformation-based methods and flow / transform-based methods. Direct deformation methods typically employ an encoder-decoder convolutional structure and incorporate an attention module to perform the transformation task. On the other hand, flow / transform-based methods estimate an appearance flow or transformation matrix to guide image generation. They often transform the source image and its feature curves to the target pose to obtain the appearance flow or learn feature-level mappings guided by segmentation maps.

[0004] Despite performance improvements achieved through previous methods, state-of-the-art approaches suffer from two problems due to their convolutional architecture. First, convolutional neural networks extract features within a local sliding window. They cannot handle long-range dependencies unless very deep convolutional layers are used, leading to a loss of feature resolution and detail. Incomplete features also affect the decoder if the convolution-based encoder cannot preserve more detail, especially in spatial transformation tasks. Therefore, traditional convolution-based models are not well-suited for capturing critical global information. Some methods can obtain a certain degree of global context by introducing non-local modules into convolutional-based architectures. However, their Q, K, and V bonds are concentrated in different domains, resulting in inefficient spatial transformations and obscure results. Figure 1 As shown in columns three and four. In contrast, the Transformer-based method of this invention can achieve high-quality results, such as... Figure 1 As shown in the second column. Secondly, most methods encode style and pose images into latent vectors, which are then used for network synthesis. However, latent vectors describe the semantic information of the image while ignoring style features. Therefore, they cause some local style features to be washed away in the final image generation. Summary of the Invention

[0005] To address the challenges of utilizing global receptive domains, preserving detailed features, and preventing style features from being washed away, thereby improving network convergence speed, this invention proposes the following technical solution: a human pose transfer method based on image feature alignment, comprising...

[0006] Input semantic graph I p The first Transformer encoder is input as input, and its output is input to the first MBConv. The output of the first Transformer encoder is tracked, and the pose feature map is output from the first MBConv to obtain the style mapping F. p ;

[0007] The source image Is is input to the second Transformer encoder, and the output of the second Transformer encoder is input to the second MBConv. The output of the second Transformer encoder is tracked, and the style feature map is extracted from the output of the second MBConv to obtain the pose feature map F. s ;

[0008] The feature alignment network consists of n feature alignment network modules, where n ≥ 2. Each feature alignment network module includes a first input, a second input, and an output, and the style mapping F... p The first input to the first feature alignment network module is the pose feature map F. s The second input of each feature alignment network module is input in parallel. The output of the previous feature alignment network module is input as the first input of the next feature alignment network module. The output of the nth feature alignment network module is... It is the output of the feature alignment network.

[0009] Output of the feature alignment network and style feature diagram F s Through matrix operations, the correlation matrix M of the aligned samples is obtained. The correlation matrix M includes target style and semantic information. The correlation matrix M is multiplied with the source image Is to output the aligned feature map S1.

[0010] The Transformer synthesis network consists of n synthesis network modules, where n ≥ 2. Each synthesis network module includes a first input, a second input, and an output. The aligned feature map S1 is input to the first input of the first synthesis network module. The correlation matrix M is input to the second input of each synthesis network module. The output of the previous synthesis network module is input to the first input of the next synthesis network module. The nth synthesis network module outputs the feature map F. g .

[0011] According to the human pose transfer method based on image feature alignment in this application, the feature alignment network module includes a Transformer network and a convolutional network, i = 2, ..., n, where the output of the (i-1)th feature alignment network module is... As the first input to the i-th feature alignment network module, the pose feature map F s As the second input to the i-th feature alignment network module, the first input to the i-th feature alignment network module includes parallel first input branches. and the first input branch two First input branch 1 With the second input F s Through the connection layer, the outputs of the connection layer are input to the Transformer network and the convolutional network respectively. The dot product operation of the outputs of the Transformer network and the convolutional network yields the attention diagram F. attn Note Figure F attn Including the first output F in parallel attn And the second output F attn First output F attn Through a 1×1 convolution, and then with the first input branch two... The dot product operation yields the first result, and the second output is F. attn The output of the i-th feature alignment network module is obtained by performing a 1×1 convolution followed by adding it to the first result.

[0012] According to the human pose transfer method based on image feature alignment in this application, the synthesis network module includes a Transformer network, a 1×1 convolution, a reshape module, and an upsampling module. The output of the (i-1)th synthesis network module is a feature map S of C×(i-1)H×(i-1)W. i-1 Let i = 2, ..., n, be the first input of the i-th synthetic network module, and let the correlation matrix M be the second input of the i-th synthetic network module. The first input of the i-th synthetic network module includes a parallel first input branch—feature map S. i-1 and the first input branch feature map S i-1 The first input branch is a feature map S. i-1 After passing through a Transformer network and a 1×1 convolutional network, we obtain 1×H×W features. Figure 1 ,feature Figure 1 After the Reshape module, the first vector of HW is obtained. The correlation matrix M is then passed through the average pooling layer of the (i-1)th synthesis network module to obtain the output vector. The first vector is multiplied by the output vector to obtain the second vector of HW. The second vector of HW is then passed through the Reshape module to obtain the 1×H×W feature vector. Figure 2 ,feature Figure 2 With feature map S i-1 The dot product operation yields the features of C×(i-1)H×(i-1)W. Figure 3 ,feature Figure 3 After upsampling, the feature map S of C×iH×iW is obtained. i .

[0013] According to the human pose transfer method based on image feature alignment in this application, the attention graph F of the i-th feature alignment network module is... attn It can be expressed by the following formula:

[0014]

[0015] In the formula, Tran represents Transformer, C represents Concat, S represents Sigmoid, and × represents element-wise multiplication.

[0016] According to the human pose transfer method based on image feature alignment in this application, the output of the i-th feature alignment network module is... It can be expressed by the following formula:

[0017]

[0018] In the formula, c∈C i ,h∈H i ,w∈W i C, H, and W represent the channel, height, and width of the tensor;

[0019] and Indicates the modulation parameters learned;

[0020] yes Enter the i-th... th Output features after layer normalization and activation;

[0021] and yes The mean and standard deviation in channel c.

[0022] According to the human pose transfer method based on image feature alignment in this application, the correlation matrix M is represented by the following formula:

[0023]

[0024] In the formula, M(u,v) represents the correlation matrix M. and Indicates positions u and v and F s The channel concentration characteristics.

[0025] According to the human pose transfer method based on image feature alignment in this application, feature map S i-1 It can be expressed by the following formula:

[0026]

[0027] Representing feature map S i-1 α represents the weight, I s (v) represents the source image distorted by the correlation matrix M.

[0028] According to the human pose transfer method based on image feature alignment in this application, feature map S i It can be expressed by the following formula:

[0029]

[0030] In the formula, Up represents upsampling, E represents the relevant paradigm field, and × represents element-wise multiplication.

[0031] According to the human pose transfer method based on image feature alignment in this application, the total loss function L total :

[0032] L total =λ a L adv +λ L1 λ L1 +λ p L per +λ CX L CX

[0033] In the formula, λ a , λ L1 , λ p , λ CX L is a hyperparameter of the equilibrium function. adv L1 represents the conditional adversarial loss, and L1 represents the L1 loss function. per L represents the perceptual loss function. CX This represents the context loss function.

[0034] According to the human pose transfer method based on image feature alignment in this application,

[0035]

[0036] In the formula, E represents the mathematical expectation, and I... p Let I represent the input semantic graph, and Is represent the distribution of human poses. t Let P represent a real-life image with the appearance of the source image and the target pose, and D represent the distribution of the real-life image. AF represents the appearance discriminator. g P represents the output of a dummy image that has the appearance of the source image and the pose of the target image. g D represents the distribution of the mannequin figures. p Represents a shape discriminator;

[0037] L1=||F g -I t ||1

[0038]

[0039] In the formula, Indicates the activation layer

[0040] L CX =-log(CX(F) l (F g ),F l (I t )))

[0041] In the formula, F l (F g ) and F l (I t ) respectively represent F g and I t The feature maps in the pre-trained network, where CX represents the similarity measure between matching features.

[0042] Beneficial Effects: This invention proposes a portrait generation framework based on the Transformer Feature Alignment Network (FaTNET), which consists of a Transformer encoder, a feature alignment network, and a Transformer synthesis network. Compared to previous convolutional and non-local methods, this invention can utilize global receptive domains and preserve detailed features. Furthermore, this invention uses feature alignment to obtain prior images to prevent style features from being washed away, such as... Figure 2As shown. Finally, considering that convolutional layers can bring strong induced bias priors, enabling the network to converge faster, this invention introduces convolutional layers into a transformer-based network structure. Specifically, the method of this invention has three steps. First, this invention proposes an encoder that extracts features from the source image and the target pose image by introducing a Swin-Transformer and then using deep convolution. Second, a feature alignment network based on multi-cascaded blocks is proposed to align the source pose image with the target pose image. Specifically, in each alignment block, this invention uses a Transformer network to maintain the global relationship between the tagged features of the target pose and the source image, and guides the matching of source features with target features. This invention also uses a convolutional network to focus on local information and preserve detailed features. The shape and style of the image are separated using the alignment network, resulting in a preliminary alignment feature map. Third, a Transformer synthesis network is proposed, which pays attention to the globally relevant features of the samples in order to recover detailed information and predict invisible regions during decoding. Experimental results show that the performance of this method is superior to current methods. Attached Figure Description

[0043] Figure 1 A comparison diagram of the method of the present invention with two other methods.

[0044] Figure 2 A visualization of the feature alignment network of this invention.

[0045] Figure 3 Self-attention model diagram.

[0046] Figure 4 The structural diagram of the present invention.

[0047] Figure 5 The feature alignment network structure diagram of the present invention.

[0048] Figure 6 The Transformer synthesis network structure diagram of the present invention.

[0049] Figure 7 The results are shown on the Market-1501 dataset.

[0050] Figure 8 The result image on the DeepFashion dataset.

[0051] Figure 9 A comparison graph of our method with other methods on the DeepFashion dataset.

[0052] Figure 10 A comparison plot of our method with other methods on the Market-1501 dataset.

[0053] Figure 11 Visualization of the ablation experiment results. Detailed Implementation

[0054] Pose-guided human image generation transforms source human images into target human pose images. This task is highly challenging due to significant spatial deformation and occlusion. Existing methods heavily rely on CNN-based architectures, thus being limited by local receptive fields and failing to preserve stylistic and shape details. To address this issue, this invention proposes a Transformer-based human pose transfer framework that leverages global dependencies while preserving local features. The framework consists of a Transformer encoder, a feature alignment network, and a Transformer synthesis network, capable of generating realistic human images with ideal poses. The core idea is to obtain a new prior image aligned with the target image in a disentangled feature space through the feature alignment network, then transform the result of the previous stage using the correlation matrix between the aligned feature image and the source image, and finally synthesize the final refined image through the Transformer synthesis network. Compared to previous methods, this invention utilizes the global receptive field while preserving detailed features. Qualitative and quantitative experimental results demonstrate the superiority of this model in human pose transfer.

[0055] Generative Adversarial Networks

[0056] Generative Adversarial Networks (GANs) consist of a generator and a discriminator, trained adversarially to generate realistic images. Since their inception, GANs have rapidly adopted a fully convolutional backbone and have been successfully applied to image-to-image transformation, image enhancement, and image editing. Recent work has combined Transformer modules with image generation models, replacing some convolutional components. Some researchers have introduced self-inventive attention mechanisms into GAN frameworks, where self-inventive attention modules complement convolutions, facilitating the establishment of long-range, multi-level dependencies across image regions. Recent research uses a convolutional GAN ​​to learn a codebook of image components and leverages Transformers to effectively model their structure in high-resolution images. Another work completely removes convolutions from their generative framework, using only two Transformers to create a stronger GAN.

[0057] Human posture transfer

[0058] Early human pose transfer methods proposed a two-level network to generate images with the target pose. It concatenates the source image, source pose, and target pose as input, learning the target image from coarse to fine. Zhu et al. proposed a progressive attention model to transfer the source image. However, using local attention mechanisms, useful information is lost during multiple transfers. Some methods employ non-local attention mechanisms, using keypoints as guidance for pose representation. Tang et al. used two generations of branches to simulate human appearance and shape information respectively. To better utilize pose and image features, Li et al. designed pre-localized pose-guided pose feature updates and post-localized pose-guided image feature updates. These methods can establish long-range dependency models. However, due to the sparse correspondence between the source and target images, image transformation is difficult. Han et al. were the first to generate human grammar analysis graphs as semantic guidance. They estimated dense flow fields at the pixel level to distort the signal source, which did not produce good results under large occlusion conditions. Li et al. distorted the input at the feature level. They proposed estimating dense and intrinsic 3D appearance flow to better guide pose transfer between pixels. However, these methods require an additional 3D human model for the flow field between the source and target, which incurs significant computational costs. Yu et al. proposed a different global flow local attention framework to recombine the input. They require no supplementary information and obtain the flow field in an unsupervised manner. However, their work is divided into two phases rather than an end-to-end approach.

[0059] Transformer in Computer Vision

[0060] The Transformer, designed with multi-head self-attention and feedforward MLP layers, was initially widely applied in neural language processing. In recent years, more and more researchers have used Transformers to replace some or all of the spatial convolutional layers in various computer vision tasks, achieving good results. Some works use pure Transformer models to process images. ViT reshapes images into a series of flat 2D patches and uses a Transformer structure for image classification. However, it requires a large-scale training dataset. DeIT introduces several training strategies to improve the data efficiency of ViT. PVT applies the ViT model to intensive semantic segmentation and object detection prediction tasks. Unlike traditional ViT, which has low-resolution output and high computational and storage costs, PVT proposes a pyramid structure that achieves high output resolution and low computational cost for large feature maps. T2T-Vit utilizes T2T modules to assist each token in modeling locally important structural information, thereby improving network performance. Other works use Transformers to supplement convolutional networks. They use self-attention layers to complete the backbone by providing the ability to encode long-range dependencies or heterogeneous interactions. They combine CNNs and Transformers into an encoder-decoder architecture for object detection and segmentation tasks.

[0061] Main tasks

[0062] Method Overview

[0063] The goal of this invention is to generate realistic images from a target pose while preserving the original style. This requires the network to have the ability to establish long-range dependencies and maintain local details. To address this challenge, this invention trains a Transformer network to generate human images. This invention proposes a novel framework including a Transformer encoder, a feature alignment network, and a Transformer synthesis network. Utilizing these modules, this invention can model the long-range interaction relationships between pixels from the source and target images. Simultaneously, this invention incorporates convolutional networks into its modules to effectively utilize the inductive bias capability and the ability to preserve local feature details of convolution. For arbitrary pose transfer, this invention uses commonly used pose representations to guide the transfer. Specifically, this invention uses 18 human keypoints extracted by the Human Pose Estimator (HPE) and represents a pose using a heatmap. This heatmap includes 18 channels, each encoding the position of joints within the human body, allowing any heatmap to represent any pose. This encoding method also facilitates semantic graph-guided image generation. During the training phase, the model uses the source image I... s Target attitude I p and target image I tThe generator uses I as input. s and I p The final output F is obtained. g .

[0064] Transformer encoder

[0065] Since its introduction in 2017, the Transformer has been widely used in Natural Language Processing (NLP). Recently, it has been applied in Computer Vision (CV) with promising results. Although the Transformer is a sequence model, it can process data in parallel, much like convolutions. Furthermore, it utilizes extensive self-attention, giving it a global receptive field that convolutions lack. The Transformer primarily consists of two parts: multi-head self-attention and a feedforward neural network. The feedforward neural network comprises two fully connected layers and one normalization layer. These layers are designed to enhance the network's expressive power.

[0066] The core of multi-head attention models is self-attention. It uses matrix transformations and multiplications to allow each input sequence to relate to other sequences, thus gaining a global receptive field, as shown in the figure. Assuming the input sequence is a sequence, it is first multiplied by a matrix to obtain an embedding. Then, the embedding enters the self-attention layer, where each vector is multiplied by three different Transformer matrices to obtain three different vectors Q, K, and V. The three key values ​​Q, K, and V have the same dimension. K is transposed and multiplied by Q to obtain the attention matrix. This matrix is ​​then subjected to a softmax operation and finally multiplied by V to obtain the final output O, as shown in the figure. Figure 3 This means, that is:

[0067] O = V·softmax(K) T ·Q)

[0068] Previous methods typically employ convolutional encoders, which often present a dilemma: preserving stylistic details versus building a global receptive field. If this invention aims to obtain a global receptive field, it must perform multiple downsampling operations using a sufficiently deep network, sacrificing stylistic details to gain adequate semantic information. However, in practice, to retain more detailed style information, this invention typically uses only 2-4 downsampling operations. This results in the network not obtaining a sufficient receptive field, nor capturing enough semantic information.

[0069] However, for human pose transformation tasks, the output image is a rearranged set of input image pixels. This requires the network to not only preserve source style information but also establish long-range dependencies on the images. However, using only a convolutional framework is insufficient to meet these requirements. To address this issue, this invention introduces a Transformer into the encoder, giving the network stronger encoding capabilities. Given that the Swin-Transformer can effectively establish long-range dependencies and preserve local features, this invention introduces the Swin-Transformer as the backbone network to extract style features from the source image Is and from the input semantic map I. p Pose features are extracted. Then, this invention employs depthwise convolution in MBConv to track the Swin-Transformer layer and utilizes a shortcut method to preserve source image characteristics. Together, these constitute a downsampling module. In this way, the network not only possesses strong expressive power but also exhibits fast convergence speed and good generalization ability. Figure 4 This is a general overview of the model of this invention. The style feature map F of the encoded result. s It contains style information from the source image, while the pose feature map F p It contains semantic information about the target pose.

[0070] Feature Alignment Network

[0071] Pose transformation moves pixel blocks from a conditional pose to a target pose, establishing dependencies between different pixel blocks. From this perspective, pose guides transmission by finding where to extract conditional pixel blocks and where to place target pixel blocks, while preserving the relationships between pixel blocks. Based on this, this invention proposes a feature alignment network combining Transformer and convolution to establish global dependencies between pixel blocks and local feature relationships within pixel blocks. This helps the pose feature map guide the transmission of texture information by finding where to extract original pixel blocks and where to place target pixel blocks. Figure 5 This invention demonstrates the final module of a feature alignment network. Through multiple alignment blocks, image features are progressively transferred from the source pose to the target pose. Each module uses style features to render pose features, completing the pose-guided transformation task. The input to the first module is a style-style map F. s and pose feature mapping F p When outputting, F p Updated to In the nth alignment module, Updated to F is aligned using n alignment blocks. p The feature output is updated to obtain the final feature output, and then sent to the decoder to generate the final result.

[0072] Since the alignment modules of this invention have the same structure, this invention only uses the last module as an example to illustrate how they work; the other modules are similar. Figure 5 As shown. The alignment module of this invention consists of a convolutional network and a Transformer network. The Transformer network focuses on establishing global dependencies, while the convolutional network focuses on local features. Finally, an attention mask image is generated, where the values ​​of the attention mask image are between 0 and 1, indicating the importance of each pixel position. Then, the results of the two networks are multiplied element-wise to obtain the attention map F. attn for:

[0073]

[0074] Where Tran represents Transformer, C represents Concat, S represents Sigmoid, and × represents element-wise multiplication. Convolutional networks and Transformer networks share a common input, which is the style feature map F. s and pose semantic graph F p The connection was obtained.

[0075] The convolutional network is a residual convolutional layer containing a normalized batch normalization (BN) layer and a ReLU activation function layer. The Transformer network consists of two Transformer modules and a normalized BN layer. This method segments the feature map into pixel blocks and then uses multi-head self-attention to establish relationships between these pixel blocks. Inspired by this invention, the normalized features can then be adjusted. The preposition and bias are determined by two fully connected convolutional layers from F. attn The prediction was obtained. Updated based on settings and deviations Where (c∈C) i ,h∈H i ,w∈W i ) as a representation of equation (2.3):

[0076]

[0077] Where C, H, and W represent the channel, height, and width of the tensor. and These are the modulation parameters learned. yes Enter the i-th... th Output features after layer normalization and activation layers, and These are the mean and standard deviation of channel c.

[0078] Transformer Synthesis Network

[0079] The Transformer synthesis network generates high-quality images that conform to a set semantics from a target pose. By referencing the correct corresponding regions in the paradigm, this synthesis network can more easily generate high-quality output. Throughout the final image synthesis process, it is crucial that the network continuously focuses on the correct corresponding regions. This results in fewer artifacts in the generated image and reduces prediction errors. Figure 6 An example module of the Transformer synthesis network is given.

[0080] To ensure the network consistently retains the correct corresponding regions, this invention first uses alignment samples generated by the alignment network. and style feature diagram F s Perform matrix operations to obtain the correlation matrix M∈R of the aligned samples. N×N N represents the size of the matrix, N = 4096. Each element is a pairwise eigencorrelation matrix, represented as:

[0081]

[0082] in, and Indicates positions u and v and F s The channel-focused features are obtained. The correlation matrix contains target style and semantic information, which is applied to each part of the synthesis stage to obtain a correlation paradigm field to constrain the network to generate correct images. First, following the approach of CocosNet, this invention uses the correlation matrix to warp the original image to obtain aligned feature maps. (S i-1 ), can be represented as:

[0083]

[0084] α represents the weight.

[0085] However, each subsequent synthesis stage requires a correlation matrix of different sizes. To ensure its applicability at each stage, this invention employs pooling layers to adjust its size. However, pooling layers lose the spatial context of the correlation matrix. To address this issue, this invention extracts the spatial context of the feature map. Specifically, this invention extracts the spatial context of the feature map S. i-1 The feature vector is converted using a modified Swin-Transformer module and a 1×1 conv module. This invention utilizes the Swin-Transformer module to globally focus on region features to model the feature map S. i-1The contextual relationship. Then, this invention uses a 1×1 conv module to extract the feature map S. i-1 Spatial scale information. By multiplying the feature vectors by the correlation matrix, a correlation paradigm field E can be obtained. The correlation paradigm field E can preserve both the spatial context of the image and the most relevant pixel matrix. It can be expressed by the formula:

[0086]

[0087] Where × represents element-wise multiplication, and Up represents upsampling.

[0088] Visformer demonstrated the importance of Transformer embedding layers for preserving features. Compared to the original Swin-Transformer design, this invention further introduces consecutive small convolutional layers instead of the original single large convolutional layer. The method of this invention can better preserve the information of each pixel patch, significantly improving the Transformer's ability to model and establish long-range dependencies between pixel patches. Specifically, this invention sets the kernel size to 4, 2, and 1, and the stride to 4, 2, and 1, respectively, in the embedding.

[0089] loss function

[0090] The loss function is used to constrain the network to generate correct images. The total training loss in this invention includes a conditional adversarial loss, an L1 loss, a perceptual loss, and a contextual loss. The total loss function is defined as:

[0091] L total =λ a L adv +λ L1 λ L1 +λ p L per +λ CX L CX

[0092] Where λ a , λ L1 , λ p , λ CX It is a hyperparameter of the equilibrium function.

[0093] Conditional adversarial loss function

[0094] This invention employs two discriminators, namely, appearance discriminator D. A To determine the output image F g Compared with the original image I s (Appearance Consistency) The likelihood of identical human body styles, shape discriminator D p To determine the output image F gWith target attitude I p The degree of alignment (shape consistency). Conditional adversarial loss is defined as:

[0095]

[0096] In the formula, E represents the mathematical expectation, and I... p Let I represent the input semantic graph, and Is represent the distribution of human poses. t Let P represent a real-life image with the appearance of the source image and the target pose, and D represent the distribution of the real-life image. A F represents the appearance discriminator. g P represents the output of a dummy image that has the appearance of the source image and the pose of the target image. g D represents the distribution of the mannequin figures. p This represents a shape discriminator.

[0097] L1 loss function

[0098] L1 loss represents the generated image F g With real image I t The pixel-level difference between the two values ​​is calculated as the absolute value of their error. The penalty is fixed for any difference. Regardless of the input value, it exhibits a stable gradient, preventing gradient explosion and providing a relatively robust solution. Its definition is:

[0099] L1=||F g -I t ||1

[0100] Perceptual loss function

[0101] This invention employs perceptual loss to generate more realistic images. It uses a fixed network, taking the output values ​​and real values ​​as input, to obtain corresponding output features. The deeper the network, the more sophisticated the corresponding output features, and the closer they are to human perception. The perceptual loss will generate the image F... g And real pictures I t The activation maps were obtained through a pre-trained VGG-19 network. Then, the L1 distance between the activation maps was calculated, and the results are as follows:

[0102]

[0103] In VGG-19 networks, this invention selects It serves as the activation layer following the ReLU42 layer, as this layer primarily contains high-level semantics.

[0104] Context loss function

[0105] This invention employs a proposed semantic loss method to match and generate image F.g and real image I t The difference between them. The context loss function can guide the spatial deformation of the target, reduce texture distortion, and improve the reasonableness of the output. This invention calculates the context loss function as:

[0106] L CX =-log(CX(F) l (F g ),F l (I t )))

[0107] Here F l (F g ) and F l (I t ) represent the image F respectively g and I t The feature maps in the pre-trained VGG19 network are used. This invention uses an activation layer l = relu{3_2,4_2} to compute the context loss. CX represents the similarity measure between matching features, taking into account both pixel semantics and contextual information of the entire image.

[0108] Experimental results

[0109] This section provides a comprehensive evaluation and validation of the framework and method. The invention begins by introducing the dataset used. Next, it describes the various metrics used to evaluate the framework. The training strategies and details are also presented. Furthermore, the method is compared with other methods from both qualitative and quantitative perspectives. Finally, the effectiveness of the proposed components is validated through ablation experiments. Figure 7 This image shows the results generated by this invention on the Market-1501 dataset. The leftmost green image of a man is the input style map. Columns two, four, and six are the input target semantic maps, representing different human poses from various angles, including six different viewpoints. Columns three, five, and seven are the generated target images. The target images contain the style information of the original image and the corresponding human pose from the semantic map. Although the input only provides a frontal view of the human body, the image shows that this invention successfully generated realistic images of the corresponding poses, indicating that the network can successfully predict various angles and poses of the human body. Furthermore, it successfully generated realistic images in some details, such as the human hair, hand movements, and feet. Figure 8This is the result image generated by this invention on the Deep Fashion In-shop dataset. The leftmost blue image of a woman is the input style map. Columns two, four, and six are the input semantic maps, represented by heatmaps, which depict different human poses. Columns three, five, and seven are the corresponding generated target images. The target images contain both the style information of the original image and the corresponding human pose from the semantic map. This invention also successfully generated realistic images.

[0110] Dataset

[0111] This invention conducted experiments on two datasets: the Market-1501 human pose dataset and the DeepFashion In-shop clothing retrieval benchmark dataset. The images in Market-1501 have a lower resolution of 128×64, and vary in viewpoint, background, and lighting, with complex backgrounds. The DeepFashion In-shop model images feature fashionable clothing, clean backgrounds, different human poses, and different viewpoints, with a resolution of 256×256. This invention collected 263,632 training pairs and 12,000 test pairs from 101,966 training pairs in the DeepFashion dataset and 8,570 test pairs in Market-1501. Notably, the identities of the people in the test set differ from those in the training set.

[0112] Evaluation indicators

[0113] This invention utilizes the Learning Perceptual Image Patch Similarity (LPIPS) algorithm to calculate the reconstruction error between the generated image and the real image at the perceptual level. Simultaneously, this invention uses FID to calculate the initial distance between the generated image and the real image to measure the realism of the generated image. It calculates the distance between the Wasserstein-2 generated image distribution and the real image. Furthermore, this invention employs Peak Signal-to-Noise Ratio (PSNR) to calculate the pixel-level error between the generated image and the real image.

[0114] Training details

[0115] The model trained in this invention was trained on the Deep Fashion dataset (256,256 images) and the Market-1501 dataset (12,864 images). This invention employs the Adam optimizer with β1 = 0 and β2 = 0.999. The learning rates were set to 0.0001 and 0.0002 respectively, and the generator and discriminator followed the TTUR rule. The weights of the loss function terms were set to λ. a =10, λ p =0.0001, λ L1 =10, λ CX=0.001. The training framework of this invention is implemented in PyTorch using an NVIDIA RTX 3090 GPU with 24GB of memory. Furthermore, this invention uses three alignment modules in the feature alignment network and two synthesis modules in the Transformer synthesis network. During training, the input of this invention is the concatenation of the original image and the target pose image, which are then fed into the network model. When training on the Deep Fashion dataset, the input resolution is 256×256, the feature map resolution for the alignment module is 16×16, and the feature map resolutions for the synthesis module are 16×16, 32×32, and 64×64, respectively. When training on the Market-1501 dataset, the input resolution is 128×64, the feature map resolution for the alignment module is 16×8, and the feature map resolutions for the synthesis module are 16×8, 32×16, and 64×32, respectively. This invention employs end-to-end training, training for a total of 150 rounds on the Deep Fashion dataset. The learning rate decreases at round 75 and approaches 0 at round 150.

[0116] This invention was trained on the Market-1501 dataset for 200 epochs, with the learning rate decreasing at epoch 100 and approaching 0 at epoch 200. The batch size for all training was set to 8. The network structure of this invention is shown in Table 1, where k is the kernel size and s is the stride of the convolution. Table 2 shows the overall structure of the model and the specific parameters used in each network. The first column represents the proposed model, namely the Transformer encoder, alignment module, and synthesis network. The second column shows the number of layers in each model, with the Alignment Block representing the module in the proposed alignment network and the Synthesis Block representing the module in the proposed Transformer synthesis network. The third column shows the output image size for each model, represented as a three-dimensional matrix of height, width, and channels.

[0117] Table 1. Model Structure

[0118]

[0119] Qualitative comparison

[0120] This invention performs a qualitative comparison of the Market-1501 and Deep Fashion datasets, employing several advanced methods, including Pose-Attn, BiGraph, XingGAN, PoNA, GFLA, PISE, and Pose2Pose. Figure 9 and Figure 10 Some qualitative comparisons were shown.

[0121] For the Deep Fashion dataset, PATN proposed several cascaded attention transformation modules, but these modules have low transformation efficiency. For example... Figure 9 As shown in the fourth column, the generated images have a blurry style and texture, and it produced incorrect results in the last row. Figure 9 The fifth column shows the results generated by BiGrapth. As can be seen from the image, this method fails to generate sharp images with blurry textures because the model lacks a global receptive field, thus failing to predict complex textures. The sixth and seventh columns show the results generated by XingGAN and PoNA. These two methods utilize a non-local attention module to obtain a global receptive field, but the key-value pairs in the non-local module focus on different regions: the style map and the semantic map. This reduces the network's generative ability, resulting in low-quality images. Figure 9 In the third row, all of them generated incorrect result images. Therefore, it is difficult for these two methods to generate reasonable images. The eighth column shows images generated by GFLA. They use a flow-based method, which uses the original style map, original pose map, and target pose map to predict a global flow field, generating better results. However, they require two stages of training, and they cannot predict some subtle textures and shapes in invisible areas of the source image. For example, on the human arm in the last row, they generated blurry artifacts. Moreover, this method is not end-to-end trained, requiring more training time and resources. The ninth column shows the result images generated by PISE. This method introduces an additional semantic map to assist in completing the human pose transformation task. However, the increased computation does not produce satisfactory results. The method of this invention matches the paradigm of semantic mapping and alignment, and introduces the Transformer into the method of this invention. It can generate realistic images with correct pose and vivid details.

[0122] The Market-1501 dataset, with its low resolution and complex background, presents a challenge in generating accurate, high-resolution images. However, the method presented in this invention demonstrates powerful modeling and generative capabilities, producing more natural and sharper images. Figure 10 As shown, the PATN method in the fourth column produces a rather blurry result image. The BiGraph and XingGAN methods in the fifth and sixth columns show a significant difference in color perception compared to the target image. Compared to the PoNA and GFLA methods in the seventh and eighth columns, the method of this invention can recover more details, has fewer artifacts, and generates a high-resolution image.

[0123] Quantitative comparison

[0124] Table 2 presents the quantitative results of the model of this invention compared with several state-of-the-art methods: Pose-Attn, PoNA, XingGAN, BiGraph, GFLA, Pose2Pose, and PISE. Since Pose2Pose and PISE do not provide pre-trained models on the Market-1501 dataset, this invention is only compared with them on the Deep Fashion dataset. For Deep Fashion, the results of this invention have the best FID score, exceeding the current best method by 1.71 points. This means that the initial distance between the images generated by this invention and the real images is closer. It also proves that the method of this invention generates more realistic and closer images to reality. Furthermore, this invention also uses the LPISP algorithm to calculate perceptual similarity and the PSNR algorithm to measure pixel-level error. The method of this invention achieves the best results in both LPIPS and PSNR. In terms of LPIPS, this invention scores 0.0116 points higher than the best method in the table. In terms of PSNR, this invention also scores 0.59 points higher than the best method in the table. This indicates that the method of the present invention has smaller pixel-level errors, is more consistent with the target image in shape and texture, and is more perceptually close to the real image. For Market-1501, quantitative results show that the image generated by the method of the present invention is closer to the real image in shape and texture, and the indicators of the present invention are better than most other methods, even with low resolution of the conditional image and significant changes in pose and background.

[0125] Table 2 Quantitative Comparison

[0126]

[0127] ablation experiment

[0128] This invention trains several ablation models to verify the hypothesis and the effectiveness of each component.

[0129] 1) Global Encoder: This invention uses only the Transformer as the backbone network of the encoder, enabling the network to obtain a global receptive domain, but lacking the inductive bias capability of convolution. This invention uses a global encoder to verify whether the network can generate correct images by focusing only on global information. This invention first uses convolution to downsample the image to 128×128. This is because when the resolution is too high, the Transformer network consumes enormous computational resources, so this invention first downsamples the image to a suitable resolution. Then, a Transformer backbone network is used to extract image features.

[0130] 2) Local Encoder: This invention uses only convolution as the backbone network of the encoder, so that the network only pays attention to local information and does not have the ability to model global dependencies, to compare the advantages of convolutional backbone networks and Transformer backbone networks.

[0131] 3) Feature-Alignment-Free Network Model (w / o FA): In this model, the feature alignment network is removed from the full model to verify the impact of features on the network's effect on the resulting image. Without a feature alignment network, image information cannot be transformed from the source image to the target image. This invention directly feeds the image, after passing it through an encoder, into a Transformer synthesis network to generate the final image.

[0132] 4) Model without Transformer Synthesis Network (w / o SN): In this model, the Transformer Synthesis Network is removed from the complete model to verify its impact on the resulting image. Without the Transformer Synthesis Network, the network cannot fully focus on the correct regional features.

[0133] 5) Full Model: This invention uses the framework proposed in this invention in this model, including the convolutional and Transformer encoders, feature alignment network and Transformer synthesis network proposed in this invention.

[0134] Table 3 Quantitative Comparison of Ablation Experiments

[0135]

[0136] The quantitative results of the Deep Fashion test images are shown in Table 3. The global encoder and local encoder show good values. However, the FID value of the local encoder is 5.19 higher than that of the global encoder. This indicates that the local bias of the convolutional network has a significant impact on the entire model. However, the Transformer's ability to establish global dependencies is also indispensable. Compared to the global encoder and local encoder, the encoder composed of convolution and Transformer in this invention produces better results than pure convolution and pure Transformer encoders. This shows that combining convolution and Transformer in this invention allows each to leverage its inherent advantages, enhancing the network's capabilities. Furthermore, the feature alignment network gradually transfers the original image to the target image through several cascaded alignment modules. This aligns the source style map with the target style map, providing better prior information for synthesizing the final image. The global receptive field provided by the Transformer synthesis network is also crucial during image synthesis. This module ensures that the network retains the most relevant regions and provides the correct relevant pixels when synthesizing the final image, thus generating better image quality. The evaluation metrics FID, PSNR, and LPIPS demonstrate this.

[0137] Figure 11 This section presents some intuitive visual results from ablation studies. It can be seen that the global encoder focuses only on global information of the image, neglecting local details. For example, in the first row of the fourth column, the details on the shorts are not generated. The local encoder focuses only on local information, but cannot fully establish the image's context. As shown in the second row of the fifth column, the generated human body's arms and hands are not semantically correct. This is due to neglecting the image's context. While both encoders can generate correct structural images, their focus on either global or local information results in unsatisfactory texture details or overall image accuracy. Image information transformation occurs in the alignment network, where image features are gradually transformed from the original image to the target image through several cascaded alignment modules based on semantic information. Without a feature alignment network, the transformation efficiency is low, failing to completely convert the image from the original to the target image, and the resulting image will be partially blurred, as shown in the fourth row of the fifth column for the shirt. The Transformer synthesis network provides the correct region features for the synthesized final image, preventing the network from synthesizing incorrect region features, which is crucial for the final image. Without the Transformer synthesis network, the model cannot retain more details and will generate some incorrect region features. Furthermore, it cannot generate correct images, as shown in the seventh column, third row (shorts), and fourth and fifth row (shirts).

[0138] This invention explores a Transformer-based method for human pose transformation. The method first generates an image aligned with the target pose, then progressively generates high-quality images. Experimental results show that the model can simultaneously consider global and local information, generating realistic images with lifelike details. Furthermore, ablation studies validate the effectiveness of each design component. However, the method of this invention has some drawbacks. The model of this invention may require more computation than previous convolution-based methods. In the future, this invention will improve the efficiency of the model method by introducing new Transformers for other spatial deformation tasks, such as facial animation.

[0139] 1. This invention proposes a Transformer-based feature-aligned pose transformation network framework that can utilize global dependencies while preserving detailed features. Experimental results on a human pose transformation network demonstrate the flexibility and superiority of this proposed method.

[0140] 2. This invention proposes an alignment network based on Transformer and multi-cascaded block convolution, which can align source and target images in the embedding and untangling feature space.

[0141] 3. This invention proposes a Transformer synthesis network to maintain the features of the corresponding regions in the samples, and repeatedly generates high-quality images by distorting the correlation matrix between the results of the previous stage and the alignment features and the source image.

[0142] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A human pose transfer method based on image feature alignment, characterized in that, include Input semantic graph I p The first Transformer encoder is input as input, and its output is input to the first MBConv. The output of the first Transformer encoder is tracked, and the pose feature map is output from the first MBConv to obtain the style mapping F. p ; The source image Is is input to the second Transformer encoder, and the output of the second Transformer encoder is input to the second MBConv. The output of the second Transformer encoder is tracked, and the style feature map is extracted from the output of the second MBConv to obtain the pose feature map F. s ; The feature alignment network consists of n feature alignment network modules, where n ≥ 2. Each feature alignment network module includes a first input, a second input, and an output, with a style mapping F. p The first input to the first feature alignment network module is the pose feature map F. s The second input of each feature alignment network module is input in parallel. The output of the previous feature alignment network module is input as the first input of the next feature alignment network module. The output of the nth feature alignment network module is... It is the output of the feature alignment network. Output of the feature alignment network and style feature diagram F s Through matrix operations, the correlation matrix M of the aligned samples is obtained. The correlation matrix M includes target style and semantic information. The correlation matrix M is multiplied with the source image Is to output the aligned feature map S1. The Transformer synthesis network consists of n synthesis network modules, where n ≥ 2. Each synthesis network module includes a first input, a second input, and an output. The aligned feature map S1 is input to the first input of the first synthesis network module. The correlation matrix M is input to the second input of each synthesis network module. The output of the previous synthesis network module is input to the first input of the next synthesis network module. The nth synthesis network module outputs the feature map F. g .

2. The human pose transfer method based on image feature alignment according to claim 1, characterized in that, in, The feature alignment network module includes a Transformer network and a convolutional network, i = 2, ..., n, where i is the output of the (i-1)th feature alignment network module. As the first input to the i-th feature alignment network module, the pose feature map F s As the second input to the i-th feature alignment network module, the first input to the i-th feature alignment network module includes parallel first input branches. and the first input branch two First input branch 1 With the second input F s Through the connection layer, the outputs of the connection layer are input to the Transformer network and the convolutional network respectively. The dot product operation of the outputs of the Transformer network and the convolutional network yields the attention diagram F. attn Note Figure F attn Including the first output F in parallel attn And the second output F attn First output F attn Through a 1×1 convolution, and then with the first input branch two... The dot product operation yields the first result, and the second output is F. attn The output of the i-th feature alignment network module is obtained by performing a 1×1 convolution followed by adding it to the first result.

3. The human pose transfer method based on image feature alignment according to claim 1 or 2, characterized in that, in, The synthesized network module includes a Transformer network, a 1×1 convolution, a reshape module, and an upsampling module. The output of the (i-1)th synthesized network module is a feature map S of C×(i-1)H×(i-1)W. i-1 Let i = 2, ..., n, be the first input of the i-th synthetic network module, and let the correlation matrix M be the second input of the i-th synthetic network module. The first input of the i-th synthetic network module includes a parallel first input branch—feature map S. i-1 and the first input branch feature map S i-1 The first input branch is a feature map S. i-1 After passing through a Transformer network and a 1×1 convolutional network, we obtain a 1×H×W feature map 1. Feature map 1 is then reshaped to obtain the first vector of HW. The correlation matrix M is then passed through the average pooling layer of the (i-1)th synthesis network module to obtain the output vector. The first vector is multiplied by the output vector to obtain the second vector of HW. The second vector of HW is then reshaped to obtain a 1×H×W feature map 2. Feature map 2 and feature map S are then compared... i-1 The dot product operation yields feature map 3, which is C×(i-1)H×(i-1)W. Feature map 3 is then upsampled to obtain feature map S, which is C×iH×iW. i .

4. The human pose transfer method based on image feature alignment according to claim 3, characterized in that, The attention graph F of the i-th feature aligned with the network module attn It can be expressed by the following formula: In the formula, Tran represents Transformer, C represents Concat, S represents Sigmoid, and × represents element-wise multiplication.

5. The human pose transfer method based on image feature alignment according to claim 4, characterized in that, The output of the i-th feature alignment network module It can be expressed by the following formula: In the formula, c∈C i ,h∈H i ,w∈W i C, H, and W represent the channel, height, and width of the tensor; and This represents the modulation parameters being learned; yes Enter the i-th... th Output features after layer normalization and activation; and yes The mean and standard deviation in channel c.

6. The human pose transfer method based on image feature alignment according to claim 5, characterized in that, The correlation matrix M is represented by the following formula: In the formula, M(u,v) represents the correlation matrix M. and Indicates positions u and v and F s The channel concentration characteristics.

7. The human pose transfer method based on image feature alignment according to claim 6, characterized in that, Feature map S i-1 It can be expressed by the following formula: Representing feature map S i-1 α represents the weight, I s (v) represents the source image distorted by the correlation matrix M.

8. The human pose transfer method based on image feature alignment according to claim 7, characterized in that, Feature map S i It can be expressed by the following formula: In the formula, Up represents upsampling, E represents the relevant paradigm field, and × represents element-wise multiplication.

9. The human pose transfer method based on image feature alignment according to claim 8, characterized in that, Total loss function L total : L total =λ a L adv +λ L1 L L1 +λ p L per +λ CX L CX In the formula, λ a , λ L1 , λ p , λ CX L is a hyperparameter of the equilibrium function. adv L1 represents the conditional adversarial loss, and L1 represents the L1 loss function. per L represents the perceptual loss function. CX This represents the context loss function.

10. The human pose transfer method based on image feature alignment according to claim 9, characterized in that, In the formula, E represents the mathematical expectation, and I... p Let I represent the input semantic graph, I represent the distribution of human poses, and Is represent the source image input. t Let P represent a real-life image with the appearance of the source image and the target pose, and D represent the distribution of the real-life image. A F represents the appearance discriminator. g P represents the output of a dummy image that has the appearance of the source image and the pose of the target image. g D represents the distribution of the mannequin figures. p Represents a shape discriminator; L1=||F g -I t ||1 In the formula, Indicates the activation layer L CX =-log(CX(F l (F g ),F l (I t ))) In the formula, F l (F g ) and F l (I t ) respectively represent F g and I t The feature maps in the pre-trained network, where CX represents the similarity measure between matching features.

Images