Human body and posture transformation image generation method and device

Abstract

The invention relates to the technical field of computer vision, and discloses a human body and posture transformation image generation method and device. Comprising the following steps: respectively compressing a source human body image, a target human body image, a source posture image and a target posture image into a tensor space by using an encoder to form a source human body tensor, a target human body tensor, a source posture tensor and a target posture tensor; the tensor sequentially passes through a plurality of functional modules in the 'W'-type migration network, and each functional module comprises an information introduction module, a weight map module, a tensor alignment module and an attention module; and utilizing the two trained 'W' networks to respectively predict a target human body tensor and a target attitude tensor, and then respectively obtaining a predicted target human body image and a predicted target attitude image through a decoder. According to the method, the human body and posture transformation task is completed at the same time, and the generated target image is ensured to have high-quality texture features.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method and a device for generating images of human bodies and posture transformations. Background Art

[0002] As a subfield of machine learning, deep learning uses multi-layer neural networks to automatically learn and extract data features, and has achieved breakthroughs in image recognition, natural language processing, speech recognition, and recommendation systems. Its development has benefited from the advancement of big data, computing power, and optimization algorithms, which can reduce reliance on manual feature engineering and thus improve model performance. Computer vision aims to enable machines to understand the content of images and videos, covering object recognition, image segmentation, posture estimation, and motion analysis. With the help of convolutional neural networks, the accuracy and efficiency of this field have been significantly improved, which has promoted the development of applications such as autonomous driving and medical image analysis, but it also faces challenges such as lighting changes, occlusion, and real-time processing.

[0003] In this context, pose transfer and pose estimation have become important research areas. Pose transfer aims to transfer the motion characteristics of one object to another to achieve natural and smooth animation, while pose estimation focuses on accurately identifying joint positions and poses from images or videos. Through deep learning technology, these applications have shown broad potential, although they still need to overcome challenges in accuracy, real-time performance, and complex scenarios. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a method and device for generating human body and posture transformation images, which can efficiently generate human body and posture transformation images, overcoming the problems caused by slow generation speed of human body images or posture images and low efficiency of single task in previous work.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a method for generating a human body and posture transformation image, comprising the following steps:

[0007] Acquire a source input image and source guidance information; the source input image includes a source human image and a source posture image; the source target information includes a target human image and a target posture image;

[0008] Using the trained human encoder, the source human image and the target human image are compressed into the tensor space to form a source human tensor and a target human tensor;

[0009] The source pose image and the target pose image are compressed into the tensor space using the trained pose encoder to form the source pose tensor and the target pose tensor;

[0010] Connecting the source human body tensor with the target human body tensor to form posture guidance information; connecting the source posture tensor with the target posture tensor to form human body guidance information;

[0011] The source human body tensor combined with the human body guidance information is used as input information and input into the trained human body 'W' type migration network to obtain the predicted human body tensor.

[0012] The source posture tensor combined with the posture guidance information is used as input information and input into the trained posture 'W' type migration network to obtain the predicted posture tensor;

[0013] The human body 'W' type migration network has the same structure as the posture 'W' type migration network;

[0014] Input the predicted human body tensor into the trained human body decoder to obtain the predicted human body image;

[0015] The predicted pose tensor is input into the trained pose decoder to obtain the predicted pose image.

[0016] Furthermore, the end of the human encoder and the posture encoder and the front end of the human decoder and the posture decoder are provided with a codebook mechanism;

[0017] The codebook mechanism converts the input tensor Replaced by the nearest tensor in a learnable randomly initialized tensor table , and use the codebook loss function to learn the parameters in the tensor table.

[0018] Furthermore, the end of the human body encoder and the posture encoder and the front end of the human body decoder and the posture decoder are provided with a layer Attention layer;

[0019] The encoder structure of the human body encoder and posture encoder is expressed as:

[0020]

[0021] The decoder structure of the human body decoder and posture decoder is expressed as:

[0022]

[0023] in, is the attention mechanism layer, is the residual layer, For the The residual layer in the layer network layer, the encoder is , the decoder is , the encoder input is , the decoder input is .

[0024] Furthermore, the functional modules include a residual connection module, an information import module, a weight map module, a tensor constraint alignment module, an attention module and a channel weight redistribution module;

[0025] The information import module, weight graph module, tensor constraint alignment module, and attention module are connected in series in pairs using residual links, and then connected in parallel with the weight module. Residual connection modules are introduced at the input and output ends of the functional module to change the size and number of channels of the input tensor.

[0026] Furthermore, the information introduction module is used to perform the following steps:

[0027] Will guide the tensor Split into current boot information With target guidance information ;

[0028] Use the target tensor go through Use after module projection Module Updates , with the source tensor use Module Updates , the two are recombined to form a new guidance tensor , and update the front and back boot information With target guidance information Splicing together Add after layer The residual connection of the information introduction module outputs the current tensor , which is expressed as follows:

[0029]

[0030]

[0031]

[0032] in, The module is a gate control update module. Indicates that the tensor is Branch, is all The output tensor of the layer network, represents splicing along the channel dimension, represents the convolutional layer module, represents the residual connection layer module, represents the target tensor, Represents the updated tensor , represents the input guide tensor, Indicates the tensor to be migrated as input to the information introduction module. Indicates the current boot information of the information import module. Indicates the information introduced into the module target guidance information, and The input tensor of the information introduction module, and It is the guide tensor and current tensor output by the information introduction module. The superscript l represents the functional module of the lth layer.

[0033] Furthermore, the weight map module is used to perform the following steps:

[0034] The guide tensor and current tensor output by the information introduction module and When passing through the weight map module, the information is introduced into the guidance tensor output by the module The whole is updated through the convolutional layer to obtain the guide tensor output by the weight map module , the guide tensor output by the weight map module Decomposed into the target guidance tensor output by the weight map module The current guide tensor output by the weight map module After taking the difference, input it into the hyperbolic tangent function Get the tensor weight graph , the weight graph module inputs the tensor to be migrated With tensor weight graph After the point multiplication, add the convolution layer The residual connection of the weight map module obtains the current tensor output by , as follows:

[0035]

[0036]

[0037]

[0038] in, Indicates that the tensor is Branch, is all The output tensor of the layer network, represents splicing along the channel dimension, represents the convolutional layer module, Represents the tensor after the weight map module is updated , represents the weight map module input guide tensor, Represents the weight graph module input tensor to be migrated, Depend on and Spliced ​​along the channel dimension, Represents the current guidance information of the weight map module, and Input tensor for weight graph module, and The guide tensor and current tensor output by the weight map module, represents the dot multiplication operation between tensors, represents the hyperbolic tangent function, Represents a weight graph module.

[0039] Furthermore, the execution method of the tensor alignment module includes:

[0040] The guide tensor output by the weight map module The whole is updated through the convolution layer to obtain the guide tensor output by the tensor alignment module , using the aligned tensor Approximate with a series of convolutions The tensor alignment module currently guides the tensor To the tensor alignment module target guide tensor The transformation aligns the tensor to the current guide tensor of the module Through two learnable fully connected modules and Activation function proximity tensor alignment module target guide tensor To simulate the alignment of tensors , and finally the current tensor output by the weight map module Aligning Tensors The guide tensor output by the tensor alignment module in the task of doing matrix operations close to symmetry Get the hidden tensor , and then after the convolution layer, add the current tensor output by the tensor alignment module The residual connection of the tensor alignment module obtains the current tensor output , and use the tensor alignment method to control Effect.

[0041] The current tensor output by the tensor alignment module With the guide tensor When passing through the attention module, The whole is updated through the convolutional layer to obtain the guidance tensor output by the attention module ,Will Respectively with the guide tensor The current guide tensor in and the target guide tensor The concatenation is fed into the cross attention mechanism and updated using the cross attention mechanism And then add after the convolution layer The residual connection of the attention module obtains the current tensor output by .

[0042] Furthermore, the training methods of the human encoder, posture encoder, human decoder and posture decoder include:

[0043] Acquire a training data set; the training data set includes at least human body images and posture images;

[0044] The human encoder, posture encoder, human decoder and posture decoder are trained using the training data set and the loss function to obtain the trained human encoder, posture encoder, human decoder and posture decoder.

[0045] Furthermore, the method of obtaining the training data set includes:

[0046] Human image data is obtained from the DeepFashion public human dataset.

[0047] The posture estimation algorithm OpenPose is used to extract the posture of human image data to obtain human posture data.

[0048] Pair the same human body image with the posture image data, and take the paired data of two sets of different actions of the same person to form a set of training data, so that one set of training data contains the source human body image, the target human body image, the source posture image and the target posture image, and thus the training data set is obtained.

[0049] Furthermore, the training dataset and the loss function are used to train the human encoder, the pose encoder, the human decoder and the pose decoder, including:

[0050] Based on the loss function used to train the overall VQVAE framework, the human encoder and human image decoder are trained using images of the same human in the training dataset;

[0051] Based on the loss function used to train the overall VQVAE framework, the pose human encoder and pose image decoder are trained using images of the same pose in the training dataset.

[0052] Furthermore, the training method of the human body 'W' type transfer network and the posture 'W' type transfer network includes:

[0053] Get the tensor training dataset;

[0054] Based on the tensor training data set and the loss function for training the overall migration network 'W', the human body 'W' type migration network and the posture 'W' type migration network are trained by using the tensor alignment method to obtain the trained human body 'W' type migration network and the posture 'W' type migration network.

[0055] Furthermore, the tensor training data set is obtained, including:

[0056] Use the trained human image encoder to map the human image into the tensor space and transform it into a human tensor, and pass it through the human tensor codebook. Use the trained posture image encoder to map the posture image into the tensor space and transform it into a posture tensor, and pass it through the posture tensor codebook.

[0057] The human body tensor and the posture tensor are paired data, and two sets of paired data of different actions of the same person are taken to form a set of training data, so that one set of training data contains the source human body tensor, the target human body tensor, the source posture tensor and the target posture tensor, thereby obtaining a tensor training data set.

[0058] In a second aspect, the present invention provides a device for generating images of human body and posture transformation, including a processor and a storage medium;

[0059] The storage medium is used to store instructions;

[0060] The processor is used to operate according to the instructions to execute the steps of the method according to the first aspect.

[0061] Compared with the prior art, the present invention provides a method for generating human body and posture transformation images, which has the following beneficial effects:

[0062] (1) The present invention uses a trained encoder, decoder and two symmetrical 'W'-shaped networks to directly convert the original image and the target image, and can efficiently generate transformed images of human bodies and postures, overcoming the problems of slow generation speed of human body images or postures and low efficiency of single tasks in previous work; (2) The present invention uses a 'W' type network and the functional modules therein, and adopts a residual connection module, an information import module, a weight map module, a tensor constraint alignment module, an attention module and a channel weight redistribution module to process the tensor, thereby realizing the conversion of the image tensor, realizing the conversion of the human body and posture and improving the details of the converted image.

[0063] (3) The human body image and posture image generation method of the present invention can be mainly applied in four fields:

[0064] 1) In the field of film production, this method can replace real people to complete difficult and dangerous actions; 2) In the field of recognition, this method can be used to identify specific dangerous actions and issue alarms to reduce the difficulty of recognition; 3) In the field of virtual reality, this method can be used to guide and design the actions of virtual characters; 3) In the field of games, this method can be used to capture motion using character skeletons.

[0065] (4) The human body image and posture image generation method of the present invention utilizes the characteristic that posture migration and posture estimation tasks have partially overlapping features, designs a method for simultaneously training the two tasks, and accelerates the training speed of the two tasks by aligning the constraints in tensor space. Based on the symmetric acceleration of the 'W' type network, the present invention shortens the training time while completing the posture migration and posture estimation tasks at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of the method for generating human body and posture transformation images based on symmetric acceleration of 'W' type network of the present invention;

[0067] Figure 2 It is the overall structure diagram of the method for generating human body and posture transformation images based on symmetric acceleration of 'W' type network of the present invention;

[0068] Figure 3 The structure diagram of the encoder and decoder in the method for generating human body and posture transformation images based on symmetric acceleration of 'W' type network of the present invention;

[0069] Figure 4 The overall structure diagram of the 'W' type network in the method for generating human body and posture transformation images based on symmetric acceleration of the 'W' type network of the present invention;

[0070] Figure 5 It is a block structure diagram of the core functional module RA in the 'W' type network of the human body and posture transformation image generation method based on the symmetric acceleration of the 'W' type network of the present invention;

[0071] Figure 6 This is a block structure diagram of the information introduction module Proj in the core function module RA block of the 'W' type network of the human body and posture transformation image generation method based on the symmetric acceleration of the 'W' type network of the present invention;

[0072] Figure 7 It is a block structure diagram of the weight map module Wt in the core function module RA block of the 'W' type network of the method for generating human body and posture transformation images based on symmetric acceleration of the 'W' type network of the present invention;

[0073] Figure 8 This is a block structure diagram of the tensor constraint alignment module Tran in the core function module RA block of the 'W' type network of the human body and posture transformation image generation method based on symmetric acceleration of the 'W' type network of the present invention;

[0074] Fig. 9 This is a block structure diagram of the attention module Attn in the core functional module RA block of the 'W' type network of the human body and posture transformation image generation method based on symmetric acceleration of the 'W' type network of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Embodiment 1:

[0077] This embodiment proposes a method for generating human body and posture transformation images based on symmetric acceleration of a 'W' type network. The flowchart and structure diagram of the overall framework are shown in FIG. Figure 1 and Figure 2 As shown, the specific steps include:

[0078] Step 1: Make a dataset.

[0079] Step 1.1: Obtain human image data from the DeepFashion public human dataset.

[0080] Step 1.2: Use the posture estimation algorithm OpenPose to extract the posture of the human image data and obtain the human posture data.

[0081] Step 1.3: Pair the same human image and posture image data, take two sets of paired data of different actions of the same person to form a set of training data, so that one set of training data contains the source human image, the target human image, the source posture image and the target posture image, and thus make the entire training data set.

[0082] Step 2: Train the encoder, decoder and codebook

[0083] Step 2.1: Use the human images in the training dataset to train the human encoder, human image decoder, and human tensor codebook, and use the pose images in the training dataset to train the pose encoder, pose image decoder, and pose tensor codebook.

[0084] Step 2.2: Use loss functions such as perceptual loss and structural loss to constrain the model during training.

[0085] Step 2.3: Constrain the codebook during training using methods such as gradient-free optimization.

[0086] Step 2.3: Use the trained human image encoder to map the human image into the tensor space and transform it into a human tensor, and pass it through the human tensor codebook. Use the trained posture image encoder to map the posture image into the tensor space and transform it into a posture tensor, and pass it through the posture tensor codebook.

[0087] Step 2.4: Pair the human body tensor and the posture tensor in the same way as in step 1.3 to form a training data set in tensor form.

[0088] Step 3: Train the 'W'-type transfer network.

[0089] Step 3.1: Use the source human tensor, source posture tensor and target posture tensor in the tensor training dataset to train the human 'W' type transfer network, and use the source posture tensor, source human tensor and target human tensor in the tensor training dataset to train the posture 'W' type transfer network.

[0090] Step 3.2: During training, the tensor will pass through several core functional modules of different sizes to assist migration.

[0091] Step 3.3: During training, the tensor in the core functional module will pass through the information introduction module for introducing guidance information, the weight graph module for forming weight updates according to the difference between the input and the target, the tensor constraint alignment module for simultaneous dual-task training and accelerated training, and the attention module for obtaining richer target features.

[0092] Step 3.4: Use a loss function such as style loss to constrain the model during training.

[0093] Step 3.5: Use the tensor constraint alignment loss function during training to simultaneously train two 'W'-type migration network models and accelerate the training of the two 'W'-type migration network models.

[0094] Step 4: Use the trained human body 'W' type transfer network to input the source human body tensor, the source posture tensor and the target posture tensor to predict the predicted target human body tensor. After passing the human body tensor codebook specification, the final output predicted target human body image is obtained through the human body image decoder. Use the trained posture 'W' type transfer network to input the source posture tensor, the source human body tensor and the target human body tensor to predict the predicted target posture tensor. After passing the posture tensor codebook specification, the final output predicted target posture image is obtained through the posture image decoder.

[0095] Step 4.1: Obtain source input image and source guidance information; the source input image includes a source human body image With the source pose image ; The source target information includes a target human image and the target pose image ;

[0096] Step 4.2: Utilize the trained human encoder Source human image With the target human image Compressed into tensor space to form the source human body tensor With the target human body tensor ;

[0097] Using the trained pose encoder The source pose image With the target pose image Compress to tensor space to form the source posture tensor and the target pose tensor ;

[0098] Step 4.3: Convert the source human body tensor With the target human body tensor Together they form posture guidance information ; The source posture tensor and the target pose tensor Connection, forming human body guidance information ;

[0099] Step 4.4: Convert the source human body tensor Combined with human guidance information As input information, input the trained human 'W' type transfer network , get the predicted human body tensor ,

[0100] The source pose tensor Combined with posture guidance information As input information, input the trained posture 'W' type transfer network , get the predicted posture tensor ;

[0101] The human 'W'-shaped migration network and posture 'W' type migration network The structure is exactly the same;

[0102] Step 4.5: Feed the predicted human tensor into the trained human decoder , get the predicted human image ;

[0103] Feed the predicted pose tensor into the trained pose decoder , get the predicted posture image .

[0104] Specifically, the method includes:

[0105] Create a human body and posture image training dataset and pair human body images Posture Image Then two different groups of the same person form a training data set, namely the source human image With the target human image and source pose image With the target pose image Together they constitute a training data set. By using the perceptual loss function, The loss function and the MS-SSIM loss function are and Train the human encoder separately , Human Image Decoder , Human pose encoder Human pose decoder ; The encoder and It consists of a series of continuously downsampling convolutional layers, and the input is a human body image. With pose image , the outputs are human body tensors With the attitude tensor ;Decoder and It consists of a series of continuously upsampling convolutional layers, and the input is the human body tensor With the attitude tensor , the outputs are the target human images With the target pose image ;

[0106] Utilizing the Human Body Encoder Source human image With the target human image Compressed into tensor space to form the source human body tensor With the target human body tensor ;

[0107] Using the Human Pose Encoder The source pose image With the target pose image Compress to tensor space to form the source posture tensor and the target pose tensor ;

[0108] Construct two symmetrical 'W'-shaped migration networks with the same structure but different inputs and outputs and , where the 'W'-shaped migration network consists of a series of core functional modules For a 'W' type migration network In this method, As input information and Together they form guidance information train predict ; For 'W' type migration network In this method, As input information and Together they form guidance information train predict ;

[0109] There are several functional modules of different sizes in the 'W'-shaped migration network , functional modules Using guidance information Constantly guiding the tensor To the target tensor Migration; Functional modules Contains information import module , weight graph module , tensor alignment module With attention module ; Information introduction module Make the input tensor Introducing guidance information of target tensor, weight map module Update the input tensor by calculating the difference between the input information and the target information in the guidance information to build a weight graph , tensor alignment module By migrating in a 'W' shaped network and Inter-constrained intermediate tensor alignment achieves dual-task acceleration, attention module Use the cross attention mechanism to make the tensor Extract more target information features.

[0110] In the input tensor With the guide tensor Enter together Functional modules of the layer After that, the information will be introduced into the module in sequence. Will Introduction ,Weight Graph Module By calculation and The difference between the two is used to update the weight map , tensor alignment module simulation arrive The transformed output tensor is aligned with the constraint , attention module use and Update of Cross-Attention Mechanism ;

[0111] Use constrained tensor alignment to speed up parameter training and Loss function constraint tensor and Approach, training two symmetric 'W' transfer networks simultaneously and ;

[0112] Using the trained network Predict target human body tensor , the trained network Predicted target pose tensor , and finally the tensor and Through the decoder and Get the predicted target human body and posture image and .

[0113] Preferably, the human image dataset comes from the public DeepFashion dataset, and the posture graph dataset uses the posture estimation algorithm OpenPose to extract the posture of the human image dataset, and the two are paired to form a training dataset.

[0114] Specifically, the end of the human body encoder and the posture encoder and the front end of the human body decoder and the posture decoder are provided with a layer Attention layer.

[0115] For input , the intermediate tensor , the final encoder With decoder The predicted output of is expressed as follows:

[0116]

[0117]

[0118] in Indicates Layer Network Layer Different piece, is the attention mechanism layer, is the residual layer, For the The residual layer in the layer network layer, the encoder is , the decoder is , the encoder input is , the decoder input is .

[0119] like Figure 3 As shown in the figure, the architecture of the encoder and decoder comes from VQVAE. In addition to the encoder and decoder, VQVAE also has a codebook mechanism. The encoder and decoder of VQVAE come from a series of convolutional layers, normalization layers, and activation layers. The encoder takes the input image Transformed into a tensor with high channel count and small size ; while the decoder is the input tensor , after a series of residual connections piece, The block consists of a convolutional layer, a normalization layer, an activation layer, and a residual connection to restore the tensor to an image. .

[0120] The codebook mechanism converts tensors Replace with the nearest tensor in a tensor table , and use the codebook loss function to learn the parameters in the tensor table, satisfying the following replacement process:

[0121]

[0122]

[0123]

[0124] in, Indicates the first A replacement tensor, Representing a tensor The last two dimensions are located at The tensor components of Indicates updated , Indicates when When taking the minimum value, The value of , the output is , Represents along the tensor The last two dimensions are concatenated. Represents the codebook mechanism.

[0125] On this basis, an additional layer is added at the end of the encoder and the front end of the decoder Attention layer, for input , the intermediate tensor With output , the final encoder With decoder The predicted output of is expressed as follows:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] in is the activation function, For group normalization along the channel dimension, Indicates Layer Network Layer Different piece, is convolution, is the convolution block, is the nth convolutional block, is the attention mechanism layer, is the residual layer, For the The residual layer in the layer network layer, the encoder is , the decoder is , the input is , the intermediate tensor is And the output is , Represents the codebook mechanism.

[0132] The compression of the image into the tensor space satisfies: , the above general standardization process is similar to The specific form is as follows:

[0133]

[0134]

[0135]

[0136]

[0137] in From Tensor The number of channels, is the activation function, For group normalization along the channel dimension, Indicates Layer Network Layer Different piece, is convolution, is the convolution block, is the residual layer, From Tensor The number of channels, express Mean, is an activation function, express In the channel dimension Quantity, Express Perform standardized operations. For input.

[0138] It is a convolution, which is essentially a matrix multiplication operation of a high-dimensional tensor. The convolution process is to learn the weights of a certain layer of tensors. With offset process, represents the fully connected layer, Convolution kernel Block composition, The attention layer consists of a series of Layers and tensor multiplication operations are composed of three different Layer , and The three weights are normalized and converted into weight tensors, which are multiplied with the query tensor to obtain the final output tensor. , On this basis, and The specific expressions are as follows:

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] The above represents the matrix multiplication operation, For tensors The number of channels, represents an activation function, express In the channel dimension Quantity, middle Indicates that the tensor channel dimension is permuted to the last dimension, express The tensors are all passed through the same , represents convolution, is the fully connected layer, Indicates Layer Network Layer Different piece, Represents three tensors for attention calculation, represents the attention block, represents the residual connection layer, represents the attention calculation function, Represents the updated tensor , and Indicates input, is the output.

[0145] like Figure 4 As shown, preferably, the human body 'W' type migration network and the posture 'W' type migration network are both composed of multiple functional modules The 'W' shaped network structure is formed, and the residuals are connected to the same level Block, for the overall dual-task 'W' type transfer network and Framework, the 'W' type migration network output passes through the codebook mechanism again, and the final decoder output and for:

[0146]

[0147]

[0148]

[0149]

[0150] in Represents the tensor arrive All tensors are concatenated along the channel dimension, is the source human image, is the source pose image, is the target pose image, is the target human image, is the attitude guide tensor, Guide tensors for the human body; is the source attitude tensor, is the source human body tensor.

[0151] and They are image decoder, posture decoder, and They are the human body encoder, posture decoder, and They are image transfer network and posture transfer network respectively. and They represent the image codebook mechanism and the pose codebook mechanism respectively. and They represent the predicted image after migration and the predicted posture after migration, respectively.

[0152] Among them, the number of layers of the 'W' main layer is , the number of sub-layers is , the number of intermediate layers is ,After the 'W' type network consists of 2 main layers, 2 auxiliary layers, and 3 intermediate layers, the outputs of different layers are as follows:

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161] in Represents the tensor arrive All tensors are concatenated along the channel dimension, Indicates that the tensor is Branches, all The output tensor of the layer network. After calculation, the final output of the 'W' type network is , Indicates Layer Network Layer Different Block, superscript m means the Layer Network Layer, Indicates the core functional module in the 'W' type network, with the superscript Indicates network layer, V is the input of the 'W' type network.

[0162] The traditional Unet consists of a pair of up and down sampling, that is, 2 main layers and an intermediate layer connecting the up and down sampling; the improvement of this method is that an additional small 'reverse' Unet is added in the middle, so there are 2 more auxiliary layers and 1 intermediate layer, and the original 1 intermediate layer is also split into two.

[0163] If the number of layers of Unet is further increased, the prediction effect will be better, but the computing performance requirements will be too high and the computing cost will increase. Therefore, this method is replaced by the reverse Unet method with a small increase in video memory. Of course, continuing this method can also form the so-called VvvV network, but at the same time, the required computing performance will also increase significantly. Therefore, the only preferred structure of this method with a small increase in additional overhead is: 2 main layers, 2 secondary layers, and 3 intermediate layers.

[0164] like Figure 5 As shown, specifically, the functional module of the first layer of the 'W' type migration network It includes residual connection module, information import module, weight graph module, tensor constraint alignment module, attention module and channel weight redistribution module;

[0165] Use residual connection modules to import information into the module , weight graph module , tensor constraint alignment module With attention module Connect two by two in series, and then use the channel weight redistribution module In parallel, finally introduce at the input and output ends The block changes the size and number of channels of the input tensor, and the final functional module of the lth layer Input Tensor Functional modules through layer l The layer process can be expressed as:

[0166]

[0167]

[0168] in represents the dot multiplication operation, is the intermediate tensor, Indicates updated , Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, represents the residual connection module, Indicates the information import module. represents the channel weight redistribution module, represents the weight graph module, represents the tensor constraint alignment module, represents the attention module, For functional modules Input tensor, For functional modules Output tensor, Represents the updated tensor .

[0169] Channel weight redistribution module The data processing process is as follows:

[0170]

[0171]

[0172]

[0173] in is average pooling, is an activation function, Represented as being located in the network The tensor of the layer Batch The number of channels is located at The value of the pixel point, omitting some subscripts means all selected tensors. In this method, the input tensor is averaged along the channel dimension. Indicates Layer Network Layer Different piece, represents the fully connected layer, represents splicing along the channel dimension, is an activation function, represents the channel weight redistribution module, and is the input tensor. Represents the number of batches, width, and height respectively.

[0174] like Figure 6 As shown, specifically, the information introduction module ,in:

[0175] Guided Tensor Contains two pieces of information, originally derived from and Splice into Tensor, used as guidance information to guide the input tensor To the target tensor Transformation, for posture transfer tasks in the 'W' type transfer network middle for , for the pose estimation task in the 'W' type migration network middle for .

[0176] Information introduction module entering this layer ,first Split into current boot information With target guidance information , using the target information tensor go through Use after module projection Module Updates , using tensor use Module Updates , the two are reunited to form a new , and update and Splicing together Add after layer The residual connection of , and As shown below:

[0177]

[0178]

[0179]

[0180] Above The module is a gate control update module. Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, represents splicing along the channel dimension, represents the convolutional layer module, represents the residual connection layer module, represents the target tensor, Represents the updated tensor , represents the guide tensor of the input, represents the input tensor to be migrated, Depend on and Spliced ​​along the channel dimension, Indicates the current boot information. Indicates target guidance information. and The input tensor of the information introduction module, and The output tensor of the information introduction module. The upper right subscript l indicates the lth functional module middle.

[0181] The module can introduce some updated information into the original information, which is divided into reset gate With update gate , by updating the gate output and generating the hidden state The dot product finally gives Output is , this method is implemented using the convolution version, as follows:

[0182]

[0183]

[0184]

[0185]

[0186]

[0187] in represents the hyperbolic tangent function, is an activation function, Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, The module is a gate control update module. represents the fully connected layer, represents the intermediate tensor, represents the dot multiplication operation between tensors, , and is the input tensor, is the output tensor.

[0188] like Figure 7 As shown, the specific functional modules Weight Graph Module in Used to perform the following steps:

[0189] Output tensor of the information import module and go through Module, Overall Layer update ,Will middle and After taking the difference, input it into the hyperbolic tangent function Get the tensor weight graph , and After multiplying Add after layer The residual connection of , and As shown below:

[0190]

[0191]

[0192]

[0193] Above Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, represents splicing along the channel dimension, represents the convolutional layer module, Represents the updated tensor , represents the weight map module input guide tensor, Represents the weight graph module input tensor to be migrated, From the source tensor With the target tensor Spliced ​​along the channel dimension, Indicates the current boot information. and Input tensor for weight graph module, and Output tensor for weight graph module, represents the dot multiplication operation between tensors, represents the hyperbolic tangent function, Represents a weight graph module.

[0194] like Figure 8 As shown, specifically, the functional modules Tensor alignment module in The execution methods include:

[0195] Weight Graph Module Output Tensor and go through Module, Overall Layer update , using tensor Approximate with a series of convolutions middle arrive The transformation of Through two learnable Layer and The activation function is close to To simulate , and finally and Doing matrix operations close to symmetric tasks get , and then go through Add after layer The residual connection of Finally, this module outputs additional and , and use the tensor alignment method to control Effect. and It can be as follows:

[0196]

[0197]

[0198]

[0199]

[0200]

[0201] Among them, the above Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, represents splicing along the channel dimension, represents the convolutional layer module, Represents the tensor updated by the tensor alignment module , represents the tensor alignment module input guide tensor, Indicates the tensor to be migrated as input to the tensor alignment module. Depend on and Spliced ​​along the channel dimension, Indicates the current guidance information of the tensor alignment module. Represents the tensor alignment module target guidance information, and is the input tensor of the tensor alignment module, and is the output tensor of the tensor alignment module, represents matrix multiplication operation between tensors, represents an activation function, Represents a functional module The tensor constraint alignment module in , represents a fully connected module, Indicates Layer Network Layer Different fully connected modules.

[0202] like Fig. 9 As shown, the specific functional modules The attention module in The execution methods include:

[0203] Tensor alignment module output tensor and go through Module, Overall Layer update ,Will Respectively and Splicing Feed Layer, updated using the cross attention mechanism And then go through Add after layer The residual connection of , and It can be expressed as follows:

[0204]

[0205]

[0206]

[0207]

[0208] Among them, the above Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, represents splicing along the channel dimension, represents the convolutional layer module, Represents the updated tensor , represents the input guide tensor, Represents the input tensor to be migrated, Depend on and Spliced ​​along the channel dimension, Indicates the current boot information. Indicates target guidance information. and is the input tensor, and is the output tensor, represents the attention mechanism function, represents the cross attention mechanism, represents the attention module, Represents a fully connected module.

[0209] The tensor alignment method described:

[0210] 'W'-shaped transfer network when performing dual-task transfer middle The module outputs a series of and , for the posture transfer network , Symbolizes the tensor that undergoes simulation changes After transformation, it is close to the target tensor , Representative Network Target guidance information here ; Similar to the pose estimation network , Symbolizes the tensor that undergoes simulation changes After transformation, it is close to the target tensor , Representative Network Target guidance information here .

[0211] Thus, for two sets of tasks, the posture transfer network Medium Tensor With the pose estimation network Medium Tensor Same meaning, posture transfer network Medium Tensor With the pose estimation network Medium Tensor The meaning is the same, so fast network training can be achieved by directly constraining tensor alignment in the middle layer of the network.

[0212] Tensor alignment method uses Loss Function and Loss Function of Gradient-Free Methods To constrain the alignment of the intermediate tensors of the two groups of networks, thereby achieving accelerated training and simultaneous dual-task training by leveraging the commonalities between pose transfer and pose estimation tasks.

[0213] Total loss function It can be divided into two parts, one part is used for overall training of the encoder in the VQVAE framework ,Decoder Loss function with codebook structure , and the other part is used to train the loss function of the overall migration network 'W' , the overall:

[0214]

[0215] in represents the total loss function, represents the loss function used for the overall training VQVAE framework, represents the loss function for training the overall transfer network ‘W’.

[0216] For the loss function Divided into a structural loss function that directly constrains image quality and the loss function used to constrain the codebook structure , for the loss function Divided into loss functions that directly constrain the output tensor to align with the target tensor Constrained loss function used with tensor alignment methods for accelerated training , specifically expressed as follows:

[0217]

[0218]

[0219] in , , and is the weight value between each loss function, specifically , , and , represents the loss function used for the overall training VQVAE framework, represents the loss function for training the overall migration network 'W', represents the structural loss function that directly constrains the image quality, represents the loss function that constrains the codebook structure, represents the loss function that directly constrains the output tensor to align with the target tensor, Represents the constrained loss function used by the tensor alignment method for accelerated training.

[0220] Loss functions are widely used between various loss functions and , which is specifically defined as follows:

[0221]

[0222] in, Represented as being located in the network The tensor of the layer Batch The number of channels is located at The value of the pixel point, omitting some subscripts means all selected tensors, and is the input, x is the network prediction value, and y is the true value; and Respectively represent the norm as and The loss function is Represent the number of batches, number of channels, width, and height respectively.

[0223] Specifically Three loss functions are used to improve the quality of generated images, including loss function , and , It is the perceptual loss that makes the output image more like the human eye sees it clearly rather than being close to the target image on average. The VGG network is used to continuously downsample the input tensor in pairs using a series of convolution blocks. Layer Output , by inputting two pictures into the VGG network to get the output of each layer, using The loss function calculates the difference between each layer and uses Weighted, where Represents the VGG network Layer output; , and besides The representation is as follows:

[0224]

[0225]

[0226]

[0227]

[0228]

[0229]

[0230]

[0231]

[0232] Above The mean and variance of each image are calculated along the channel dimension for weighted comparison. Represented as being located in the network The tensor of the layer Batch The number of channels is located at The values ​​at the pixel points of , omitting some subscripts means all selected tensors, x is the network prediction value, y is the true value; finally, different exponential weighted product operations are used for the values ​​calculated for different layers, where , and To prevent the denominator from The parameters are , and , use Calculate the total layer Value, where the exponential weight , , , and They are , , , and , and equal, represents the structural loss function that directly constrains the image quality, represents the dot multiplication operation between tensors, represents matrix multiplication operation between tensors, Respectively represent the number of batches, number of channels, width, height, Indicates the total number of layers, represents the function for calculating the mean, represents the function for calculating the variance, , and represents the intermediate calculation function, Indicates Layer Network Layer Different piece, Indicates Layer weights, represents the perceived loss, represents the multi-layer structure loss, The norm is The loss function is Represents the VGG network Layer output, input is and .

[0233] For the loss function The codebook structure interrupts the gradient generation. To help the codebook structure replace the tensor well, use The loss function calculates the difference before and after. Represents a tensor without gradients , for the tensors before and after encoding The specific expression is as follows:

[0234]

[0235] The above The norm is The loss function is Represents a tensor without gradients , represents the loss function that constrains the codebook structure, is the input intermediate tensor, Represents the codebook mechanism.

[0236] In the loss function In the , since the nearest neighbor encoding tensor will be passed through the codebook again after the 'W' type migration network, there is no need to use Loss function, a smoother overall average can be used Loss function to constrain, in addition to auxiliary style loss , used for local region feature consistency in high-dimensional tensors of newspapers, style loss The local detail features are reflected by calculating the Gram matrix of the tensor, where the Gram matrix is ​​obtained by the inner product between tensors. The loss function The expression is as follows:

[0237]

[0238]

[0239]

[0240] Among them, the above represents the style loss, The norm is The loss function is Represented as being located in the network The tensor of the layer Batch The number of channels is located at The value at the pixel point of , omitting some subscripts means all selected tensors, x is the network prediction value, y is the true value; Represents general multiplication, Respectively represent the number of batches, number of channels, width, height, Indicates Layer Network Layer Different piece, represents matrix multiplication operation between tensors, represents the loss function that directly constrains the output tensor to align with the target tensor, and is the input tensor, Indicates calculation of Gram matrix.

[0241] Finally, the loss function , which is also a constraint tensor. Although the intermediate tensor has no codebook re-encoding, it only needs to provide migration information. To ensure smoothness, only and The loss function constraint can ensure accelerated training, and the expression is as follows: With the pose estimation network Medium Tensor

[0242]

[0243] represents the constraint loss function used by the tensor alignment method for accelerating training, Represented as being located in the network The tensor of the layer Batch The number of channels is located at The value of the pixel point, omitting some subscripts means all selected tensors, Represents general multiplication, Represents a tensor without gradients , The norm is The loss function is Indicates the total number of layers, represents the style loss, and the input is , , and ,in and express The intermediate tensor output by the layer, and express The target guide tensor output by the layer, x is the network prediction value, and y is the true value.

[0244] The human body image and posture image generation method of the present invention utilizes the characteristic that posture migration and posture estimation tasks have partially overlapping features, designs the simultaneous training of the two tasks, and accelerates the training speed of the two tasks by aligning the constraints in the tensor space. Based on the symmetric acceleration of the 'W' type network, the present invention shortens the training time while completing the posture migration and posture estimation tasks at the same time.

[0245] Related meaning explanation:

[0246] ResBlock and Res represent residual blocks; CodeBook represents the tensor recoding process; RA represents the core functional module of the 'W' type network; VQVAE represents the overall encoder-decoder structure; Encoder represents the encoder; Decoder represents the decoder; Wnet represents the 'W' type network structure; RA represents the overall encoder-decoder structure; SE represents the weight distribution block; Proj represents the information introduction module; Wt represents the weight map module; Tran represents the Zhang Liang constraint alignment module; Attn represents the attention module; GRU represents the gate control module; Conv represents the convolution layer normalization layer activation layer combination module; CrossAttn is the cross attention module; Res' is the fully connected residual layer; cat represents the splicing module; They represent tensor addition, subtraction, matrix operation, and dot multiplication operation respectively.

[0247] Embodiment 2:

[0248] This embodiment provides a human body and posture transformation image generation device, including a processor and a storage medium;

[0249] The storage medium is used to store instructions;

[0250] The processor is used to operate according to the instructions to execute the steps of the method according to embodiment 1.

[0251] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0252] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating a human body and posture transformation image, characterized in that: The following steps are involved: Acquire a source input image and source guidance information; the source input image includes a source human image and a source posture image; the source target information includes a target human image and a target posture image; Using the trained human encoder, the source human image and the target human image are compressed into the tensor space to form a source human tensor and a target human tensor; The source pose image and the target pose image are compressed into the tensor space using the trained pose encoder to form the source pose tensor and the target pose tensor; Connect the source human body tensor with the target human body tensor to form posture guidance information; Connect the source posture tensor and the target posture tensor to form human body guidance information; The source human body tensor combined with the human body guidance information is used as input information and input into the trained human body 'W' type migration network to obtain the predicted human body tensor. The source posture tensor combined with the posture guidance information is used as input information and input into the trained posture 'W' type migration network to obtain the predicted posture tensor; The human body 'W' type migration network has the same structure as the posture 'W' type migration network; Input the predicted human body tensor into the trained human body decoder to obtain the predicted human body image; The predicted pose tensor is input into the trained pose decoder to obtain the predicted pose image.

2. The method for generating a human body and posture transformation image according to claim 1, characterized in that: The end of the human encoder and the posture encoder and the front end of the human decoder and the posture decoder are provided with a codebook mechanism; The codebook mechanism takes the input tensor Replaced by the nearest tensor in a learnable randomly initialized tensor table , and use the codebook loss function to learn the parameters in the tensor table. The replacement process is expressed as follows: ； ； ； in, Indicates the first A replacement tensor, Representing a tensor The last two dimensions are located at The tensor components of Indicates updated , Indicates when When taking the minimum value, The value of is the input tensor, is the output tensor, Represents along the tensor The last two dimensions are concatenated. Represents the codebook mechanism.

3. The method for generating a human body and posture transformation image according to claim 2, characterized in that: The ends of the human body encoder and the posture encoder and the front ends of the human body decoder and the posture decoder are provided with a layer Attention layer; The encoder structure of the human body encoder and posture encoder is expressed as: ； The decoder structure of the human body decoder and posture decoder is expressed as: ； in, is the attention mechanism layer, is the residual layer, For the The residual layer in the layer network layer, the encoder is , the decoder is , the encoder input is , the decoder input is .

4. The method for generating a human body and posture transformation image according to claim 3, characterized in that: The human body 'W' type transfer network and the posture 'W' type transfer network are both 'W' type transfer networks formed by a plurality of functional modules forming a 'W' shaped network structure, and the functional modules at the same level are connected through residuals to form a 'W' type transfer network; The output of the human body decoder and posture decoder is expressed as: ； ； ； ； Among them, cat() represents the splicing operation, is the source human image, is the source pose image, is the target pose image, is the target human image, is the human body guide tensor, Guide tensor for attitude; and They represent the predicted image after migration and the predicted posture after migration, is the predicted human image output by the human decoder, The predicted pose image output by the pose decoder; in and They are human body decoder, posture decoder, and They are body encoder, posture encoder, and They are image transfer network and posture transfer network respectively. and They represent the image codebook mechanism and the pose codebook mechanism respectively; Among them, the 'W' type migration network includes 2 main layers, 2 secondary layers, and 3 intermediate layers; and the number of main layers is , the number of sub-layers is , the number of intermediate layers is ,The outputs at different levels are as follows: ； ； ； ； ； ； ； ； in Represents a splicing operation, Indicates that the tensor is Branch, is all The output tensor of the layer network, The final output of the 'W' network is , Indicates the core functional module in the 'W' type network, with the superscript Indicates network layer, V is the input of the 'W' type migration network.

5. The method for generating a human body and posture transformation image according to claim 4, characterized in that: The functional modules include a residual connection module, an information import module, a weight map module, a tensor constraint alignment module, an attention module and a channel weight redistribution module; The information import module, weight graph module, tensor constraint alignment module, and attention module are connected in series using residual links, and then connected in parallel with the weight module. The residual connection module is introduced at the input and output ends of the functional module to change the size and number of channels of the input tensor. The function module is represented as: ； ； in represents the dot multiplication operation, is the intermediate tensor, Indicates that the tensor is Branch, is all The output tensor of the layer network, Indicates Layer Network Layer Different piece, represents the residual connection module, Indicates the information import module. represents the channel weight redistribution module, represents the weight graph module, represents the tensor constraint alignment module, represents the attention module, The functional module of the first layer The input tensor of The functional module of the first layer Output tensor, Represents the updated input tensor .

6. The method for generating a human body and posture transformation image according to claim 5, characterized in that: The weight module satisfies: ； ； ； in, is average pooling, is an activation function, Represented as being located in the network The tensor of the layer Batch The number of channels is located at The value of the pixel point, omitting some subscripts means all selected tensors. In this method, the input tensor is averaged along the channel dimension. represents the fully connected layer, Indicates Layer Network Layer Different fully connected modules; represents splicing along the channel dimension, is an activation function, represents the channel weight redistribution module, and is the input tensor of the functional module of the lth layer; Represents the number of batches, width, and height respectively.

7. The method for generating a human body and posture transformation image according to claim 6, characterized in that: The information introduction module is used to perform the following steps: Will guide the tensor Split into current boot information With target guidance information ; Use the target tensor go through Use after module projection Module Updates , with the source tensor use Module Updates , the two are recombined to form a new guidance tensor , and update the front and back boot information With target guidance information Splicing together Add after layer The residual connection of the information introduction module outputs the current tensor , which is expressed as follows: ； ； ； in, The module is a gate control update module. Indicates that the tensor is Branch, is all The output tensor of the layer network, represents splicing along the channel dimension, represents the convolutional layer module, represents the residual connection layer module, represents the target tensor, Represents the updated tensor , represents the input guide tensor, Indicates the tensor to be migrated as input to the information introduction module. Indicates the current boot information of the information import module. Indicates the information introduced into the module target guidance information, and The input tensor of the information introduction module, and It is the guide tensor and current tensor output by the information introduction module. The superscript l represents the functional module of the lth layer. The weight map module is used to perform the following steps: The guide tensor and current tensor output by the information introduction module and When passing through the weight map module, the information is introduced into the guidance tensor output by the module The whole is updated through the convolutional layer to obtain the guide tensor output by the weight map module , the guide tensor output by the weight map module Decomposed into the target guidance tensor output by the weight map module The current guide tensor output by the weight map module After taking the difference, input it into the hyperbolic tangent function Get the tensor weight graph , the weight graph module inputs the tensor to be migrated With tensor weight graph After the point multiplication, add the convolution layer The residual connection of the weight map module obtains the current tensor output by , as follows: ； ； ； in, Indicates that the tensor is Branch, is all The output tensor of the layer network, represents splicing along the channel dimension, represents the convolutional layer module, Represents the tensor after the weight map module is updated , represents the guidance tensor output by the information introduction module and input by the weight map module, Represents the tensor to be migrated output by the information introduction module and input by the weight graph module, Depend on and Spliced ​​along the channel dimension, Represents the current guidance information of the weight map module, and Input tensor for weight graph module, and The guide tensor and current tensor output by the weight map module, represents the dot multiplication operation between tensors, represents the hyperbolic tangent function, Represents the weight graph module; The execution method of the tensor alignment module includes: The guide tensor output by the weight map module The whole is updated through the convolution layer to obtain the guide tensor output by the tensor alignment module , using the aligned tensor Approximate with a series of convolutions The tensor alignment module currently guides the tensor To the tensor alignment module target guide tensor The transformation aligns the tensor to the current guide tensor of the module Through two learnable fully connected modules and Activation function proximity tensor alignment module target guide tensor To simulate the alignment of tensors , and finally the current tensor output by the weight map module Aligning Tensors The guide tensor output by the tensor alignment module in the task of doing matrix operations close to symmetry Get the hidden tensor , and then after the convolution layer, add the current tensor output by the tensor alignment module The residual connection of the tensor alignment module obtains the current tensor output ; As follows: ； ； ； ； ； Among them, the above Indicates that the tensor is Branch, is all The output tensor of the layer network, represents splicing along the channel dimension, represents the convolutional layer module, represents the kth different convolutional layer module in the lth network layer, Represents the tensor updated by the tensor alignment module , Represents the guide tensor input to the tensor alignment module and is also the guide tensor output by the weight map module. It represents the tensor to be migrated input by the tensor alignment module and the tensor to be migrated output by the weight map module. Indicates the current guidance information of the tensor alignment module. Represents the tensor alignment module target guidance information, and is the input tensor of the tensor alignment module, and The guide tensor and the current tensor are the outputs of the tensor alignment module, represents matrix multiplication operation between tensors, represents an activation function, Represents a functional module The tensor constraint alignment module in , represents a fully connected module, Indicates Layer Network Layer Different fully connected modules; The execution method of the attention module includes: The current tensor output by the tensor alignment module With the guide tensor When passing through the attention module, The whole is updated through the convolutional layer to obtain the guidance tensor output by the attention module ,Will Respectively with the guide tensor The current guide tensor in and the target guide tensor The concatenation is fed into the cross attention mechanism and updated using the cross attention mechanism And then add after the convolution layer The residual connection of the attention module obtains the current tensor output by , which is expressed as follows: ; ； ； ； Among them, the above Indicates that the tensor is Branch, is all The output tensor of the layer network, represents splicing along the channel dimension, represents the convolutional layer module, Represents the tensor after the attention module is updated , Depend on and Spliced ​​along the channel dimension, Indicates the current guidance information of the attention module, represents the target guidance information of the attention module, and is the input tensor of the attention module, and is the output tensor of the attention module, represents the attention mechanism function, represents the cross attention mechanism, represents the attention module, It represents a fully connected module, the superscript l represents the lth functional module, and the subscript n represents the nth module.

8. The method for generating a human body and posture transformation image according to claim 7, characterized in that: The training method of the human body encoder, posture encoder, human body decoder and posture decoder comprises: Acquire a training data set; the training data set includes at least human body images and posture images; The human encoder, the pose encoder, the human decoder and the pose decoder are trained using the training data set and the loss function based on the overall training VQVAE framework to obtain the trained human encoder, pose encoder, human decoder and pose decoder; Methods for obtaining training data sets include: Obtain human image data from the DeepFashion public human dataset; The posture estimation algorithm OpenPose is used to extract the posture of human image data to obtain human posture data; Pair the data of the same human body image and posture image, and take the paired data of two groups of different actions of the same person to form a set of training data, so that one set of training data contains the source human body image, the target human body image, the source posture image and the target posture image, and a training data set is produced.

9. The method for generating a human body and posture transformation image according to claim 8, characterized in that: The training method of the human body 'W' type transfer network and the posture 'W' type transfer network includes: Get the tensor training dataset; Based on the tensor training data set and the loss function for training the overall migration network 'W', the human body 'W' type migration network and the posture 'W' type migration network are trained to obtain the trained human body 'W' type migration network and the posture 'W' type migration network.

10. A human body and posture transformation image generation device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-9.

Images