A method and device for generating human body and posture transformation images

Abstract

The present invention relates to the field of computer vision technology, and discloses a method and device for generating human body and posture transformation images, comprising the following steps: using an encoder to compress a source human body image, a target human body image, a source posture image and a target posture image into a tensor space to form a source human body tensor, a target human body tensor, a source posture tensor and a target posture tensor; the tensors sequentially pass through several functional modules in a 'W' type migration network, each functional module comprising an information introduction module, a weight map module, a tensor alignment module and an attention module; using two trained 'W' networks, respectively predicting a target human body tensor and a target posture tensor, and then respectively obtaining the predicted target human body image and target posture image through a decoder. The present invention will simultaneously complete the human body and posture transformation tasks, ensuring that the generated target image has high-quality texture features.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method and device for generating images of human bodies and posture transformations. Background Art

[0002] As a subfield of machine learning, deep learning uses multi-layer neural networks to automatically learn and extract data features, and has achieved breakthroughs in image recognition, natural language processing, speech recognition, and recommendation systems. Its development has benefited from the advancement of big data, computing power, and optimization algorithms, which can reduce reliance on manual feature engineering and thus improve model performance. Computer vision aims to enable machines to understand the content of images and videos, covering object recognition, image segmentation, posture estimation, and motion analysis. With the help of convolutional neural networks, the accuracy and efficiency of this field have been significantly improved, which has promoted the development of applications such as autonomous driving and medical image analysis, but it also faces challenges such as lighting changes, occlusion, and real-time processing.

[0003] In this context, pose transfer and pose estimation have become important research areas. Pose transfer aims to transfer the motion characteristics of one object to another to achieve natural and smooth animation, while pose estimation focuses on accurately identifying joint positions and poses from images or videos. Through deep learning technology, these applications have shown broad potential, although they still need to overcome challenges in accuracy, real-time performance, and complex scenarios. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a method and device for generating human body and posture transformation images, which can efficiently generate human body and posture transformation images, overcoming the problems of slow human body image or posture generation speed and low single-task efficiency in previous work.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a method for generating a human body and posture transformation image, comprising the following steps:

[0007] Acquire a source input image and source guidance information; the source input image includes a source human image and a source posture image; the source guidance information includes a target human image and a target posture image;

[0008] Using the trained human encoder, the source human image and the target human image are compressed into the tensor space to form a source human tensor and a target human tensor;

[0009] The source pose image and the target pose image are compressed into the tensor space using the trained pose encoder to form the source pose tensor and the target pose tensor;

[0010] Connecting the source human body tensor with the target human body tensor to form posture guidance information; connecting the source posture tensor with the target posture tensor to form human body guidance information;

[0011] The source human body tensor combined with the human body guidance information is used as input information and input into the trained human body 'W' type migration network to obtain the predicted human body tensor.

[0012] The source posture tensor combined with the posture guidance information is used as input information and input into the trained posture 'W' type migration network to obtain the predicted posture tensor;

[0013] The human body 'W' type migration network has the same structure as the posture 'W' type migration network;

[0014] Input the predicted human body tensor into the trained human body decoder to obtain the predicted human body image;

[0015] The predicted pose tensor is input into the trained pose decoder to obtain the predicted pose image.

[0016] Furthermore, the end of the human encoder and the posture encoder and the front end of the human decoder and the posture decoder are provided with a codebook mechanism;

[0017] The codebook mechanism replaces the input tensor z with the tensor z′ that is closest to it in a learnable randomly initialized tensor table, and uses the codebook loss function to learn the parameters in the tensor table.

[0018] Furthermore, an Attn attention layer is provided at the end of the human encoder and the posture encoder and at the front end of the human decoder and the posture decoder;

[0019] The encoder structure of the human body encoder and posture encoder is expressed as:

[0020] E(x)=AttnBlock(ResBlock m (ResBlock m-1 (...ResBlock 1 (x))))

[0021] The decoder structure of the human body decoder and posture decoder is expressed as:

[0022] D(z)=ResBlock m (ResBlock m-1 (...ResBlock 1 (AttnBlock(z))))

[0023] Among them, AttnBlock is the attention mechanism layer, ResBlock is the residual layer, and ResBlock m is the residual layer in the mth network layer, the encoder is E, the decoder is D, the encoder input is x, and the decoder input is z.

[0024] Furthermore, two symmetrical 'W'-shaped migration networks W with the same structure but different inputs and outputs are constructed. i With W g , where the 'W'-shaped migration network consists of a series of core functional modules RA l For the 'W' type migration network W i In this method, As input information and Together they form guidance information Training i predict For a 'W' type migration network W g In this method, As input information and Together they form guidance information Training g predict

[0025] There are several functional modules of different sizes in the 'W' type migration network l , functional module RA l Using the guide information V G Constantly guide the tensor V S To the target tensor V T migrate;

[0026] The functional modules include a residual connection module, an information import module, a weight map module, a tensor constraint alignment module, an attention module and a channel weight redistribution module.

[0027] The information import module, weight graph module, tensor constraint alignment module, and attention module are connected in series in pairs using residual links, and then connected in parallel with the channel weight redistribution module. A residual connection module is introduced at the input and output ends of the functional module to change the size and number of channels of the input tensor.

[0028] Furthermore, the information introduction module is used to perform the following steps:

[0029] Will guide the tensor Split into current boot information With target guidance information

[0030] With the target tensor V TVia ResBlock l After module projection, use GRU module update Using source tensor Update using GRU module The two are recombined to form a new guide tensor At the same time, the front and rear boot information will be updated With target guidance information Splice together through ConvBlock l Add after layer The residual connection of the information introduction module outputs the current tensor It is expressed as follows:

[0031]

[0032] Among them, GRU l The module is a gate control update module. Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. cat indicates concatenation along the channel dimension. ConvBlock indicates the convolutional layer module. ResBlock indicates the residual connection layer module. V T represents the target tensor, V′ represents the updated tensor V, represents the input guide tensor, Indicates the tensor to be migrated as input to the information introduction module. Indicates the current boot information of the information import module. Indicates the information introduced into the module target guidance information, and The input tensor of the information introduction module, and It is the guide tensor and current tensor output by the information introduction module. The superscript l represents the functional module of the lth layer.

[0033] Furthermore, the weight map module is used to perform the following steps:

[0034] The guide tensor and current tensor output by the information introduction module and When passing through the weight graph module, the information is introduced into the guidance tensor output by the module The whole is updated through the convolutional layer to obtain the guide tensor output by the weight map module The guide tensor output by the weight map module Decomposed into the target guidance tensor output by the weight map module The current guide tensor output by the weight map module After the difference is made, it is fed into the hyperbolic tangent function tanh to obtain the tensor weight graph The weight graph module inputs the tensor to be migrated With tensor weight graph After the point multiplication, add the convolution layer The residual connection of the weight graph module obtains the current tensor output by As shown below:

[0035]

[0036] in, Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. cat indicates concatenation along the channel dimension. ConvBlock indicates the convolutional layer module. V″ indicates the tensor V′ after the weight map module is updated. Represents the weight graph module input guide tensor, Represents the weight graph module input tensor to be migrated, Depend on and It is spliced ​​along the channel dimension. Indicates the current guidance information of the weight graph module, and Input tensor for weight graph module, and is the guide tensor and the current tensor output by the weight map module, ⊙ represents the point multiplication operation between tensors, tanh represents the hyperbolic tangent function, and Wt represents the weight map module. Further, the execution method of the tensor alignment module includes:

[0037] The guide tensor output by the weight map module The whole is updated through the convolution layer to obtain the guide tensor output by the tensor alignment module Align the tensor M l Approximate with a series of convolutions The tensor alignment module currently guides the tensor To the tensor alignment module target guide tensor The transformation aligns the tensor to the current guide tensor of the module Tensor alignment module target guided tensor is approached through two learnable fully connected modules and LeakyReLU activation function To simulate the alignment tensor M l , and finally the current tensor output by the weight graph module Aligned with tensor M l The guide tensor output by the tensor alignment module in the task of doing matrix operations close to symmetry Get the hidden tensor After the convolution layer, add the current tensor output by the tensor alignment module The residual connection of the tensor alignment module obtains the current tensor output And use tensor alignment method to control M lEffect. The current tensor output by the tensor alignment module With the guide tensor When passing through the attention module, The whole is updated through the convolutional layer to obtain the guidance tensor output by the attention module Will Respectively with the guide tensor The current guide tensor in and the target guide tensor The concatenation is fed into the cross attention mechanism and updated using the cross attention mechanism And then add after the convolution layer The residual connection of the attention module obtains the current tensor output by

[0038] Furthermore, the training methods of the human encoder, posture encoder, human decoder and posture decoder include:

[0039] Acquire a training data set; the training data set includes at least human body images and posture images;

[0040] The human encoder, posture encoder, human decoder and posture decoder are trained using the training data set and the loss function to obtain the trained human encoder, posture encoder, human decoder and posture decoder.

[0041] Furthermore, the method of obtaining the training data set includes:

[0042] Human image data is obtained from the DeepFashion public human dataset.

[0043] The posture estimation algorithm OpenPose is used to extract the posture of human image data to obtain human posture data.

[0044] Pair the same human body image with the posture image data, and take the paired data of two sets of different actions of the same person to form a set of training data, so that one set of training data contains the source human body image, the target human body image, the source posture image and the target posture image, and thus the training data set is obtained.

[0045] Furthermore, the training dataset and the loss function are used to train the human encoder, the pose encoder, the human decoder and the pose decoder, including:

[0046] Based on the loss function used to train the overall VQVAE framework, the human encoder and human image decoder are trained using images of the same human body in the training dataset;

[0047] Based on the loss function used to train the overall VQVAE framework, the pose human encoder and pose image decoder are trained using images of the same pose in the training dataset.

[0048] Furthermore, the training method of the human body 'W' type transfer network and the posture 'W' type transfer network includes:

[0049] Get the tensor training dataset;

[0050] Based on the tensor training data set and the loss function for training the overall migration network 'W', the human body 'W' type migration network and the posture 'W' type migration network are trained by using the tensor alignment method to obtain the trained human body 'W' type migration network and the posture 'W' type migration network.

[0051] Furthermore, the tensor training data set is obtained, including:

[0052] Use the trained human image encoder to map the human image into the tensor space and turn it into a human tensor, and pass it through the human tensor codebook. Use the trained posture image encoder to map the posture image into the tensor space and turn it into a posture tensor, and pass it through the posture tensor codebook.

[0053] The human body tensor and the posture tensor are paired data, and two sets of paired data of different actions of the same person are taken to form a set of training data, so that one set of training data contains the source human body tensor, the target human body tensor, the source posture tensor and the target posture tensor, thereby obtaining a tensor training data set.

[0054] In a second aspect, the present invention provides a device for generating images of human body and posture transformation, including a processor and a storage medium;

[0055] The storage medium is used to store instructions;

[0056] The processor is used to operate according to the instructions to execute the steps of the method according to the first aspect.

[0057] Compared with the prior art, the present invention provides a method for generating human body and posture transformation images, which has the following beneficial effects:

[0058] (1) The present invention uses a trained encoder, decoder and two symmetrical 'W'-shaped networks to directly convert the original image and the target image, and can efficiently generate transformed images of human bodies and postures, overcoming the problems of slow human body image or posture generation speed and low single-task efficiency in previous work;

[0059] (2) The present invention uses a 'W' type network and the functional modules therein, and adopts a residual connection module, an information import module, a weight map module, a tensor constraint alignment module, an attention module and a channel weight redistribution module to process the tensor, thereby realizing the conversion of the image tensor, realizing the conversion of the human body and posture and improving the details of the converted image.

[0060] (3) The human body image and posture image generation method of the present invention can be mainly applied in four fields:

[0061] 1) In the field of film production, this method can replace real people to complete difficult and dangerous actions; 2) In the field of recognition, this method can be used to identify specific dangerous actions and issue alarms to reduce the difficulty of recognition; 3) In the field of virtual reality, this method can be used to guide and design the actions of virtual characters; 3) In the field of games, this method can be used to capture motion using character skeletons.

[0062] (4) The human body image and posture image generation method of the present invention utilizes the characteristic that posture migration and posture estimation tasks have partially overlapping features, designs the simultaneous training of the two tasks, and accelerates the training speed of the two tasks by aligning the constraints in the tensor space. Based on the symmetric acceleration of the 'W' type network, the present invention shortens the training time while completing the posture migration and posture estimation tasks at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the method for generating human body and posture transformation images based on symmetric acceleration of 'W' type network of the present invention;

[0064] Figure 2 It is the overall structure diagram of the method for generating human body and posture transformation images based on symmetric acceleration of 'W' type network of the present invention;

[0065] Figure 3 The structure diagram of the encoder and decoder in the method for generating human body and posture transformation images based on symmetric acceleration of 'W' type network of the present invention;

[0066] Figure 4 The overall structure diagram of the 'W' type network in the method for generating human body and posture transformation images based on symmetric acceleration of the 'W' type network of the present invention;

[0067] Figure 5 It is a block structure diagram of the core functional module RA in the 'W' type network of the human body and posture transformation image generation method based on the symmetric acceleration of the 'W' type network of the present invention;

[0068] Figure 6This is a block structure diagram of the information introduction module Proj in the core function module RA block of the 'W' type network of the human body and posture transformation image generation method based on symmetric acceleration of the 'W' type network of the present invention;

[0069] Figure 7 It is a block structure diagram of the weight map module Wt in the core function module RA block of the 'W' type network of the method for generating human body and posture transformation images based on symmetric acceleration of the 'W' type network of the present invention;

[0070] Figure 8 This is a block structure diagram of the tensor constraint alignment module Tran in the core function module RA block of the 'W' type network of the human body and posture transformation image generation method based on symmetric acceleration of the 'W' type network of the present invention;

[0071] Fig. 9 This is a block structure diagram of the attention module Attn in the core functional module RA block of the 'W' type network of the human body and posture transformation image generation method based on symmetric acceleration of the 'W' type network of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1:

[0074] This embodiment proposes a method for generating human body and posture transformation images based on symmetric acceleration of a 'W' type network. The flowchart and structure diagram of the overall framework are shown in FIG. Figure 1 and Figure 2 As shown, the specific steps include:

[0075] Step 1: Make a dataset.

[0076] Step 1.1: Obtain human image data from the DeepFashion public human dataset.

[0077] Step 1.2: Use the posture estimation algorithm OpenPose to extract the posture of the human image data and obtain the human posture data.

[0078] Step 1.3: Pair the same human image and posture image data, take two sets of paired data of different actions of the same person to form a set of training data, so that one set of training data contains the source human image, the target human image, the source posture image and the target posture image, and thus make the entire training data set.

[0079] Step 2: Train the encoder, decoder and codebook

[0080] Step 2.1: Use the human images in the training dataset to train the human encoder, human image decoder, and human tensor codebook, and use the pose images in the training dataset to train the pose human encoder, pose image decoder, and pose tensor codebook.

[0081] Step 2.2: Use loss functions such as perceptual loss and structural loss to constrain the model during training.

[0082] Step 2.3: Constrain the codebook during training using methods such as gradient-free optimization.

[0083] Step 2.3: Use the trained human image encoder to map the human image into the tensor space and transform it into a human tensor, and pass it through the human tensor codebook. Use the trained posture image encoder to map the posture image into the tensor space and transform it into a posture tensor, and pass it through the posture tensor codebook.

[0084] Step 2.4: Pair the human body tensor and the posture tensor in the same way as in step 1.3 to form a training data set in tensor form.

[0085] Step 3: Train the 'W'-type transfer network.

[0086] Step 3.1: Use the source human tensor, source posture tensor and target posture tensor in the tensor training dataset to train the human 'W' type transfer network, and use the source posture tensor, source human tensor and target human tensor in the tensor training dataset to train the posture 'W' type transfer network.

[0087] Step 3.2: During training, the tensor will pass through several core functional modules of different sizes to assist migration.

[0088] Step 3.3: During training, the tensor in the core functional module will pass through the information introduction module for introducing guidance information, the weight graph module for forming weight updates according to the difference between the input and the target, the tensor constraint alignment module for simultaneous dual-task training and accelerated training, and the attention module for obtaining richer target features.

[0089] Step 3.4: Use a loss function such as style loss to constrain the model during training.

[0090] Step 3.5: Use the tensor constraint alignment loss function during training to simultaneously train two 'W'-type migration network models and accelerate the training of the two 'W'-type migration network models.

[0091] Step 4: Use the trained human body 'W' type transfer network to input the source human body tensor, the source posture tensor and the target posture tensor to predict the predicted target human body tensor. After passing the human body tensor codebook specification, the final output predicted target human body image is obtained through the human body image decoder. Use the trained posture 'W' type transfer network to input the source posture tensor, the source human body tensor and the target human body tensor to predict the predicted target posture tensor. After passing the posture tensor codebook specification, the final output predicted target posture image is obtained through the posture image decoder.

[0092] Step 4.1: Obtain source input image and source guidance information; the source input image includes a source human body image With the source pose image The source guidance information includes a target human image and the target pose image

[0093] Step 4.2: Using the trained human encoder E i Source human image With the target human image Compressed into tensor space to form the source human body tensor With the target human body tensor

[0094] Using the trained pose encoder E g The source pose image With the target pose image Compress to tensor space to form the source posture tensor and the target pose tensor

[0095] Step 4.3: Convert the source body tensor With the target human body tensor Together they form posture guidance information The source pose tensor and the target pose tensor Connection, forming human body guidance information

[0096] Step 4.4: Convert the source human body tensor Combined with human guidance information As input information, input the trained human 'W' type transfer network W i , get the predicted human body tensor

[0097] The source pose tensor Combined with posture guidance information As input information, input the trained posture 'W' type transfer network W g, get the predicted posture tensor

[0098] The human 'W' type migration network W i With posture 'W' type migration network W g The structure is exactly the same;

[0099] Step 4.5: Input the predicted human tensor into the trained human decoder D i , get the predicted human image

[0100] Feed the predicted pose tensor into the trained pose decoder D g , get the predicted posture image

[0101] Specifically, the method includes:

[0102] Create a human body and posture image training dataset, pair human body images P i Pose image P g Then two different groups of the same person form a training data set, namely the source human image With the target human image and source pose image With the target pose image Together they constitute a training data set. By using the perceptual loss function, L1 loss function, and MS-SSIM loss function, the P i With P g Train the human encoder E separately i , Human image decoder D i 、Human posture encoder E g and human pose decoder D g ; Among them, encoder E i With E g It consists of a series of continuously downsampling convolutional layers, and the input is the human image P i With the pose image P g , the outputs are the human body tensor V i With the attitude tensor V g ; Decoder D i With D g It consists of a series of continuously upsampling convolutional layers, and the input is the human body tensor V i With the attitude tensor V g , the outputs are the target human images With the target pose image

[0103] Using human encoder E i Source human image With the target human image Compressed into tensor space to form the source human body tensor With the target human body tensor

[0104] Using the human pose encoder E g The source pose image With the target pose image Compress to tensor space to form the source posture tensor and the target pose tensor

[0105] Construct two symmetric 'W'-shaped migration networks W with the same structure but different inputs and outputs i With W g , where the 'W'-shaped migration network consists of a series of core functional modules RA l For the 'W' type migration network W i In this method, As input information and Together they form guidance information Training i predict For a 'W' type migration network W g In this method, As input information and Together they form guidance information Training g predict

[0106] There are several functional modules of different sizes in the 'W' type migration network l , functional module RA l Using the guide information V G Constantly guide the tensor V S To the target tensor V T Migration; Functional module RA l Contains information import module Proj l , weight graph module Wt l , tensor alignment module Tran l With the attention module Attn l ; The information import module Proj l Let the input tensor V S Introduce the guidance information of the target tensor, the weight graph module Wt l By calculating the difference between the input information and the target information in the guidance information, the weight graph is constructed to update the input tensor V S , tensor alignment module Tran l By using the 'W' type migration network W iWith W g The intermediate tensor alignment between constraints realizes dual-task acceleration, and the attention module Attn l Use the cross attention mechanism to make the tensor V S Extract more target information features.

[0107] In the input tensor V S With the guide tensor V G Enter the functional module RA of the first layer together l After that, the information will be introduced into the module Proj l V T Introducing V S , weight graph module Wt l By calculating V S With V T The difference between them is used to get the weight map to update V S , tensor alignment module Tran l Analog V S To V T The transformed output tensor is aligned with the constraint H, the attention module Attn l Use V T With V S Cross-Attention Mechanism Update V S ;

[0108] Use the constrained tensor alignment method to accelerate parameter training and constrain the tensor W(V S ) and V T Approximately, train two symmetric 'W'-shaped migration networks W at the same time i With W g ;

[0109] Using the trained network W i Predict target human body tensor The trained network W g Predicted target pose tensor Finally, the tensor and Through the decoder D i With D g Get the predicted target human body and posture image and

[0110] Preferably, the human image dataset comes from the public DeepFashion dataset, and the posture graph dataset uses the posture estimation algorithm OpenPose to extract the posture of the human image dataset, and the two are paired to form a training dataset.

[0111] Specifically, an Attn attention layer is provided at the end of the human body encoder and the posture encoder and at the front end of the human body decoder and the posture decoder.

[0112] For input x, intermediate tensor z, the predicted output of the final encoder E and decoder D is expressed by the following formula:

[0113] E(x)=AttnBlock(ResBlock m (ResBlock m-1 (...ResBlock 1 (x))))

[0114] D(z)=ResBlock m (ResBlock m-1 (...ResBlock 1 (AttnBlock(z))))

[0115] in Represents the nth different Block in the mth network layer, AttnBlock is the attention mechanism layer, ResBlock is the residual layer, and ResBlock m is the residual layer in the mth network layer, the encoder is E, the decoder is D, the encoder input is x, and the decoder input is z.

[0116] like Figure 3 As shown in the figure, the architecture of the encoder and decoder comes from VQVAE. In addition to the encoder and decoder, VQVAE also has a codebook mechanism. The encoder and decoder of VQVAE come from a series of convolutional layers, normalization layers, and activation layers. The encoder converts the input image x into a high-channel, small-size tensor z; while the decoder takes the input tensor z and restores it to image y through a series of residual connection ResBlock blocks. The ResBlock block consists of convolutional layers, normalization layers, activation layers, and residual connections.

[0117] The codebook mechanism replaces the tensor z with the tensor z′ that is closest to it in a tensor table, and uses the codebook loss function to learn the parameters in the tensor table to satisfy the following replacement process:

[0118] z′=vq(z)

[0119]

[0120] Z h,w ′=e k , k = argmin i ||z h,w -e i ||

[0121] Among them, e k represents the kth replacement tensor in the codebook, z h,w Represents the tensor component whose last two dimensions of tensor z are located at h and w, z h，w ′ represents the updated z h,w , argmin i ||z h,w -e i || means when ||z h,w -e i || When the minimum value is taken, the value of i, the input is z, the output is z′, cat h,w It indicates concatenation along the last two dimensions of tensor z, and vq indicates the codebook mechanism.

[0122] On this basis, an additional Attn attention layer is added at the end of the encoder and the front end of the decoder. For the input x, the intermediate tensor z and the output y, the final predicted output of the encoder E and the decoder D is expressed by the following formula:

[0123] ConvBlock(x)=CNN(SiLU(GN(x)))

[0124] ResBlock(x)=ConvBlock2(ConvBlock1(x))+x

[0125] z=E(x)=AttnBlock(ResBlock m (ResBlock m-1 (...ResBlock 1 (x))))

[0126] D(z)=ResBlock m (ResBlock m-1 (...ResBlock 1 (AttnBlock(z))))

[0127] y=D(vq(E(x)))

[0128] Where SiLU is the activation function, GN is the group normalization along the channel dimension, Indicates the nth different Block in the mth network layer, CNN is convolution, ConvBlock is convolution block, ConvBlock n is the nth convolutional block, AttnBlock is the attention mechanism layer, ResBlock is the residual layer, and ResBlock mis the residual layer in the mth network layer, the encoder is E, the decoder is D, the input is x, the intermediate tensor is z and the output is y, and vq represents the codebook mechanism.

[0129] The compression of the image into the tensor space satisfies: V = vq(E(P)). The above general standardization process and SiLU are in the following specific forms:

[0130] SiLU(x)=x·Sigmoid(x)

[0131]

[0132] Where n comes from the number of channels of tensor x, SiLU is the activation function, and GN is the group normalization along the channel dimension. Represents the nth different Block in the mth network layer, CNN is convolution, ConvBlock is convolution block, ResBlock is residual layer, n comes from the number of channels of tensor x, u(x) represents the mean of x, Sigmoid is an activation function, x i represents the i-th component of x in the channel dimension, N(x) represents the normalization operation on x, and x is the input.

[0133] CNN is a convolution, which is essentially a matrix multiplication operation of high-dimensional tensors. The convolution process is the process of learning the tensor weight w and offset b of a certain layer. FN represents the fully connected layer, which is composed of CNN blocks with 1×1 convolution kernels. The Attn attention layer is composed of a series of FN layers and tensor multiplication operations. V is obtained through three different FN layers. Q , V K With V V The three weights are normalized and converted into weight tensors, which are multiplied with the query tensor to obtain the final output tensor V′. Based on this, the AttnBlock block is specifically expressed as follows:

[0134]

[0135] V Q =FN1(V),V K =FN2(V),V V =FN3(V)

[0136]

[0137] AttnBlock(V)=ResBlock2(FN4(attention(ResBlock1(V Q , V K , V V ))))

[0138] The above Represents matrix multiplication operation, d K is the tensor V K The number of channels, Softmax represents an activation function, x i represents the i-th component of x in the channel dimension, V T T in the figure indicates that the tensor channel dimension is permuted to the last dimension. ResBlock(V Q , V K , V V ) indicates V Q , V K , V V The tensors all go through the same ResBlock, CNN represents convolution, and FN represents the fully connected layer. Represents the nth different Block in the mth network layer, V Q , V K , V V Represents three tensors for attention calculation, AttnBlock represents the attention block, ResBlock represents the residual connection layer, attention represents the attention calculation function, V′ represents the updated tensor V, V and x represent the input, and V′ is the output.

[0139] like Figure 4 As shown, preferably, the human body 'W' type migration network and the posture 'W' type migration network are both 'W' shaped network structures composed of multiple functional modules RA, and the RAs at the same level are connected by residuals. n Block, for the overall dual-task 'W' type transfer network W i With W g Framework, the 'W' type migration network output passes through the codebook mechanism again, and the final decoder output and for:

[0140]

[0141] in Indicates that the tensor V1 to V n All tensors are concatenated along the channel dimension, is the source human image, is the source pose image, is the target pose image, is the target human image, is the attitude guide tensor, Guide tensors for the human body; is the source attitude tensor, is the source human body tensor.

[0142] Di With D g They are image decoder, posture decoder, E i With E g They are the human encoder, posture decoder, and W i With W g They are image transfer network and posture transfer network, vq i With vq g They represent the image codebook mechanism and the pose codebook mechanism respectively. and They represent the predicted image after migration and the predicted posture after migration, respectively.

[0143] Among them, the number of layers of the 'W' main layer is n p , the number of sub-layers is n q , the number of intermediate layers is n m , after the 'W' type network consists of 2 main layers, 2 auxiliary layers, and 3 intermediate layers, the outputs of different layers are as follows:

[0144]

[0145]

[0146]

[0147] in Indicates that the tensor V1 to V n All tensors are concatenated along the channel dimension, It means that the tensor is the output tensor of the nth layer network in the mth branch. After calculation, the final output of the 'W' type network is It represents the nth different Block in the mth network layer. The superscript m represents the mth network layer. RA represents the core functional module in the 'W' type network. The superscript n represents the nth layer of the network. V is the input of the 'W' type network.

[0148] The traditional Unet consists of a pair of up and down sampling, that is, 2 main layers and an intermediate layer connecting the up and down sampling; the improvement of this method is that an additional small 'reverse' Unet is added in the middle, so there are 2 more auxiliary layers and 1 intermediate layer, and the original 1 intermediate layer is also split into two.

[0149] If the number of layers of Unet is further increased, the prediction effect will be better, but the computing performance requirements will be too high and the computing cost will increase. Therefore, this method is replaced by the reverse Unet method with a small increase in video memory. Of course, continuing this method can also form the so-called VvvV network, but it will also greatly increase the required computing performance. Therefore, the only preferred structure of this method with a small increase in additional overhead is: 2 main layers, 2 secondary layers, and 3 intermediate layers.

[0150] like Figure 5 As shown, specifically, the functional module RA of the first layer of the 'W' type migration network l It includes residual connection module, information import module, weight graph module, tensor constraint alignment module, attention module and channel weight redistribution module;

[0151] Use the residual connection module to import information into the module Proj l , weight graph module Wt l , tensor constraint alignment module Tran l With the attention module Attn l Two by two are connected in series, and then the channel weight redistribution module SE is used l Connect in parallel, and finally introduce ResBlock at the input and output ends l The block changes the size and number of channels of the input tensor, and finally the functional module RA of the lth layer l Input tensor V l Through the functional module RA of layer l l The layer process can be expressed as:

[0152]

[0153] Where ⊙ represents the dot multiplication operation, V temp is the intermediate tensor, V l′ Represents the updated V l , Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, ResBlock represents the residual connection module, Proj represents the information introduction module, SE represents the channel weight redistribution module, Wt represents the weight map module, Tran represents the tensor constraint alignment module, Attn represents the attention module, V l For the functional module RA l Input tensor, V l′ For the functional module RA l The output tensor, V′, represents the updated tensor V.

[0154] The data processing process of the channel weight redistribution module SE is as follows:

[0155]

[0156]

[0157] Where AvgPool l is average pooling, LeakyReLU is an activation function, It is represented as the value of the pixel at (w, h) in the cth channel in the bth batch of the tensor located at the lth layer of the network. If some subscripts are omitted, it means all selected tensors. In this method, the input tensor is averaged along the channel dimension. represents the nth different Block in the mth network layer, FN represents the fully connected layer, cat represents concatenation along the channel dimension, Sigmoid is an activation function, SE represents the channel weight redistribution module, V l and x is the input tensor. B, W, H represent the batch number, width, and height respectively. Figure 6 As shown, specifically, the information introduction module Proj l ,in:

[0158] Guided Tensor Contains two parts of information, originally derived from V S With V T Splice into Tensor, used as guidance information to guide the input tensor V S To the target tensor V T Transformation, for the posture transfer task in the 'W' type transfer network W i middle for For the pose estimation task, in the 'W' type migration network W g middle for

[0159] Information entering this layer is introduced into the module Proj l ,first Split into current boot information With target guidance information Use the target information tensor V T Via ResBlock l After module projection, use GRU module update Using Tensor Update using GRU module The two merged to form a new At the same time, the update and Splice together through ConvBlock l Add after layer The residual connection of and As shown below:

[0160]

[0161]

[0162] The above GRU l The module is a gate control update module. Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, cat represents concatenation along the channel dimension, ConvBlock represents the convolutional layer module, ResBlock represents the residual connection layer module, V T represents the target tensor, V′ represents the updated tensor V, represents the guide tensor of the input, represents the input tensor to be migrated, Depend on and It is spliced ​​along the channel dimension. Indicates the current boot information. Indicates target guidance information. and The input tensor of the information introduction module, and The output tensor of the information introduction module. The upper right subscript l indicates the lth functional module RA l middle.

[0163] GRU l The module can introduce some updated information into the original information, which is divided into reset gate With update gate By updating the gate output and generating the hidden state h t Point multiplication finally gets GRU l The output is h1. This method is implemented using the convolution version as follows:

[0164]

[0165] Among them, tanh represents the hyperbolic tangent function, Sigmoid is an activation function, Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, the GRU module is a gate control update module, FN represents the fully connected layer, and h t represents the intermediate tensor, ⊙ represents the dot multiplication operation between tensors, x l, x and is the input tensor, is the output tensor.

[0166] like Figure 7 As shown, specifically, the functional module RA l The weight graph module Wt in l Used to perform the following steps:

[0167] Output tensor of the information import module and By Wt l Module, Overall Layer update Will middle and After the difference is made, it is fed into the hyperbolic tangent function tanh to obtain the tensor weight graph and After multiplying Add after layer The residual connection of and As shown below:

[0168]

[0169] Above Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, cat represents concatenation along the channel dimension, ConvBlock represents the convolutional layer module, V″ represents the updated tensor V′, Represents the weight graph module input guide tensor, Represents the weight graph module input tensor to be migrated, From the source tensor With the target tensor It is spliced ​​along the channel dimension. Indicates the current boot information. and Input tensor for weight graph module, and Output tensor for the weight graph module, ⊙ represents the dot multiplication operation between tensors, tanh represents the hyperbolic tangent function, and Wt represents the weight graph module.

[0170] like Figure 8 As shown, specifically, the functional module RA l Tensor alignment module Tran l The execution methods include:

[0171] Weight graph module output tensor and By Tran l Module, Overall Layer update With tensor M l Approximate with a series of convolutions middle arrive The transformation of Through two learnable FNs l Layer is close to LeakyReLU activation function To simulate M l , and finally With M l Doing matrix operations close to symmetric tasks get Then go through Add after layer The residual connection of Finally, this module outputs additional and And use tensor alignment method to control M l Effect. and It can be as follows:

[0172]

[0173] Among them, the above Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, cat represents concatenation along the channel dimension, ConvBlock represents the convolutional layer module, V″′ represents the tensor V″ updated by the tensor alignment module, represents the tensor alignment module input guide tensor, Indicates the tensor to be migrated as input to the tensor alignment module. Depend on and Spliced ​​along the channel dimension, Indicates the current guidance information of the tensor alignment module. Represents the tensor alignment module target guidance information, and is the input tensor of the tensor alignment module, and is the output tensor of the tensor alignment module, Represents matrix multiplication between tensors, LeakyReLU represents an activation function, Tran l Indicates the function module RAl The tensor constraint alignment module in , FN represents the fully connected module, represents the kth different fully connected module in the lth network layer.

[0174] like Fig. 9 As shown, specifically, the functional module RA l Attention module Attn l The execution methods include:

[0175] Tensor alignment module output tensor and By Attn l Module, Overall Layer update Will Respectively and Splice into CrossAttn l Layer, updated using the cross attention mechanism And then go through Add after layer The residual connection of and It can be expressed as follows:

[0176] CrossAttn l (x l ,y l )=FN l (attention(x l ,y l ,y l ))

[0177]

[0178] Among them, the above Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, cat represents concatenation along the channel dimension, ConvBlock represents the convolutional layer module, V″″ represents the updated tensor V″′, represents the input guide tensor, Represents the input tensor to be migrated, Depend on and Spliced ​​along the channel dimension, Indicates the current boot information. Indicates target guidance information. and is the input tensor, and is the output tensor, attention represents the attention mechanism function, CrossAttn represents the cross attention mechanism, Attn represents the attention module, and FN represents the fully connected module.

[0179] The tensor alignment method described:

[0180] When performing dual-task transfer, the Tran l The module outputs a series of and For the posture transfer network W i , Symbolizes the simulated change tensor M l After transformation, it is close to the target tensor Represents network W i Target guidance information here Similarly, for the pose estimation network W g , Symbolizes the simulated change tensor M l After transformation, it is close to the target tensor Represents network W g Target guidance information here

[0181] Thus, for the two groups of tasks, the posture transfer network W i Medium Tensor With the pose estimation network W g Medium Tensor The same meaning, posture transfer network W i Medium Tensor With the pose estimation network W g Medium Tensor The meaning is the same, so fast network training can be achieved by directly constraining tensor alignment in the middle layer of the network.

[0182] Tensor alignment method uses Loss Function and Loss Function of Gradient-Free Methods To constrain the alignment of the intermediate tensors of the two groups of networks, thereby achieving accelerated training and simultaneous dual-task training by leveraging the commonalities between pose transfer and pose estimation tasks.

[0183] Total loss function It can be divided into two parts. One part is used to train the loss function of the encoder E, decoder D and codebook structure in the VQVAE framework as a whole. The other part is used to train the loss function of the overall migration network 'W' The overall:

[0184]

[0185] in represents the total loss function, represents the loss function used for the overall training VQVAE framework, represents the loss function for training the overall transfer network ‘W’.

[0186] For the loss function Divided into a structural loss function that directly constrains image quality and the loss function used to constrain the codebook structure For the loss function Divided into loss functions that directly constrain the output tensor to align with the target tensor Constrained loss function used with tensor alignment methods for accelerated training The specific expressions are as follows:

[0187]

[0188] Among them, λ1, λ2, λ3 and λ4 are the weight values ​​between the loss functions, specifically 1.2, 0.7, 1.0 and 1.6, represents the loss function used for the overall training VQVAE framework, represents the loss function for training the overall migration network 'W', represents the structural loss function that directly constrains the image quality, represents the loss function that constrains the codebook structure, represents the loss function that directly constrains the output tensor to align with the target tensor, Represents the constrained loss function used by the tensor alignment method for accelerated training.

[0189] Loss functions are widely used between various loss functions and The specific definitions are as follows:

[0190]

[0191] in, It is represented as the value of the pixel at (w, h) in the cth channel in the bth batch of the tensor at the lth layer of the network. If some subscripts are omitted, it means all the selected tensors, x b，c，h,w With y b，c，h,w is the input, x is the network prediction value, and y is the true value; and They represent the loss functions with norms of 1 and 2 respectively, and B, C, W, and H represent the number of batches, the number of channels, the width, and the height respectively.

[0192] Specifically Three loss functions are used to improve the quality of generated images, including loss function and It is the perceptual loss that makes the output image more like the human eye sees it clearly rather than being close to the target image on average. A series of convolution blocks in the VGG network are used to continuously downsample the input tensor to obtain the l-layer output. By inputting two pictures into the VGG network to get the output of each layer, The loss function calculates the difference between each layer and uses w l Weighted, where VGG l (x l ) represents the output of the lth layer of the VGG network; and besides The representation is as follows:

[0193]

[0194] Above The mean and variance of each image are calculated along the channel dimension for weighted comparison. It is represented as the value of the pixel at (w, h) on the cth channel in the bth batch of the tensor located in the lth layer of the network. If some subscripts are omitted, all selected tensors are represented. x is the network prediction value and y is the true value. Finally, different exponential weighted product operations are used for the values ​​calculated for different layers. ε1, ε2 and ε3 are parameters to prevent the denominator from being 0, which are 0.01, 0.03 and 0.015 respectively. m uses 5 to calculate a total of 5 layers of ssim values. The exponential weights β1, β2, β3, β4 and β5 are 0.0448, 0.2856, 0.3001, 0.2463 and 0.13333 respectively. α0 and β5 are equal. represents the structural loss function that directly constrains the image quality, ⊙ represents the point multiplication operation between tensors, represents the matrix multiplication operation between tensors, B, C, W, H represent the batch number, channel number, width, height respectively, L represents the total number of layers, μ represents the function of calculating the mean, σ represents the function of calculating the variance, l, c and s represent the intermediate calculation functions, represents the nth different Block in the mth network layer, w l represents the weight of the lth layer, represents the perceived loss, represents the multi-layer structure loss, Represents a loss function with a norm of 1, VGG l (x l ) represents the output of the lth layer of the VGG network, and the input is x i,j，h,w With y i,j，h,w .

[0195] For the loss function The codebook structure interrupts the gradient generation. To help the codebook structure replace the tensor well, use The loss function calculates the difference before and after. sg(x) represents the gradient-free tensor x. The specific expression for the tensor z before and after encoding is as follows:

[0196]

[0197] The above represents a loss function with a norm of 2, sg(x) represents a tensor x without gradient, represents the loss function that constrains the codebook structure, z is the input intermediate tensor, and vq represents the codebook mechanism.

[0198] In the loss function In the , since the nearest neighbor encoding tensor will be passed through the codebook again after the 'W' type migration network, there is no need to use Loss function, a smoother overall average can be used Loss function to constrain, in addition to auxiliary style loss For local region feature consistency in high-dimensional tensors of newspapers, style loss The local detail features are reflected by calculating the Gram matrix of the tensor, where the Gram matrix is ​​obtained by the inner product between tensors. The loss function The expression is as follows:

[0199]

[0200] Among them, the above represents the style loss, represents the loss function with norm 2, It is represented as the value of the pixel at (w, h) in the cth channel of the bth batch of the tensor in the lth layer of the network. If some subscripts are omitted, it means all the selected tensors. x is the network prediction value and y is the true value. . represents general multiplication. B, C, W, and H represent the batch number, channel number, width, and height, respectively. Indicates the nth different Block in the mth network layer. represents matrix multiplication operation between tensors, represents the loss function that directly constrains the output tensor to align with the target tensor, and For the input tensor, Gram means calculating the Gram matrix.

[0201] Finally, the loss function It is also a constraint tensor. Although the intermediate tensor has no codebook re-encoding, it only needs to provide migration information. To ensure smoothness, only and The loss function constraint can ensure accelerated training, and the expression is as follows: With the pose estimation network W g Medium Tensor

[0202]

[0203] represents the constraint loss function used by the tensor alignment method for accelerating training, It is represented as the value of the pixel at (w, h) in the cth channel in the bth batch of the tensor located in the lth layer of the network. If some subscripts are omitted, it means all the selected tensors. . means general multiplication, sg(x) means the tensor x without gradient, represents the loss function with a norm of 2, L represents the total number of layers, represents the style loss, and the input is and in and represents the intermediate tensor output by the Tran layer, and Represents the target guidance tensor output by the Tran layer, where x is the network prediction value and y is the true value.

[0204] The human body image and posture image generation method of the present invention utilizes the characteristic that posture migration and posture estimation tasks have partially overlapping features, designs the simultaneous training of the two tasks, and accelerates the training speed of the two tasks by aligning the constraints in the tensor space. Based on the symmetric acceleration of the 'W' type network, the present invention shortens the training time while completing the posture migration and posture estimation tasks at the same time.

[0205] Related meaning explanation:

[0206] ResBIock and Res represent residual blocks; CodeBook represents the tensor recoding process; RA represents the core functional module of the 'W' type network; VQVAE represents the overall encoder-decoder structure; Encoder represents the encoder; Decoder represents the decoder; Wnet represents the 'W' type network structure; RA represents the overall encoder-decoder structure; SE represents the weight distribution block; Proj represents the information introduction module; Wt represents the weight map module; Tran represents the Zhang Liang constraint alignment module; Attn represents the attention module; GRU represents the gate control module; Conv represents the convolution layer normalization layer activation layer combination module; CrossAttn is the cross attention module; Res' is the fully connected residual layer; cat represents the splicing module; They represent tensor addition, subtraction, matrix operation, and dot multiplication operation respectively.

[0207] Embodiment 2:

[0208] This embodiment provides a human body and posture transformation image generation device, including a processor and a storage medium;

[0209] The storage medium is used to store instructions;

[0210] The processor is used to operate according to the instructions to execute the steps of the method according to embodiment 1.

[0211] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0212] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating a human body and posture transformation image, characterized in that: The following steps are involved: Acquire a source input image and source guidance information; the source input image includes a source human body image and a source posture image; the source guidance information includes a target human body image and a target posture image; Using the trained human encoder, the source human image and the target human image are compressed into the tensor space to form a source human tensor and a target human tensor; The source pose image and the target pose image are compressed into the tensor space using the trained pose encoder to form the source pose tensor and the target pose tensor; Connect the source human body tensor with the target human body tensor to form posture guidance information; Connect the source posture tensor and the target posture tensor to form human body guidance information; The source human body tensor combined with the human body guidance information is used as input information and input into the trained human body 'W' type migration network to obtain the predicted human body tensor. The source posture tensor combined with the posture guidance information is used as input information and input into the trained posture 'W' type migration network to obtain the predicted posture tensor; Input the predicted human body tensor into the trained human body decoder to obtain the predicted human body image; Input the predicted posture tensor into the trained posture decoder to obtain the predicted posture image; The human body ‘W’-type transfer network has the same structure as the posture ‘W’-type transfer network, both of which include multiple functional modules; Functional modules use the boot information V G Constantly guide the tensor V S To the target tensor V T Migration; functional modules include: Information is introduced into the module, so that the input tensor V S Introduce the guidance information of the target tensor, The weight graph module constructs a weight graph by calculating the difference between the input information and the target information in the guidance information to update the input tensor V S , the tensor alignment module, by migrating the network W in the 'W' type i With W g The intermediate tensor alignment between the constraints realizes dual-task acceleration, and the attention module uses the cross attention mechanism to make the tensor V S Extract more target information features.

2. The method for generating a human body and posture transformation image according to claim 1, characterized in that: The end of the human encoder and the posture encoder and the front end of the human decoder and the posture decoder are provided with a codebook mechanism; The codebook mechanism replaces the input tensor z with the tensor z′ that is closest to it in a learnable randomly initialized tensor table, and uses the codebook loss function to learn the parameters in the tensor table. The replacement process is expressed as follows: z′=vq(z); With h,w ′=e k ,k=argmin i ||from h,w -e i ||; Among them, e k represents the kth replacement tensor in the codebook, z h，w Represents the tensor component whose last two dimensions of tensor z are located at h and w, z h，w ′ represents the updated z h，w , argmin i ||z h，w -e i || means when |z h，w -e i || When the minimum value is taken, the value of i, z is the input tensor, z′ is the output tensor, cat h，w It indicates concatenation along the last two dimensions of tensor z, and vq indicates the codebook mechanism.

3. The method for generating a human body and posture transformation image according to claim 2, characterized in that: The end of the human encoder and posture encoder and the front end of the human decoder and posture decoder are provided with an Attn attention layer; The encoder structure of the human body encoder and posture encoder is expressed as: E(x)=AttnBlock(ResBlock m (ResBlock m-1 (...ResBlock 1 (x)))); The decoder structure of the human body decoder and posture decoder is expressed as: D(z)=ResBlock m (ResBlock m-1 (...ResBlock 1 (AttnBlock(z)))); Among them, AttnBlock is the attention mechanism layer, ResBlock is the residual layer, and ResBlock is the m is the residual layer in the mth network layer, the encoder is E, the decoder is D, the encoder input is x, and the decoder input is z.

4. The method for generating a human body and posture transformation image according to claim 3, characterized in that: The human body 'W' type transfer network and the posture 'W' type transfer network are both 'W' type transfer networks formed by a plurality of functional modules forming a 'W' shaped network structure, and the functional modules at the same level are connected through residuals to form a 'W' type transfer network; The output of the human body decoder and posture decoder is expressed as: Among them, cat() represents the splicing operation, is the source human image, is the source pose image, is the target pose image, is the target human image, is the human body guide tensor, Guide tensor for attitude; and They represent the predicted image after migration and the predicted posture after migration, is the predicted human image output by the human decoder, The predicted pose image output by the pose decoder; Where D i With D g They are human body decoder, posture decoder, E i With E g They are the human body encoder, posture encoder, W i With W g They are image transfer network and posture transfer network, vq i With vq g They represent the image codebook mechanism and the pose codebook mechanism respectively; The 'W' type migration network includes 2 main layers, 2 secondary layers, and 3 intermediate layers; and the number of main layers is n p , the number of sub-layers is n q , the number of intermediate layers is n m ,The outputs at different levels are as follows: Among them, cat() represents the splicing operation, Indicates that the tensor is in the mth branch and is the output tensor of all nth layers of the network. The final output of the 'W' type network is RA represents the core functional module in the 'W' type network, the superscript n represents the nth layer of the network, and V is the input of the 'W' type migration network.

5. The method for generating a human body and posture transformation image according to claim 4, characterized in that: The functional modules include a residual connection module, an information import module, a weight map module, a tensor constraint alignment module, an attention module and a channel weight redistribution module; The information import module, weight graph module, tensor constraint alignment module, and attention module are connected in series using residual links, and then connected in parallel with the channel weight redistribution module. The residual connection module is introduced at the input and output ends of the functional module to change the size and number of channels of the input tensor. The input tensor V l The function module is represented as: Where ⊙ represents the dot multiplication operation, V temp is the intermediate tensor, Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. represents the nth different Block in the mth network layer, ResBlock represents the residual connection module, Proj represents the information introduction module, SE represents the channel weight redistribution module, Wt represents the weight map module, Tran represents the tensor constraint alignment module, Attn represents the attention module, V l RA is the functional module of layer l l The input tensor, V l′ RA is the functional module of layer l l Output tensor, V l′ Represents the updated input tensor V l .

6. The method for generating a human body and posture transformation image according to claim 5, characterized in that: The channel weight redistribution module satisfies: Among them, AvgPool l is average pooling, LeakyReLU is an activation function, It is represented as the value of the pixel at (w, h) in the cth channel in the bth batch of the tensor located in the lth layer of the network. If some subscripts are omitted, it means all selected tensors. In this method, the input tensor is averaged along the channel dimension. FN represents the fully connected layer. represents the kth different fully connected module in the lth network layer; cat represents concatenation along the channel dimension, Sigmoid is an activation function, SE represents the channel weight redistribution module, V l and x is the input tensor of the functional module of the lth layer; B, W, H represent the batch number, width, and height respectively.

7. The method for generating a human body and posture transformation image according to claim 6, characterized in that: The information introduction module is used to perform the following steps: Will guide the tensor Split into current boot information With target guidance information With the target tensor V T Via ResBlock l After module projection, use GRU module update Using source tensor Update using GRU module The two are recombined to form a new guide tensor At the same time, the front and rear boot information will be updated With target guidance information Splice together through ConvBlock l Add after layer The residual connection of the information introduction module outputs the current tensor It is expressed as follows: Among them, GRU l The module is a gate control update module. Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. cat indicates concatenation along the channel dimension. ConvBlock indicates the convolutional layer module. ResBlock indicates the residual connection layer module. V T represents the target tensor, V′ represents the updated tensor V, represents the input guide tensor, Indicates the tensor to be migrated as input to the information introduction module. Indicates the current boot information of the information import module. Indicates the information introduced into the module target guidance information, and The input tensor of the information introduction module, and It is the guide tensor and current tensor output by the information introduction module. The superscript l represents the functional module of the lth layer. The weight graph module is used to perform the following steps: The guide tensor and current tensor output by the information introduction module and When passing through the weight graph module, the information is introduced into the guidance tensor output by the module The whole is updated through the convolutional layer to obtain the guide tensor output by the weight map module The guide tensor output by the weight map module Decomposed into the target guidance tensor output by the weight map module The current guide tensor output by the weight map module After the difference is made, it is fed into the hyperbolic tangent function tanh to obtain the tensor weight graph The weight graph module inputs the tensor to be migrated With tensor weight graph After the point multiplication, add the convolution layer The residual connection of the weight graph module obtains the current tensor output by As shown below: in, Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. cat indicates concatenation along the channel dimension. ConvBlock indicates the convolutional layer module. V″ indicates the tensor V′ after the weight map module is updated. represents the guidance tensor output by the information introduction module and input by the weight map module, Represents the tensor to be migrated output by the information introduction module and input by the weight graph module, Depend on and It is spliced ​​along the channel dimension. Indicates the current guidance information of the weight graph module, and Input tensor for weight graph module, and It is the guide tensor and current tensor output by the weight graph module, ⊙ represents the point multiplication operation between tensors, tanh represents the hyperbolic tangent function, and Wt represents the weight graph module; The execution method of the tensor alignment module includes: The guide tensor output by the weight map module The whole is updated through the convolution layer to obtain the guide tensor output by the tensor alignment module Align the tensor M l Approximate with a series of convolutions The tensor alignment module currently guides the tensor To the tensor alignment module target guide tensor The transformation aligns the tensor to the current guide tensor of the module Tensor alignment module target guided tensor is approached through two learnable fully connected modules and LeakyReLU activation function To simulate the alignment tensor M l , and finally the current tensor output by the weight graph module Aligned with tensor M l The guide tensor output by the tensor alignment module in the task of doing matrix operations close to symmetry Get the hidden tensor After the convolution layer, add the current tensor output by the tensor alignment module The residual connection of the tensor alignment module obtains the current tensor output As follows: Among them, the above Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. cat indicates concatenation along the channel dimension. ConvBlock indicates the convolutional layer module. represents the kth different convolutional layer module in the lth network layer, V″′ represents the tensor V″ updated by the tensor alignment module, Represents the guide tensor input to the tensor alignment module and is also the guide tensor output by the weight graph module. It represents the tensor to be migrated input by the tensor alignment module and the tensor to be migrated output by the weight graph module. Indicates the current guidance information of the tensor alignment module. Represents the tensor alignment module target guidance information, and is the input tensor of the tensor alignment module, and The guide tensor and the current tensor are the outputs of the tensor alignment module, Represents matrix multiplication between tensors, LeakyReLU represents an activation function, Tran l Indicates the function module RA l The tensor constraint alignment module in , FN represents the fully connected module, represents the kth different fully connected module in the lth network layer; The execution method of the attention module includes: The current tensor output by the tensor alignment module With the guide tensor When passing through the attention module, The whole is updated through the convolutional layer to obtain the guidance tensor output by the attention module Will Respectively with the guide tensor The current guide tensor in and the target guide tensor The concatenation is fed into the cross attention mechanism and updated using the cross attention mechanism And then add after the convolution layer The residual connection of the attention module obtains the current tensor output by It is expressed as follows: Among them, the above Indicates that the tensor is in the mth branch and is the output tensor of the entire nth layer network. cat indicates concatenation along the channel dimension. ConvBlock indicates the convolutional layer module. V″″ indicates the tensor V″′ after the attention module updates. Depend on and It is spliced ​​along the channel dimension. Indicates the current guidance information of the attention module, represents the target guidance information of the attention module, and is the input tensor of the attention module, and is the output tensor of the attention module, attention represents the attention mechanism function, CrossAttn represents the cross attention mechanism, Attn represents the attention module, FN represents the fully connected module, the superscript l represents the lth functional module, and the subscript n represents the nth module.

8. The method for generating a human body and posture transformation image according to claim 7, characterized in that: The training method of the human body encoder, posture encoder, human body decoder and posture decoder comprises: Acquire a training data set; the training data set includes at least human body images and posture images; The human encoder, the pose encoder, the human decoder and the pose decoder are trained using the training data set and the loss function based on the overall training VQVAE framework to obtain the trained human encoder, pose encoder, human decoder and pose decoder; Methods for obtaining training data sets include: Obtain human image data from the DeepFashion public human dataset; The posture estimation algorithm OpenPose is used to extract the posture of human image data to obtain human posture data; Pair the data of the same human body image and posture image, and take the paired data of two groups of different actions of the same person to form a set of training data, so that one set of training data contains the source human body image, the target human body image, the source posture image and the target posture image, and a training data set is produced.

9. The method for generating a human body and posture transformation image according to claim 8, characterized in that: The training method of the human body 'W' type transfer network and the posture 'W' type transfer network includes: Get the tensor training dataset; Based on the tensor training data set and the loss function used to train the overall migration network 'W', the human body 'W' type migration network and the posture 'W' type migration network are trained to obtain the trained human body 'W' type migration network and the posture 'W' type migration network.

10. A human body and posture transformation image generation device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-9.

Images