A motion transfer method using depth information assistance and contour enhancement loss

Abstract

The present invention discloses a motion migration method using depth information assistance and contour enhancement loss, a depth image generator based on contour attention GAN, a motion image generator based on self-attention mechanism, posture normalization adjusted according to the target character skeleton and the source character skeleton in proportion, and motion migration is realized according to the skeleton after posture normalization. The present invention adopts a generator based on contour attention GAN combined with depth information assistance to make up for the defect of lack of depth information and improve the accuracy of generated contour details. The generator using self-attention mechanism improves the quality of generated texture details, and the generated motion image has reliable depth information and reasonable contours.

Description

Technical Field

[0001] The invention relates to a motion migration method using depth information assistance and contour enhancement loss, and belongs to the technical field of image processing. Background Art

[0002] Human posture transfer has been a research hotspot in the field of computer vision in recent years. It has many application scenarios in the artificial intelligence industry, such as film production, online entertainment, and pedestrian re-identification. Traditional motion transfer is achieved through computer graphics technology by simulating complex physical models, which will cause expensive time and computing costs. The existing motion transfer task is to generate a human image corresponding to the target posture using a generative adversarial network given the posture key point information of the source image. The generated image corresponds to the target posture in terms of human posture and movement, and the texture information is consistent with the texture information of the initial posture image. Due to the development of generative neural networks, it is possible to simulate computer graphics technology with generative networks. The probability distribution of appearance texture is captured through the GAN network to generate realistic images and videos. It has promoted the realization of our motion transfer task.

[0003] Human motion based on images and videos can be seen as a deformation process of human appearance, so motion transfer is usually seen as a pose-guided image generation task that can be solved by a GAN-based framework. However, real motion occurs in three-dimensional space, and the images generated on a two-dimensional plane inevitably lack the depth information guidance of the original motion, which will lead to confusion in the depth of different body parts. In addition, GAN is not good at handling tasks with unified regions, and adversarial loss does not have a strong constraint on the details of the contour. This will inevitably cause the contours of the generated motion images to be deformed or missing, and the details to be blurred.

[0004] Therefore, contour deformation or loss and image detail blurring that occur during human posture transfer are technical problems that those skilled in the art urgently need to solve. Summary of the invention

[0005] Purpose: In order to overcome the shortcomings of contour deformation or loss and blurred image details in the prior art, the present invention provides a motion migration method using depth information assistance and contour enhancement loss.

[0006] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A motion migration method using depth information assistance and contour enhancement loss includes the following steps:

[0008] The skeleton image and depth image of the target person are used to train a one-stage contour-attention GAN to obtain a trained one-stage contour-attention GAN-based deep image generator.

[0009] The skeleton image of the target person is input into the trained one-stage contour attention GAN-based deep image generator, and the one-stage depth image of the target person is output.

[0010] The first-stage depth map and the skeleton map of the target person are used to train the second-stage self-attention mechanism-based GAN to obtain a trained two-stage motion image generator based on the self-attention mechanism GAN.

[0011] The skeleton image of the source person in each posture is obtained, and the length of each bone segment of the skeleton image of the source person is adjusted according to the bone segment in the skeleton image of the target person, so as to obtain the normalized skeleton image of the source person corresponding to each posture of the source person.

[0012] The normalized skeleton image of the source character is input into the trained depth image generator to obtain the depth image of the target character in each posture. Then the depth image of the target character in each posture and the normalized skeleton image of the source character are input into the trained motion image generator to obtain the motion image of the target character doing the action of the source character.

[0013] As a preferred solution, the first stage based on contour attention GAN includes: a depth image generator, a HED network, and a discriminator D1. The depth image generator is used to generate a depth map according to an input skeleton map; the HED network is used to extract a skeleton map from the depth map; the discriminator D1 is used to determine whether the depth map generated by the depth image generator is Real or Fake.

[0014] As a preferred solution, the depth image generator includes: an encoder and a decoder, and n pairs of residual blocks and spatial attention blocks are arranged between the encoder and the decoder, and n≥1.

[0015] As a preferred solution, the skeleton image and depth image of the target person are used to train a first-stage contour attention GAN, including: deep image generator training and discriminator D1 training.

[0016] When training the deep image generator, in each iteration, the output of the deep image generator is placed in the discriminator D1 to calculate the overall loss function of one stage, and back propagation is performed. The network parameters of the deep image generator are updated using the stochastic gradient descent method until the upper limit of the training times is reached to obtain a trained deep image generator.

[0017] When the discriminator D1 is trained, in each iteration, an image is taken from the training data and the output of the deep image generator, scaled, and input into the discriminator D1 at the same time to obtain the overall loss function of one stage, perform back propagation, and use the stochastic gradient descent method to update the network parameters of the discriminator D1 until the upper limit of the training times is reached to obtain the trained discriminator D1.

[0018] As a preferred solution, the overall loss function calculation formula of the stage is as follows:

[0019]

[0020] Among them, L DEP (G1) is the contour loss, L GAN1 (G1, D1) is the loss of the first stage based on contour attention GAN, L FM (G1, D1) is the feature matching loss of discriminator D1, L VGG (G1) is the deep image generator perceptual reconstruction loss, λ DEP , FM , VGG are weight parameters, Minimize the loss function for the deep image generator, Maximize the loss function for the discriminator D1.

[0021] As a preferred solution, the two-stage self-attention mechanism-based GAN includes: a motion image generator and a discriminator D2. The motion image generator is used to generate a motion image based on an input depth map and a skeleton map; the discriminator D2 is used to discriminate whether the motion image generated by the motion image generator is real or fake.

[0022] As a preferred solution, the motion image generator includes: an encoder and a decoder, and m pairs of residual blocks and self-attention layers are arranged between the encoder and the decoder, and m≥1.

[0023] As a preferred solution, the first-stage depth map and the skeleton map of the target person are used to train the second-stage self-attention mechanism GAN, including: motion image generator training and discriminator D2 training.

[0024] When training the deep image generator, in each iteration, the output of the motion image generator is placed in the discriminator D2 to calculate the overall loss function of the two stages, and back propagation is performed. The network parameters of the motion image generator are updated using the stochastic gradient descent method until the upper limit of the training times is reached to obtain a trained motion image generator.

[0025] When the discriminator D2 is trained, in each iteration, an image is taken from the training data and the output of the motion image generator, scaled, and input into the discriminator D2 at the same time to obtain the overall loss function of the two stages, perform back propagation, and use the stochastic gradient descent method to update the network parameters of the discriminator D2 until the upper limit of the training times is reached to obtain the trained discriminator D2.

[0026] As a preferred solution, the calculation formula of the overall loss function of the two stages is as follows:

[0027]

[0028] in, is the global adversarial loss, L FM (G2, D2) is the feature matching loss of discriminator D2, L VGG (G2) is the perceptual reconstruction loss of the motion image generator, λ FM , VGG are weight parameters, Minimize the loss function for the motion image generator, Maximize the loss function for the discriminator D2.

[0029] Beneficial effects: The present invention provides a motion migration method using depth information assistance and contour enhancement loss. In the first stage, a contour attention GAN with contour enhancement loss is used to generate a depth map of the target person. This not only brings the depth information of the original motion in three-dimensional space, but also aligns the body area with reliable contours to facilitate the subsequent character image generation. A generator based on contour attention GAN combined with depth information assistance is used to make up for the lack of depth information and improve the accuracy of the generated contour details. In the second stage, a self-attention GAN is used to generate the final motion image with the target posture and depth map generated in the first stage as input. The generator with the self-attention mechanism used improves the quality of the generated texture details, and the generated motion image has reliable depth information and reasonable contours. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the process of the present invention.

[0031] Figure 2 Schematic diagram of one-stage contour-attention-based GAN network training.

[0032] Figure 3 Schematic diagram of two-stage GAN network training based on self-attention mechanism.

[0033] Figure 4 It is a schematic diagram of the process of movement migration of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with specific embodiments.

[0035] like Figure 1 As shown, a motion migration method using depth information assistance and contour enhancement loss includes the following steps:

[0036] The skeleton image and depth image of the target person are used to train a one-stage contour-attention GAN to obtain a trained one-stage contour-attention GAN-based deep image generator.

[0037] The skeleton image of the target person is input into the trained one-stage contour attention GAN-based deep image generator, and the one-stage depth image of the target person is output.

[0038] The first-stage depth map and the skeleton map of the target person are used to train the second-stage self-attention mechanism-based GAN to obtain a trained two-stage motion image generator based on the self-attention mechanism GAN.

[0039] The skeleton image of the source person in each posture is obtained, and the length of each bone segment of the skeleton image of the source person is adjusted according to the bone segment in the skeleton image of the target person, so as to obtain the normalized skeleton image of the source person corresponding to each posture of the source person.

[0040] The normalized skeleton image of the source character is input into the trained depth image generator to obtain the depth image of the target character in each posture. Then the depth image of the target character in each posture and the normalized skeleton image of the source character are input into the trained motion image generator to obtain the motion image of the target character doing the action of the source character.

[0041] First embodiment:

[0042] The data set made by myself is selected to verify and analyze the present invention. The model in this paper adopts Python language to construct the model. The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0043] A motion migration method using depth information assistance and contour enhancement loss includes the following steps:

[0044] Step 1: Obtain a moving image S of the target person as training data for the first-stage contour-attention GAN network and the second-stage self-attention mechanism GAN network.

[0045] The training data is the video frames of the target person. During the video shooting process, the position of the camera or mobile phone is fixed. Each video frame is cut and arranged at a resolution of 512×512 to form a motion image S of the target person. The target person's motion image S is extracted using the current more mature and accurate openpose network to extract the target person's skeleton image P. For depth information, the BMD network is used to extract the high-resolution depth map H of the target person. The target person's skeleton image P and depth map H are used as training data.

[0046] Step 2: Use the skeleton image P and depth image H of the target person to train the one-stage contour attention GAN to obtain the trained one-stage contour attention GAN-based depth image generator G1.

[0047] The first stage of contour attention GAN training process, such as Figure 3 As shown in Figure 2, a model is designed to generate a one-stage depth map He of the target person from the skeleton map P to introduce depth information.

[0048] The first stage of the contour attention GAN includes: a deep image generator G1, a HED network, and a discriminator D1. The deep image generator G1 adopts a coding and decoding architecture, and the residual block and spatial attention block added in the middle input the skeleton image P of the target person, and outputs the first-stage depth image He of the target person. In order to ensure the accuracy of the contour information of the generated depth image, the HED network is also used to extract the first-stage depth image He of the target person and the skeleton image Ye and skeleton image Y of the target person's depth image H to obtain the contour loss L. DEP (G1) is used to constrain the depth image generator G1 to generate a depth map with accurate contour information. The discriminator D1 determines whether the depth map He generated in the first stage of the target person is real or fake, which is a binary classification problem. G1 and D1 are trained alternately separately, and the number of iterations is defined at the same time, and the alternating iterations are stopped. The game is played in the continuous training process, and the capabilities of G1 and D1 are improved at the same time, and finally the desired dynamic balance is achieved.

[0049] Under the constraint of contour loss, when generating an image, the spatial attention block in the deep image generator G1 will focus more on the contour of the characters in the image, which greatly solves the contour loss problem caused by deformation, improves the computational efficiency of the network, and ensures the quality of the generated image.

[0050] The introduced spatial attention mechanism takes the downsampled feature map in the one-stage deep image generator G1 as the input of the spatial attention block, concatenates the results of the input feature map after maximum pooling and average pooling, reduces the dimension to a single channel through a layer of convolution, and then generates the weight of the spatial attention through sigmoid. Finally, the weight is multiplied by the input feature map to obtain the features after the attention block.

[0051] In order to generate a depth map with more accurate contour information to ensure spatial consistency with the moving image, the training process is constrained by contour loss, and the spatial attention focuses on the contour as the information part. Pooling operations along the channel axis can effectively highlight the information area.

[0052] Use convolutional layers to generate a spatial attention map M S (F)∈RH×W The channel information of the feature map is aggregated through two pooling operations to generate a two-dimensional map: and Represent the average pooling features and the maximum pooling features on the channel respectively. They are connected and then convolved to generate a 2D spatial attention map. The calculation of spatial attention is as follows:

[0053]

[0054] Here σ represents the sigmoid function, f 7×7 Represents a 7×7 filtering operation.

[0055] The structure of the one-stage deep image generator G1 based on contour attention GAN is an encoder, and the intermediate structure is that a layer of spatial attention block is added after each layer of residual block. The decoder, the feature map after the feature is extracted by the residual block can be expressed as follows after the spatial attention block:

[0056] F=Res(f)M S (f) (2)

[0057] Here f is the downsampled featuremap, Res(f) is the featuremap after the residual block, and M S (.) is the weight of the calculated spatial attention. The calculated F is both the output of the previous attention layer and the input of the next residual block.

[0058] The global adversarial loss has a weak constraint on shape details, so the contour loss, which is the traditional L2 loss, is added to force the first-stage contour attention GAN model to focus on the shape of the body, aiming to reduce the difference between the real sample H of the target person's depth map and the generated first-stage depth map He of the target person.

[0059] During the training process, the spatial attention module added in the first stage of the depth image generator G1 based on the contour attention GAN will increase the weight on the contour information, which plays a complementary role so that the final generated depth map has more accurate shape information. The formula of contour loss is as follows:

[0060]

[0061] where HED(.) is an accurate method for extracting contours, G1 is a one-stage contour attention GAN-based deep image generator, G1(P) is the depth map generated from the skeleton map p of the target person, H is a real sample of the depth map of the target person, and E(·) represents the mathematical expectation.

[0062] The main task of the depth image generator G1 is to capture the global body shape distribution and depth distribution of the real sample H and generate a depth map with relatively correct contours.

[0063] The loss formula of the first stage contour attention GAN is as follows:

[0064] L GAN1 (G1, D1) = E (P，H) [log D1(P,H)]+E p [log(1-D1(P,G1(P)))] (4)

[0065] E(·) represents the mathematical expectation, D1(P, H) represents P as a condition, H is the probability value obtained by splicing the two real samples as the discriminator input, G1(P) represents the sample generated after P passes through the generator, and D1(P, G1(P)) represents P as a condition, G1(P) is the probability value obtained by splicing the two fake samples as the discriminator input.

[0066] In order to ensure the global consistency of image content, feature matching loss and perceptual reconstruction loss are used. FM (G1, D1) is the feature matching loss of discriminator D1, as shown in Equation 5. The generated depth image is input to discriminator D1 to compare the differences between features at different layers. (i) N represents the i-th layer feature extractor in the discriminator D1. i It represents the number of elements in each layer, and T represents the total number of layers.

[0067]

[0068] L VGG (G1) is the G1 perceptual reconstruction loss, where the generated images and real images are fed into a pre-trained VGG network to compare features at different layers. (i) Represents the third layer feature extractor of the VGG network. M i represents the number of elements in each layer, and N represents the total number of layers.

[0069]

[0070] In order to make the difference between the generated depth map and the real sample small enough, the contour loss, feature matching loss and perceptual reconstruction loss should be minimized during the training process.

[0071] The overall loss function of one stage is as follows:

[0072]

[0073] The weight parameters of feature matching loss and perceptual reconstruction loss are set as: FM =λ VGG = 20, the weight parameter to minimize the contour loss is set to λ DEP =20, Minimize the loss function for the deep image generator, Maximize the loss function for the discriminator D1.

[0074] When training a stage of contour attention-based GAN, it includes training of the deep image generator G1 and the discriminator D1.

[0075] When the depth image generator G1 is trained, in each iteration, the output of the depth image generator G1 is placed in the discriminator D1 to calculate the overall loss function of one stage, and back propagation is performed. The network parameters of the depth image generator G1 are updated using the stochastic gradient descent method until the upper limit of the training times is reached, thereby obtaining the trained depth image generator G1.

[0076] When the discriminator D1 is trained, in each iteration, an image is taken from the training data and the output of the deep image generator G1, scaled, and input into the discriminator D1 at the same time to obtain the overall loss function of one stage, perform back propagation, and use the stochastic gradient descent method to update the network parameters of the discriminator D1 until the upper limit of the training times is reached to obtain the trained discriminator D1.

[0077] Step 3: Use the first-stage depth map He and the skeleton map P of the target person to train the second-stage self-attention mechanism-based GAN to obtain the trained two-stage self-attention mechanism-based GAN motion image generator G2.

[0078] The second stage is based on the self-attention mechanism GAN training process, such as Figure 3 As shown in the figure, the first-stage depth map H of the target person generated in step 2 and the skeleton map P of the target person are spliced ​​as the input of the second-stage motion image generator G2 based on the self-attention mechanism GAN, and the second-stage motion image Se of the target person is output.

[0079] By compensating for the depth information and shape information, the two-stage self-attention mechanism-based GAN can more accurately estimate the statistical distribution of the overall features. The two-stage self-attention mechanism-based GAN includes: a motion image generator G2 and a discriminator D2. In order to further improve the quality of generation, a self-attention mechanism is added to the motion image generator G2. The discriminator D2 determines whether the two-stage motion image Se of the target person is Real or Fake, which is a binary classification problem. G2 and D2 are trained alternately separately, and the number of iterations is defined at the same time, and the alternating iterations stop. The game is played in the continuous training process, and the capabilities of G2 and D2 are improved at the same time, and the desired dynamic balance is finally achieved.

[0080] The structure of the generator G2 is an encoder, and the intermediate structure is a self-attention layer and a decoder added after each layer of residual block. The attention mechanism takes the featuremap2 of CxHxW dimension after downsampling of the motion image generator G2 as input, where C represents the channel, H represents the height, and W represents the width. The global information is compressed into a featuremap2 of Cx1x1 dimension through the global modeling module, and then the dependency on the channel is captured through the bottleneck layer transformation module. Finally, the transformed feature dimension of Cx1x1 is added to the input featuremap2 of CxHxW dimension, making the global information common to every point. The specific formula is as follows:

[0081]

[0082] Will That is, the target person's first-stage depth map He and the target person's skeleton map P are spliced ​​and then downsampled by the motion image generator G2. The feature map of CxHxW dimensions is used as input, and Z is the output of the motion image generator G2. p is the number of positions in featuremap2 (for example, H×W for an image), and i, j, m are the position indices of the query. k , W v1 , W v2 represents a linear transformation matrix (such as a 1×1 convolution). is the weight value of global attention, δ(.) = W v2 ReLu(LN(W v1 (.))) represents the process of bottleneck layer transformation module, LN(.) represents layer normalization, and ReLu is the activation function. The features obtained after the output of each residual block and self-attention layer in G2 are upsampled by the decoder to obtain the generated motion image Se.

[0083] Correspondingly, the two-stage self-attention mechanism GAN loss function is as follows:

[0084] The ultimate goal of the motion image generator G2 is to generate a two-stage motion image Se of the target person that is consistent with the global distribution of the real target person’s motion image. The global adversarial loss is as follows:

[0085]

[0086] E(·) represents the mathematical expectation, D2(P, H, S) represents P and H as conditions, S is the probability value obtained by concatenating the three real samples as the discriminator input, G2(P, H) represents the sample generated by P and H after passing through the generator, and D2(P, H, G2(P, H)) represents P and H as conditions, G2(P, H) is the probability value obtained by concatenating the three fake samples as the discriminator input.

[0087] Similarly, in order to ensure the global consistency of image content in the second stage, we use feature matching loss and perceptual reconstruction loss. FM (G2, D2) is the feature matching loss of discriminator D2, as shown in Equation 10. The generated motion image is input to discriminator D2 to compare the differences between features at different layers. (i) N represents the i-th layer feature extractor in the discriminator D2. i It represents the number of elements in each layer, and T represents the total number of layers.

[0088]

[0089] L VGG (G2) is the G2 perceptual reconstruction loss, the generated motion images and real motion images are input into the pre-trained VGG network to compare the features of different layers. (i) Represents the third layer feature extractor of the VGG network. M i represents the number of elements in each layer, and N represents the total number of layers.

[0090]

[0091] The total losses in the second phase are as follows;

[0092]

[0093] The weight parameters of the feature matching loss and perceptual reconstruction loss in the second stage are set as: FM =λ VCG =20, Minimize the loss function for the motion image generator, Maximize the loss function for the discriminator D2.

[0094] When training the one-stage self-attention mechanism GAN, it includes the training of the motion image generator G2 and the discriminator D2.

[0095] When the deep image generator G2 is trained, in each iteration, the output of the motion image generator G2 is placed in the discriminator D2 to calculate the overall loss function of the two stages, and back propagation is performed. The network parameters of the motion image generator G2 are updated using the stochastic gradient descent method until the upper limit of the training times is reached, thereby obtaining the trained motion image generator G2.

[0096] When the discriminator D2 is trained, in each iteration, an image is taken from the training data and the output of the motion image generator G2, respectively, and after scaling, it is input into the discriminator D2 at the same time to obtain the overall loss function of the two stages, and back propagation is performed. The network parameters of the discriminator D2 are updated using the stochastic gradient descent method until the upper limit of the training times is reached, and the trained discriminator D2 is obtained.

[0097] Step 4: Figure 4 As shown, the source character motion image X is obtained, and the skeleton image X of the source character is extracted from the source character motion image X using the openpose network. p , adjust the source character's skeleton X p The length of each bone segment of the image is adjusted to make it close to the target person. The representation of 2D motion is the projection of 3D motion, so the length of the bone segment in the image is not fixed. After 2D projection, bone segments in different directions show different lengths, so it is difficult to accurately know the exact length of the bone segment.

[0098] The maximum length of the projected bone segment is closest to the true length, so the maximum projected bone segment length in the training dataset is used to approximate the true length. The actual bone segment length of the source character's new pose can be approximated with the target character's bone segment length in the same way. Then, the corresponding bone segment length is adjusted according to the ratio between the target character's pose and the source character's new pose.

[0099] d i represents the i-th joint coordinate of the new pose, Represents the coordinates of its i-th parent joint. i Depend on Make adjustments, including and They represent the maximum bone segment length between the i-th joint and its parent joint in the target person's skeleton graph P, and the source person's skeleton graph X p The maximum bone segment length between the i-th joint and its parent joint in . This method is used to obtain the normalized skeleton graph X of the source character in each posture. pn .

[0100] Step 5: The present invention proposes a motion migration method using depth information assistance and contour enhancement loss, which includes two stages of GAN, and the motion migration is completed by a first-stage deep image generator G1 based on contour attention GAN and a second-stage motion image generator G2 based on self-attention mechanism GAN.

[0101] Normalize the skeleton image X of the source character pn Input the trained depth image generator G1 to obtain the depth map D of the target person in each posture pn, and then the depth map D of the target person in each posture pn The skeleton image of the normalized source character X pn The trained motion image generator G2 is input to obtain the motion image Y of the target person doing the action of the source person, thus completing the entire process of motion migration from the source person to the target person.

[0102] The present invention takes into account that the emerging visual technology based on short video and film production in today's society can greatly enhance the user experience and inject new vitality into video culture. The application scope of the present invention is as follows:

[0103] (1) In the field of culture and entertainment, we can use our model to enable ordinary people to imitate the movements of professional dancers, thereby improving user experience and increasing happiness in life.

[0104] (2) In the field of film production, difficult stunts can be performed at a low cost.

[0105] (3) In the field of person re-identification, we can use our method to expand the dataset of a person in different postures and improve the accuracy of person re-identification.

[0106] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A motion migration method using depth information assistance and contour enhancement loss, characterized in that: The steps include: The skeleton image and depth image of the target person are used to train the one-stage contour attention GAN to obtain a trained one-stage contour attention GAN-based deep image generator; Input the skeleton image of the target person into the trained one-stage contour attention GAN-based deep image generator, and output the one-stage depth image of the target person; The first-stage depth map and the skeleton map of the target person are used to train the second-stage self-attention mechanism-based GAN to obtain a trained two-stage self-attention mechanism-based GAN motion image generator; Obtaining a skeleton image of the source person in each posture, adjusting the length of each bone segment of the skeleton image of the source person according to the bone segments in the skeleton image of the target person, and obtaining a normalized skeleton image of the source person corresponding to each posture of the source person; The normalized skeleton image of the source character is input into the trained depth image generator to obtain the depth image of the target character in each posture. Then the depth image of the target character in each posture and the normalized skeleton image of the source character are input into the trained motion image generator to obtain the motion image of the target character doing the action of the source character.

2. The motion migration method using depth information assistance and contour enhancement loss according to claim 1, characterized in that: The first stage based on contour attention GAN includes: a depth image generator, a HED network, and a discriminator D1. The depth image generator is used to generate a depth map according to an input skeleton map; the HED network is used to extract a skeleton map from the depth map; the discriminator D1 is used to determine whether the depth map generated by the depth image generator is real or fake.

3. The motion migration method using depth information assistance and contour enhancement loss according to claim 2, characterized in that: The depth image generator includes: an encoder and a decoder, and n pairs of residual blocks and spatial attention blocks are arranged between the encoder and the decoder, and n≥1.

4. The motion migration method using depth information assistance and contour enhancement loss according to claim 2, characterized in that: The skeleton image and depth image of the target person are used to train the first-stage contour attention GAN, including: training of the deep image generator and the discriminator D1; When training the deep image generator, in each iteration, the output of the deep image generator is placed in the discriminator D1 to calculate the overall loss function of one stage, and back propagation is performed. The network parameters of the deep image generator are updated using the stochastic gradient descent method until the upper limit of the training times is reached, and a trained deep image generator is obtained; When the discriminator D1 is trained, in each iteration, an image is taken from the training data and the output of the deep image generator, scaled, and input into the discriminator D1 at the same time to obtain the overall loss function of one stage, perform back propagation, and use the stochastic gradient descent method to update the network parameters of the discriminator D1 until the upper limit of the training times is reached to obtain the trained discriminator D1.

5. The motion migration method using depth information assistance and contour enhancement loss according to claim 4, characterized in that: The overall loss function calculation formula for the stage is as follows: Among them, L DEP (G1) is the contour loss, L GAN1 (G1, D1) is the loss of the first stage based on contour attention GAN, L FM (G1, D1) is the feature matching loss of discriminator D1, L VGG (G1) is the deep image generator perceptual reconstruction loss, λ DEP , FM , VGG are weight parameters, Minimize the loss function for the deep image generator, Maximize the loss function for the discriminator D1.

6. The motion migration method using depth information assistance and contour enhancement loss according to claim 1, characterized in that: The two-stage self-attention mechanism-based GAN includes: a motion image generator and a discriminator D2. The motion image generator is used to generate a motion image according to an input depth map and a skeleton map; the discriminator D2 is used to discriminate whether the motion image generated by the motion image generator is real or fake.

7. The motion migration method using depth information assistance and contour enhancement loss according to claim 6, characterized in that: The motion image generator includes: an encoder and a decoder, wherein m pairs of residual blocks and self-attention layers are arranged between the encoder and the decoder, and m≥1.

8. The motion migration method using depth information assistance and contour enhancement loss according to claim 6, characterized in that: The first-stage depth map and skeleton map of the target person are used to train the second-stage self-attention mechanism GAN, including: motion image generator training and discriminator D2 training; When training the deep image generator, in each iteration, the output of the motion image generator is put into the discriminator D2 to calculate the overall loss function of the second stage, and back propagation is performed. The network parameters of the motion image generator are updated using the stochastic gradient descent method until the upper limit of the training times is reached, and a trained motion image generator is obtained; When the discriminator D2 is trained, in each iteration, an image is taken from the training data and the output of the motion image generator, scaled, and input into the discriminator D2 at the same time to obtain the overall loss function of the two stages, perform back propagation, and use the stochastic gradient descent method to update the network parameters of the discriminator D2 until the upper limit of the training times is reached to obtain the trained discriminator D2.

9. The motion migration method using depth information assistance and contour enhancement loss according to claim 8, characterized in that: The calculation formula of the overall loss function of the two stages is as follows: in, is the global adversarial loss, L FM (G2, D2) is the feature matching loss of discriminator D2, L VGG (G2) is the perceptual reconstruction loss of the motion image generator, λ FM , VGG are weight parameters, Minimize the loss function for the motion image generator, Maximize the loss function for the discriminator D2.

Images