A method for generating human motion transfer images based on neural radiation fields

Abstract

The present invention relates to the fields of computer graphics and deep learning, and in particular to a method for generating a human motion transfer image based on a neural radiation field, comprising: preprocessing a source human motion image and a target motion semantic mask, and inputting the preprocessed NeRF decoupling generator G after training, obtaining the semantic category probability S, density σ and color C to which each pixel point output by G belongs; generating a synthetic human image, a synthetic semantic mask and an inverted semantic mask based on the semantic category probability S, density σ and color C; adaptively adjusting the parameters of the G according to the synthetic semantic mask and the inverted semantic mask; and reacquiring a synthetic human image of the human motion transfer image that is finally output. The above method solves the problem of shape and appearance being adhered to each other during motion transfer in the existing method, and eliminates the dependence of human image synthesis on bone priors.

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics and deep learning, and in particular to a method for generating human motion migration images based on neural radiation fields. Background Art

[0002] In recent years, with the rapid development of computer vision and image processing technology, human motion transfer has become a research field that has attracted much attention. Human motion transfer aims to convert an image of a person in one posture into an image of the person in another posture, thereby achieving flexible control and editing of human motion. It has broad application prospects in the fields of movie special effects, sports live broadcasts, or data enhancement. Due to the large spatial inconsistency between the source human body and the human body after motion transfer, most existing methods for generating human images with motion transfer require the use of paired source-target images, that is, real images of the human body after motion transfer are required as supervision for training.

[0003] Some of these methods adopt a dual-branch framework including an action branch and an appearance branch. Given a source human action image and a target human action image, the action branch is used to extract the information features of the human skeleton key points, and the appearance branch is used to extract the human appearance features, and then the two features are fused to achieve feature transfer of the source image.

[0004] When trying to synthesize high-definition fine-grained and plausible images, using only human skeleton keypoints as action representations can be challenging, especially when there are significant differences between the source and target actions. Most of the above-mentioned methods require paired source-target images as supervision for training. At the same time, the above methods have limitations in generating high-quality results, and cannot fully utilize the spatial information available for the same person from different views in the training dataset during the training phase. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for generating a human motion migration image based on a neural radiation field.

[0007] (II) Technical solution

[0008] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, an embodiment of the present invention provides a method for generating a human motion migration image based on a neural radiation field, comprising:

[0010] S10, for the source human action image and the target action semantic mask to be analyzed, obtain the appearance latent code and the shape latent code; input the source human action image and the target action semantic mask, the appearance latent code, and the shape latent code into the pre-trained NeRF decoupling generator G, and obtain the semantic category probability s, density σ, and color c output by G to be used for rendering;

[0011] S20, obtaining a synthetic semantic mask and a synthetic human body image by means of a rendering method according to the semantic category probability s, density σ and color c;

[0012] S30, according to the synthesized human body image and the target action semantic mask, using G to obtain the semantic category probability s and density σ to which each pixel of the inverted semantic mask belongs, and obtain the inverted semantic mask;

[0013] S40, determining whether it is necessary to adaptively adjust the parameters of G according to the synthetic semantic mask and the inverted semantic mask;

[0014] S50. If it is necessary to adjust the parameters of G, then the parameters of G are adjusted based on the adaptive adjustment, and the synthesized human body image is re-output based on the G with the adjusted parameters, until the synthesized human body image output when G does not need to be adaptively adjusted is used as the final output human body motion migration image.

[0015] Optionally, the G includes: a first mapping network based on a multi-layer perceptron, a second mapping network based on a multi-layer perceptron, a semantic neural radiation field, an appearance neural radiation field, and a semantic decoder;

[0016] The first mapping network input is a latent code of shape z s ; Output is the first frequency intermediate feature γ s and the first phase shift intermediate characteristic β s ;

[0017] The input of the second mapping network is the appearance latent code z a , the output is the second frequency intermediate feature γ a and the second phase shift intermediate characteristic β a ;

[0018] The output of the semantic neural radiation field is density σ;

[0019] The input of the appearance neural radiation field is the density σ, and the output is the color c;

[0020] The input of the semantic decoder is the density σ, and the output is the semantic category probability s;

[0021] Among them, the shape latent code z s The latent code z is generated by randomly sampling the semantic mask of the target action and follows the normal distribution.a The latent codes are generated by randomly sampling the source human action images and follow the normal distribution.

[0022] Optionally, S20 includes:

[0023] The semantic category probability s and density σ are rendered into a synthetic semantic mask by a semantic rendering module, and the color c and density σ are rendered into a synthetic human image by a color rendering module;

[0024] Alternatively, the S30 includes:

[0025] Obtaining the appearance latent code and shape latent code of the synthesized human image and the semantic mask of the target action; inputting the synthesized human image, the semantic mask of the target action, the shape latent code, and the appearance latent code into G, obtaining the semantic category probability s and density σ to which each pixel point of the inverted semantic mask belongs, and obtaining the inverted semantic mask through semantic rendering;

[0026] Alternatively, the S40 includes:

[0027] According to formula (1), the loss summary information of the synthetic semantic mask and the inverted semantic mask is obtained. If loss summary information If it exceeds the preset range, the parameters of G are adaptively adjusted;

[0028]

[0029] in, represents the loss function that ensures that the synthesized human image is similar to the source human image, represents the loss function that ensures that the synthetic semantic mask and the target action semantic mask are consistent, α and ε represent hyperparameters, and X s+1 represents the inverted semantic mask, X s represents the synthetic semantic mask, X a represents a synthesized human image, and I represents a source human image.

[0030] Optionally, the semantic neural radiation field comprises: at least n1 hidden groups, each hidden group is connected to a SIREN layer,

[0031] The appearance neural radiation field includes: at least p1 hidden groups, each hidden group is connected to a SIREN layer, p1 is less than or equal to n1, and p1 and n1 are both natural numbers greater than or equal to 1;

[0032] Each hidden group consists of a linear layer and an activation layer connected in sequence.

[0033] Optionally, the method further comprises:

[0034] S00, constructing a NeRF decoupling generator G, and training the G using the semantic masks of the multi-view appearance and front view of the same human body in the training data set to obtain the trained G, wherein the trained G can fully decouple the appearance and shape of the multi-view of the human body;

[0035] Accordingly, for each of the multiple views of the same human body, the first mapping network inputs the shape latent code z of the set action semantic mask of the front view s ; Output is the first frequency intermediate feature γ s and the first phase shift intermediate characteristic β s ;

[0036] The input of the second mapping network is the appearance latent code z of the corresponding view a , the output is the second frequency intermediate feature γ a and the second phase shift intermediate characteristic β a ;

[0037] The input of the semantic neural radiation field is the coordinate X of the sampling point, and the output is the density σ;

[0038] The input of the appearance neural radiation field is the density σ and the viewing direction d of the sampling point, and the output is the color c;

[0039] The input of the semantic decoder is the density σ, and the output is the semantic category probability s;

[0040] The coordinate X of the sampling point is at least one coordinate point sampled when the camera to which the view to be input belongs emits a ray for each pixel of the view based on the viewing angle direction;

[0041] The viewing angle parameter of the camera to which the view to be input belongs is the viewing angle direction d of the sampling point.

[0042] Optionally, the method further comprises:

[0043] The step of training G using the semantic masks of the multi-view appearance and frontal view of the same person in the training dataset comprises:

[0044] Based on a pre-given training data set, a full-body frontal image of each human body in the training data set and a set action semantic mask of each part of the frontal image are obtained;

[0045] Obtain images of the same human body from different viewpoints from the training data set as a human body image group;

[0046] Obtaining the camera viewing angle parameters of each human body image in the human body image group; and optimizing the background to obtain the set action semantic mask of the same human body frontal image;

[0047] Get each human image in each group and the optimized set action semantic mask and latent code z s′ 、z a′ , sampling point X and viewing direction d, and input G to obtain the semantic category probability s, density σ and color c to be rendered;

[0048] The semantic category probability s and density σ are output as a synthetic semantic mask through a semantic rendering module;

[0049] The density σ and the color c are outputted into a synthetic human body image through a color rendering module;

[0050] Synthesized human image and synthetic semantic mask, original human image and set action semantic mask when input G, shape latent code z s′ and viewing direction d as the shape discriminator D s Input;

[0051] Synthesized human image, original human image when input G, appearance latent code z a and viewing direction d as appearance discriminator D a Input;

[0052] In the shape discriminator D s and appearance discriminator D a When the outputs of all reach the preset threshold, the trained G is obtained.

[0053] Optionally,

[0054]

[0055]

[0056] in, D is the appearance discriminator a The output, is the shape discriminator D s Output:

[0057] Denotes the summary of formulas subject to h, D a (x) and D s (x) represents the output of x after passing through the appearance discriminator and shape discriminator, and denote the R1 gradient loss of the appearance discriminator and shape discriminator respectively, ||x|| 2 represents the square of the absolute value of x, λ 1 ,λ 2 =0.2,λ p ,λ a ,λ s= 10. Where f(u) represents the binary classification loss, g(x,y) represents the mean square error loss, and λ 1 ,λ 2 denotes the weights of the latent mean square error loss in the appearance discriminator and the shape discriminator, respectively, and λ p represents the weight of the camera pose mean square error loss, λ a ,λ s Represent the weights of the R1 gradient in the appearance discriminator and the shape discriminator, respectively, and X a and X s denote the generated human image and semantic mask respectively, S and I denote the real semantic mask and real human image respectively, ξ denotes the sampled camera pose, ξ a and s They represent the camera poses corresponding to the appearance and shape during training, respectively, and p I and p s They represent the distribution of real human images sampled from the dataset during training and the semantic masks of real human images, respectively.

[0058] I~p I ,S~p s They respectively indicate that the real human image follows the normal distribution of the real human image training data set, and the real semantic mask follows the normal distribution of the real semantic mask training data set. and represent the appearance latent code and shape latent code respectively, and Respectively represent the latent codes corresponding to the appearance and shape during training, It indicates that the appearance latent code and shape latent code follow normal distribution.

[0059] Optionally, the semantic rendering module is specifically used to

[0060]

[0061]

[0062] Among them, r(t)=o+td, d represents the camera viewing direction, r(t) represents the distance of the light from the camera origin o at time t, c(r(t), d) represents the color value of the point r(t) from the camera origin o on the light line with viewing direction d, s(r(t), d) represents the semantic category probability of the point r(t) from the camera origin o on the light line with viewing direction d, σ(x) represents the probability density of termination at the position when the light passes through the position, Indicates that the light is emitted from the starting point t n To the launch end point t f Integrate the time t, T(t) represents the time from the light line c to the starting point tn To the end point of the light emission t f The cumulative transmittance of the image is obtained, thereby obtaining a two-dimensional synthetic human image and a synthetic semantic mask at any camera viewing angle.

[0063] In a second aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory and performs the steps of any method described in the first aspect.

[0064] In a third aspect, a computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for generating a human body motion migration image as described in any of the first aspects is implemented.

[0065] (III) Beneficial effects

[0066] The method of the present invention can obtain an image whose migration result meets the requirements with the help of the trained G, and adaptively adjust the parameters of G during use to ensure the quality of the migration result, thereby solving the problem of artifacts in generating human body images and interference between shape and appearance in the prior art.

[0067] During the training process, a method is adopted in which human body images from sparse perspectives and corresponding semantic maps of human body parts are used to train and decouple human body appearance and shape, and then semantic inversion is used to realize human body motion migration image. Compared with the prior art, this method can solve the technical problems that the prior art cannot eliminate the difficulty of appearance change during motion migration and the human body image synthesis relies on skeleton prior, and can directly eliminate the difficulty of appearance change during motion migration. At the same time, in the training process, the human body shape and appearance are fully decoupled, so that the shape and appearance of the human body do not interfere with each other during the migration process, thereby eliminating the artifacts of the generated image.

[0068] Now, in the process of training G, not only semantic masks are used to extract the layout of human body parts and eliminate the difficulty of appearance changes during motion transfer; at the same time, the shape and appearance of the human body are fully decoupled during the training process, so that the shape and appearance of the human body do not interfere with each other during the migration process, thereby eliminating the artifacts of the generated image; in use, semantic masks are used as a bridge to guide human motion migration, and the spatial information of the same human body from different perspectives in the dataset is fully utilized. There is no need to pair datasets for supervised training, which increases universality and improves the quality of generated results. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1A This is a flow chart of a method for generating a human motion migration image based on a neural radiation field according to the present invention;

[0070] Figure 1B It is the overall flow chart of the human body action migration image of the present invention;

[0071] Figure 2 It is the overall training block diagram of the NeRF decoupling generator G in the present invention;

[0072] Figure 3 It is a network structure diagram of the NeRF decoupling generator G of the present invention;

[0073] Figure 4 for Figure 3 The architecture diagram of the mapping network;

[0074] Figure 5 It is a flow chart of the use phase of the NeRF decoupling generator G of the present invention;

[0075] Figure 6 A comparison chart of the effects of the present invention on the human motion migration task and other existing methods. DETAILED DESCRIPTION

[0076] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0077] In order to generate high-quality results, industry insiders have introduced additional labels in some methods, such as human body part analysis maps (which can be semantic masks) to provide semantic guidance for processing motion changes. In other methods, a dual-branch framework including an action branch and an appearance branch is used, such as using dual-task correlation to capture reasonable texture mapping. The network used at this time introduces an auxiliary task (i.e., source image to source image task) through the Siamese architecture to facilitate the conversion from source human motion images to target human motion images. However, this method will produce serious motion misalignment and detail loss in the appearance part of the generated image when there is a significant difference between the source and target motions.

[0078] Some people have introduced additional semantic information, namely semantic masks, into human body images to extract the layout of human body parts, thereby achieving automatic separation of part attributes, which can ensure that the appearance of the human body retains as much detail information as possible during complex action migration. However, these techniques cannot fully decouple the shape and appearance information of the human body, and perform poorly in preserving spatial contextual relationships, resulting in visually unrealistic appearance details and varying degrees of artifacts in the generated images.

[0079] The above solutions all use paired data sets to train the model during the training process. In practice, it is difficult to collect paired data, and thus it is impossible to complete the generation of human motion transfer images well. The industry has proposed a self-supervised correction network, such as using feature rearrangement technology or generating pseudo labels to enhance the training process. However, it should be noted that these methods that do not use paired data sets to train models have limitations in generating high-quality results.

[0080] To this end, the embodiment of the present invention combines the sampling mechanism of NeRF with the existing 2D method to fully utilize the multi-view appearance data of the same human body in the data set, thereby providing obvious advantages for generating high-quality human images. Specifically, in the implementation, not only semantic masks are used to extract the layout of human body parts, eliminating the difficulty of appearance changes during action migration; at the same time, the shape and appearance of the human body are fully decoupled during the training process, so that the shape and appearance of the human body do not interfere with each other during the migration process, thereby eliminating the artifacts of the generated image; in use, semantic masks are used as a bridge to guide the migration of human body movements, and the spatial information of different perspectives of the same human body in the data set is fully utilized, without the need for paired data sets for supervised training, which increases universality and improves the quality of the generated results.

[0081] Embodiment 1

[0082] like Figure 1A As shown, the embodiment of the present invention provides a method for generating a human motion migration image based on a neural radiation field, and the specific implementation method thereof includes the following steps:

[0083] S10, for the source human action image to be analyzed and the target action semantic mask, obtaining a shape latent code of the target action semantic mask and an appearance latent code of the source human action image;

[0084] The source human action image and the target action semantic mask, shape latent code, and appearance latent code are input into the pre-trained NeRF decoupled generator G, and the semantic category probability s, density σ, and color C output by G to be used for rendering are obtained.

[0085] It is understandable that in this embodiment, obtaining the appearance latent code of the source human action image to be analyzed and the shape latent code of the target action semantic mask are both existing methods, and this embodiment can directly obtain and perform subsequent processing.

[0086] The shape latent code and appearance latent code of this embodiment both obey normal distribution.

[0087] S20. Obtain a synthetic semantic mask and a synthetic human body image by rendering according to the semantic category probability s, density σ and color c.

[0088] For example, all semantic category probabilities s and densities σ are used to generate synthetic semantic masks through the semantic rendering module, and colors c and densities σ are used to generate synthetic human images through the color rendering module.

[0089] The semantic rendering module and the color rendering module in this step both belong to the rendering methods of the prior art.

[0090] S30, according to the synthesized human body image and the target action semantic mask, with the help of G, the semantic category probability s and density σ to which each pixel of the inverted semantic mask belongs are obtained, and then the inverted semantic mask is obtained with the help of rendering.

[0091] For example, the appearance latent code of the synthetic human image and the shape latent code of the target action semantic mask can be obtained; the synthetic human image and the target action semantic mask, the shape latent code, and the appearance latent code are input into the G to obtain the semantic category probability s and density σ of each pixel point of the inverted semantic mask; and then the semantic category probability s and density σ at this location are rendered by means of a rendering method to obtain the inverted semantic mask.

[0092] S40: Determine whether it is necessary to adaptively adjust the parameters of G according to the synthetic semantic mask and the inverted semantic mask.

[0093] For example, according to formula (1), the loss summary information of the synthetic semantic mask and the inverted semantic mask is obtained: If loss summary information If it exceeds the preset range, the parameters of G are adaptively adjusted;

[0094]

[0095] in, represents the loss function that ensures that the synthesized human image is similar to the source human image, represents the loss function that ensures that the synthetic semantic mask and the target action semantic mask are consistent, α and ε represent hyperparameters, and X s+1 represents the inverted semantic mask, X s+1 represents the inverted semantic mask, X s represents the synthetic semantic mask, X a represents a synthesized human image, and I represents a source human image.

[0096] In this embodiment, the weight parameters and bias parameters of G may be adaptively adjusted, for example, by adjusting the weight parameters and / or bias parameters according to a preset step size, and then adjusting the weight parameters and bias parameters to a desired interval.

[0097] S50. If it is necessary to adjust the parameters of G, then the parameters of G are adjusted based on the adaptive adjustment, and the synthesized human body image is re-output based on the G with the adjusted parameters, until the synthesized human body image output when G does not need to be adaptively adjusted is used as the final output human body motion migration image.

[0098] It should be noted here that the structure of G in the training phase and the usage phase is the same. The training phase needs to train the parameters of G, and the usage phase can use G based on the one obtained in the training phase. At this time, there is no need to obtain parameters such as the sampling point coordinates and camera viewing angle mentioned in the following embodiment 2, that is, the specific semantic neural radiation field can have output without direct input.

[0099] like Figure 3 As shown, G includes: a first mapping network based on a multi-layer perceptron, a second mapping network based on a multi-layer perceptron, a semantic neural radiation field, an appearance neural radiation field, and a semantic decoder;

[0100] The first mapping network input is a latent code of shape z s ; Output is the first frequency intermediate feature γ s and the first phase shift intermediate characteristic β s ;

[0101] The input of the second mapping network is the appearance latent code z a , the output is the second frequency intermediate feature γ a and the second phase shift intermediate characteristic β a ;

[0102] The output of the semantic neural radiation field is density σ;

[0103] The input of the appearance neural radiation field is the density σ, and the output is the color c;

[0104] The input of the semantic decoder is the density σ, and the output is the semantic category probability s;

[0105] Among them, the shape latent code z s The latent code z is generated by randomly sampling the semantic mask of the target action and follows the normal distribution. a The latent codes are generated by randomly sampling the source human action images and follow the normal distribution.

[0106] The above method can obtain images whose migration results meet the requirements with the help of the trained G, and adaptively adjust the parameters of G during use to ensure the quality of the migration results, solving the problems of artifacts in generating human body images and interference between shape and appearance in the prior art.

[0107] Embodiment 2

[0108] like Figures 1B to 5 As shown, the method for generating human motion migration images based on neural radiation fields includes the following steps:

[0109] Step 1: Obtain full-body frontal images and corresponding semantic masks of human body parts as training and testing data sets.

[0110] The task design of this embodiment is semantically guided human action transfer, so a large dataset containing full-body images is required to ensure the correct learning of body parts and the full decoupling of appearance and shape. This embodiment selects about 20,000 full-body images from the DeepFashion dataset and divides them into two parts: training set and test set.

[0111] Since the dataset contains some images of the same person from different perspectives, in order to obtain the camera perspective parameters of these images, the OpenPose online model can be used to obtain the skeleton key point information of the human body images in the DeepFashion dataset, and then the front skeleton key points are used as reference points. The position information of the selected reference points is combined with the internal parameters of the camera to calculate the camera position of these reference points in other perspectives. Finally, the PnP (Perspective-n-Point) algorithm (which can solve the position and direction of the camera based on the known three-dimensional points and their projection points on the image) is used to solve the camera perspective parameters of human body images from different perspectives.

[0112] Each image in the large data set is filled with a white background into a square and the size is cropped to 512×512. For example, the best online model in the field of human semantic parsing is used to segment the human image into 15 categories of segmented images, including background, head, upper body, left upper arm, left forearm, left hand, right upper arm, right forearm, right hand, left thigh, left calf, left foot, right thigh, right calf, and right foot. Then each pixel in the segmented image is associated with the corresponding category label, such as: background area pixels are represented by 0, head area pixels are represented by 1, and so on, so as to form a semantic mask with the same size as the original image. Here it is not limited to 15 categories, it can also be 10 or 20, according to specific needs. The more categories parsed, the more accurate the G processing of subsequent training will be.

[0113] In practical applications, the images used for training in the training set can be screened so that the semantic masks of each image and segmented category in the training set are also clear and meet the quality requirements.

[0114] Figure 2 A schematic diagram of the training phase, such as Figure 2 The overall training block diagram of the NeRF decoupled generator in this embodiment is shown, which includes the input of the generator: shape latent code z s 、Appearance potential code a , a semantic mask of the set action for each image and frontal view in a set of human images;

[0115] The output of the NeRF decoupled generator (Generator, G for short) is then input into the shape discriminator and appearance discriminator through the neural rendering module;

[0116] Among them, the neural rendering module includes: semantic neural radiation field, appearance neural radiation field and volume rendering module.

[0117] The following steps 2 to 4 can be understood as the training process of G.

[0118] Step 2: Take a group of human images in the training data set and the semantic masks corresponding to the group of human images as training data, and output the semantic category probability, density and color through G.

[0119] Usually, the semantic mask corresponding to the group of human body images is the set action semantic mask of the front view of the group of human body images.

[0120] Sub-step 21: Randomly sample two latent codes z that follow a normal distribution based on each image of a group of human images and the semantic mask corresponding to the group of human images s′ 、z a′ ; The z s′ 、z a′ They are used to generate shape and appearance respectively;

[0121] Sub-step 22: Obtain the coordinates of the sampling points of each pixel in each image in a set of human body images and the viewing direction d of each image.

[0122] In this embodiment, the camera viewing angle parameters of each image acquired in step 1 are used as the viewing angle direction d.

[0123] In addition, the camera emits a ray to each pixel of the training image based on the viewing direction, and samples a batch of points on these rays to obtain the 3D spatial coordinates of each sampling point.

[0124] Sub-step 23: Take each image in a group of human images, the semantic mask, and the latent code z corresponding to each image and the semantic mask of the group of human images s′ 、z a′ , sampling point coordinates X = (x, y, z), viewing direction As the input of G, it outputs the semantic category probability s, density σ and color c of the sampling point.

[0125] like Figure 3 The schematic diagram of the network structure of G shown in FIG. 1 includes two networks with the same structure and inputs z and z respectively. s′ 、z a′ The first mapping network and the second mapping network,

[0126] A semantic neural radiation field with sampling point coordinates X as input,

[0127] A semantic decoder,

[0128] An appearance neural radiation field.

[0129] Among them, the first mapping network and the second mapping network are parameterized as MLP (multi-layer perceptron), such as Figure 4 As shown in the figure, it has 4 hidden layers connected by linear layers and activation layers and a linear layer for outputting γ and β. The semantic neural radiation field, which is also parameterized as an MLP, has 8 hidden layers connected by linear layers and activation layers, and each hidden layer contains a SIREN layer.

[0130] exist Figure 3 In the SIREN layer, “*” indicates multiplication, “+” indicates addition, and “~” indicates sine calculation.

[0131] It can be understood that the semantic decoder of this embodiment includes 4 hidden layers connected by linear layers and activation layers.

[0132] The appearance neural radiance field consists of three hidden layers connected by linear layers and activation layers, and each hidden layer contains a SIREN layer.

[0133] The SIREN layer structure in this embodiment can be expressed by the following formula:

[0134] φ i (x i )=sin(γ i ·(W i x i +b i )+β i )

[0135] Among them, φ i represents the i-th SIREN layer in the semantic neural radiance field and the appearance neural radiance field, γ i , β i , W i 、b i 、x i They represent the frequency, phase shift, weight matrix, bias parameter and network layer input of the i-th layer respectively.

[0136] In this embodiment, the Figure 3 The decoupling process is further explained.

[0137] First: transform the shape and appearance of the latent code z s 、z a As the input of the first mapping network and the second mapping network respectively, the output of each mapping network is used to adjust the frequency γ and phase shift β of a specific layer in the neural radiation field.

[0138] The sampling point X is used as the input of the semantic neural radiation field, and z s γ obtained by the first mapping network s , β s As an intermediate feature, it is input to the SIREN layer of the semantic neural radiation field. The semantic neural radiation field outputs the density σ, and the density σ is input to the semantic decoder to obtain the semantic category probability s.

[0139] The output of the semantic neural radiance field and the viewing direction d are fed into the appearance neural radiance field which is parameterized as an MLP. a γ obtained by the second mapping network a , β a As an intermediate feature, it is input to the SIREN layer of the appearance neural radiance field. The appearance neural radiance field finally outputs the color c.

[0140] Compared with the existing model, the network structure of G in this embodiment has the following advantages:

[0141] (1) In order to decouple the appearance and shape of human images under the supervision of human semantic masks, G has an additional branch for outputting semantic category probability s, based on the existing method that only outputs density σ and color c;

[0142] (2) For the first time, the SIREN layer is selected as the feature representation output layer in G.

[0143] Step 3: Render the generated semantic category probabilities, densities, and colors from arbitrary camera poses into human images and semantic masks via volume rendering.

[0144] Semantic category probabilities, densities, and colors are generated by G, and volume rendering is used to render them into semantic masks and human images from arbitrary camera viewpoints.

[0145] For each sampling point coordinate in the space, first query the semantic category probability s, density σ and color c of each point. In order to obtain the color C of each pixel in the human image r and the category label S of each pixel in the semantic mask r , the sample values ​​of all sampling points on the ray are accumulated using classic volume rendering, and the formula is as follows:

[0146]

[0147] Among them, r(t)=o+td, r(t) represents the distance of the light from the camera origin o at time t, and T(t) represents the distance of the light along t n (light emission starting point) to t f The cumulative transmittance of (the end point of light emission). Thus, a two-dimensional synthetic human image and synthetic semantic mask under any camera viewing angle are obtained.

[0148] In the formula of volume rendering, C(r) refers to the color value of the pixel obtained by integrating the colors of multiple sampling points on a ray. S(r) refers to the semantic mask.

[0149] It is to be noted here that the volume rendering in this embodiment is an existing method, which is only used and not improved in this embodiment.

[0150] In addition, it should be noted that: the existing method of obtaining light sampling points is: for each pixel on the image, the sampling points on the light are evenly distributed and the intervals are consistent. The present invention proposes a semantic mirror sampling adjustment strategy in the light sampling link: Step 1 has obtained the semantic mask corresponding to each group of human images in the data set, and the semantic mask contains the category labels corresponding to different parts of the human body, such as background area pixels are represented by 0, and head area pixels are represented by 1; at this time, the human body area (i.e., the area outside the background) pixels of the human body image can be locked according to the non-zero category label, and the sampling point interval on the light received by all human body area pixels is set to the normalized value of the category label of the area pixels corresponding to the light, such as the sampling point interval on the light received by the head area pixels is 1 / 15 (because there are 15 category labels in total). Therefore, r(t)=o+Norm(M)td in the volume rendering formula, where M represents the category label.

[0151] Step 4: Through the discriminator D s and the discriminator D a The synthetic human image and synthetic semantic mask rendered in step 3 are optimized to complete the entire training process.

[0152] The rendered synthetic human image and synthetic semantic mask, each image and semantic mask of a set of human images input to G during training, and the shape latent code z s′ and viewing direction d as the shape discriminator D s Input;

[0153] The rendered synthetic human image, each image of a set of human images input to G during training, and the appearance latent code z a′ and viewing direction d as appearance discriminator D a During the training process, the losses of the two discriminators are as follows:

[0154]

[0155]

[0156] where f(u) = -log(1+exp(-u)), λ 1 ,λ 2=0.2,λ p ,λ a ,λ s = 10. Where f(u) represents the binary classification loss, g(x,y) represents the mean square error loss, and D a (x) and D S (x) represents the output of x after passing through the appearance discriminator and shape discriminator, λ 1 ,λ 2 denotes the weights of the latent mean square error loss in the appearance discriminator and the shape discriminator, respectively, and λ p represents the weight of the camera pose mean square error loss, λ a ,λ s Represent the weights of the R1 gradient in the appearance discriminator and the shape discriminator, respectively, and X a and X s denote the generated human image and semantic mask respectively, S and I denote the real semantic mask and real human image respectively, ξ denotes the sampled camera pose, ξ a and s Respectively represent the camera poses corresponding to the appearance and shape during training, and They represent the latent codes corresponding to the appearance and shape during training, respectively, and p I and p s Respectively represent the distribution of real human images and semantic masks sampled from the dataset during training. We use non-saturated GAN loss and R1 gradient to optimize the discriminator training, as shown in the last term of the loss formula. In this embodiment, the discriminator not only calculates the loss between the generated image and the real image, but also calculates the latent space loss of the generated image, which makes the generated image closer to the real image in the latent space. On this basis, the gradient loss is applied to promote the gradient smoothing of the discriminator, thereby stabilizing the training of the GAN framework. The sampled camera pose is passed to the discriminator for prediction accuracy. Finally, the MSE (mean square error) between the predicted value and the input value is used as a loss to penalize the inconsistency of the camera pose.

[0157] Step 5: Use the pre-trained G to perform human action migration on a random source human image according to the action of the target action human semantic mask by using the semantic inversion method, and output the final human migration image.

[0158] like Figure 5 The flowchart of semantic inversion human motion transfer image generation in this embodiment is shown, which includes a human image to be analyzed, a target motion human semantic mask, G, a synthesized human image, a synthesized semantic mask, and an inverted semantic mask.

[0159] In order to guide the action of a given human image to remain consistent with the semantic mask of the target action human body, this embodiment adopts a semantic inversion method.

[0160] Combination Figure 5 The process of generating human motion transfer images by semantic inversion is described in detail.

[0161] Sub-step 51: Input a random source human image (random means that the human image is an image in the test set or any full-body frontal human image found on the Internet) and a random target action human semantic mask (random means that the semantic mask is a semantic mask included in the test set or a semantic mask obtained by processing a human image found on the Internet using an online model) into G, and G outputs the semantic category probability s, density σ and color c.

[0162] Among them: semantic category probability and density are used to generate synthetic semantic masks through semantic rendering, and color and density are used to generate synthetic human images through color rendering.

[0163] Sub-step 52: Input the synthesized human image and the target action human semantic mask into G to generate new semantic category probabilities and densities, and the new semantic category probabilities and densities are used to generate inverted semantic masks through semantic rendering.

[0164] Sub-step 53: Calculate the loss between the generated inverted semantic mask and the synthesized semantic mask; Calculate the loss between the synthesized human image and the source human image.

[0165] When the loss meets expectations, the final synthesized human image is the desired human motion transfer image.

[0166] The total loss objective function of semantic reversal is expressed as follows:

[0167]

[0168] in, represents the loss function that ensures that the appearance of the synthesized human image is similar to that of the source human image, represents the loss function that ensures that the shape of the synthetic semantic mask is consistent with the shape of the target action semantic mask, α and ε represent hyperparameters, and X s+1 represents the inverted semantic mask, X s represents the synthetic semantic mask, X a represents a synthesized human image, and I represents a source human image.

[0169] During the training process, this embodiment adopts the method of training and decoupling the appearance and shape of the human body from the human body images of sparse perspectives and the corresponding semantic graphs of human body parts, and then uses the semantic inversion method to realize the method of human body motion migration image. Compared with the prior art, this embodiment can solve the technical problems that the prior art cannot eliminate the difficulty of appearance change during motion migration and the human body image synthesis relies on the skeleton prior, so as to directly eliminate the difficulty of appearance change during motion migration. At the same time, in the training process, the shape and appearance of the human body are fully decoupled, so that the shape and appearance of the human body do not interfere with each other during the migration process, thereby eliminating the artifacts of the generated image.

[0170] Now, in the process of training G, not only semantic masks are used to extract the layout of human body parts and eliminate the difficulty of appearance changes during motion transfer; at the same time, the shape and appearance of the human body are fully decoupled during the training process, so that the shape and appearance of the human body do not interfere with each other during the migration process, thereby eliminating the artifacts of the generated image; in use, semantic masks are used as a bridge to guide human motion migration, and the spatial information of the same human body from different perspectives in the dataset is fully utilized. There is no need to pair datasets for supervised training, which increases universality and improves the quality of generated results.

[0171] Verification Example

[0172] Figure 6 The comparison between the effects of the present invention and other existing methods on human motion transfer is shown. Figure 6 The last column of the left and right columns is a human action migration image generated by the method of the present invention using the human image to be analyzed and the target action semantic mask as the input of G and adopting the semantic inversion method.

[0173] Figure 6 The visual analysis in clearly illustrates the superior performance of the method of the present invention in preserving the appearance features of the human body images to be analyzed.

[0174] In particular, if Figure 6 In the fifth row, the method of the present invention shows better performance in accurately restoring the complex details of women's arms and tops; in the first row, the method of the present invention is more reasonable in generating men's jackets and linings. It is worth noting that, except for the method of the present invention and MUST-GAN, other methods need to use source-target action human images as paired training data for training. The present invention decouples the appearance and shape of the human body, and transfers the action through an inversion method guided by the semantic mask of the target action human body, which determines that the training process of G in the present invention does not require source-target action paired human images as supervision for training.

[0175] Therefore, the method for generating human motion transfer images of the present invention adopts training and decoupling of human appearance and shape from human body images of sparse perspectives and corresponding semantic graphs of human body parts, and then uses a semantic inversion method to realize the method of human motion transfer images. This solves the problem that the prior art cannot eliminate the difficulty of appearance changes during motion transfer and the problem that human body image synthesis relies on skeleton priors, and realizes the direct elimination of the difficulty of appearance changes during motion transfer. At the same time, it fully decouples the shape and appearance of the human body during the training process, so that the shape and appearance of the human body do not interfere with each other during the migration process, thereby eliminating artifacts in the generated image.

[0176] Embodiment 3

[0177] In addition, an embodiment of the present invention further provides an electronic device, characterized in that it includes a memory and a processor, the memory stores a computer program, the processor executes the computer program stored in the memory, and performs the steps of the method described in any of the above embodiments.

[0178] On the other hand, an embodiment of the present invention further provides a computer program product, comprising the steps of the methods described in the above embodiments.

[0179] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0180] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0181] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0182] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0183] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for generating human motion transfer images based on neural radiation fields, characterized in that: include: S10, obtaining an appearance latent code and a shape latent code for the source human action image to be analyzed and the target action semantic mask; Input the source human action image and the target action semantic mask, appearance latent code, and shape latent code into a pre-trained NeRF decoupling generator G, and obtain the semantic category probability s, density σ, and color c output by G to be used for rendering; S20, obtaining a synthetic semantic mask and a synthetic human body image by means of a rendering method according to the semantic category probability s, density σ and color c; S30, according to the synthesized human body image and the target action semantic mask, using G to obtain the semantic category probability s and density σ to which each pixel of the inverted semantic mask belongs, and obtain the inverted semantic mask; S40, determining whether it is necessary to adaptively adjust the parameters of G according to the synthetic semantic mask and the inverted semantic mask; S50, if it is necessary to adjust the parameters of G, then based on the adaptive adjustment of the parameters of G, and based on the G with the adjusted parameters, re-outputting the synthesized human body image, until the synthesized human body image output when G does not need to be adaptively adjusted is used as the final output human action migration image; The G includes: a first mapping network based on a multi-layer perceptron, a second mapping network based on a multi-layer perceptron, a semantic neural radiation field, an appearance neural radiation field, and a semantic decoder; Shape latent code z s The latent code z is generated by randomly sampling the semantic mask of the target action and follows the normal distribution. a The latent codes that obey the normal distribution are generated by randomly sampling the source human action images; The method further comprises: S00, constructing a NeRF decoupling generator G, and training the G using the semantic masks of the multi-view appearance and front view of the same human body in the training data set to obtain the trained G, wherein the trained G can fully decouple the appearance and shape of the multi-view of the human body; Accordingly, for each view in the multiple views of the same human body, the first mapping network input is the shape latent code z of the set action semantic mask of the front view in the multiple views. s ; Output is the first frequency intermediate feature γ s and the first phase shift intermediate characteristic β s ; The input of the second mapping network is the appearance latent code z of the corresponding view a , the output is the second frequency intermediate feature γ a and the second phase shift intermediate characteristic β a ; The input of the semantic neural radiation field is the coordinate X of the sampling point, and the output is the density σ; The input of the appearance neural radiation field is the density σ and the viewing direction d of the sampling point, and the output is the color c; The input of the semantic decoder is the density σ, and the output is the semantic category probability s; The coordinate X of the sampling point is at least one coordinate point sampled when the camera to which the view to be input belongs emits a ray for each pixel of the view based on the viewing angle direction; The viewing angle parameter of the camera to which the input view belongs is the viewing angle direction d of the sampling point; The S30 includes: obtaining the appearance latent code and shape latent code of the synthesized human image and the target action semantic mask; inputting the synthesized human image and the target action semantic mask, the shape latent code, and the appearance latent code into the G, obtaining the semantic category probability s and density σ to which each pixel point of the inverted semantic mask belongs, and obtaining the inverted semantic mask through semantic rendering.

2. The method according to claim 1, characterized in that: The semantic neural radiation field includes: at least n1 hidden groups, each hidden group is connected to a SIREN layer, The appearance neural radiation field includes: at least p1 hidden groups, each hidden group is connected to a SIREN layer, p1 is less than or equal to n1, and p1 and n1 are both natural numbers greater than or equal to 1; Each hidden group consists of a linear layer and an activation layer connected in sequence.

3. The method according to claim 1, characterized in that The method further comprises: The step of training G using the semantic masks of the multi-view appearance and frontal view of the same person in the training dataset comprises: Based on a pre-given training data set, a full-body frontal image of each human body in the training data set and a set action semantic mask of each part of the frontal image are obtained; Obtain images of the same human body from different viewpoints from the training data set as a human body image group; Obtaining the camera viewing angle parameters of each human body image in the human body image group; and optimizing the background to obtain the set action semantic mask of the frontal image of the same human body; Get each human image in each group and the optimized set action semantic mask and latent code z s′ 、z a′ , sampling point X and viewing direction d, and input G to obtain the semantic category probability s, density σ and color c to be rendered; The semantic category probability s and density σ are output as a synthetic semantic mask through a semantic rendering module; The density σ and the color c are outputted into a synthetic human body image through a color rendering module; Synthesized human image and synthetic semantic mask, original human image and optimized set action semantic mask when input G, latent code z s′ and viewing direction d as the shape discriminator D s Input; Synthesized human image, original human image when input G, appearance latent code z a and viewing direction d as appearance discriminator D a Input; In the shape discriminator D s and appearance discriminator D a When the outputs of all reach the preset threshold, the trained G is obtained.

4. The method according to claim 3, characterized in that in, D is the appearance discriminator a The output, is the shape discriminator D s Output: Denotes the summary of formulas subject to h, D a (x) and D s (x) represents the output of x after passing through the appearance discriminator and shape discriminator, and Represents the R1 gradient loss of the appearance discriminator and the shape discriminator, ∥x∥ 2 represents the square of the absolute value of x, λ1,λ2=0.2,λ p ,λ a ,λ s = 10; where f(u) represents the binary classification loss, g(x,y) represents the mean square error loss, λ1 and λ2 represent the weights of the latent mean square error loss in the appearance discriminator and shape discriminator, respectively, and λ p represents the weight of the camera pose mean square error loss, λ a ,λ s Represent the weights of the R1 gradient in the appearance discriminator and the shape discriminator, respectively, and X a and X s denote the generated human image and semantic mask respectively, S and I denote the real semantic mask and real human image respectively, ξ denotes the sampled camera pose, ξ a and s They represent the camera poses corresponding to the appearance and shape during training, respectively, and p I and p s They represent the distribution of real human images and semantic masks of real human images sampled from the dataset during training respectively; I~p I ,S~p s They respectively indicate that the real human image follows the normal distribution of the real human image training data set, the real semantic mask follows the normal distribution of the real semantic mask training data set, and z a and z s denote the appearance latent code and shape latent code respectively, z ap and z sp Respectively represent the latent codes corresponding to the appearance and shape during training, It indicates that the appearance latent code and shape latent code follow normal distribution.

5. The method according to claim 3, characterized in that: The semantic rendering module is specifically used for: Among them, r(t)=o+td, d represents the camera viewing direction, r(t) represents the distance of the light from the camera origin o at time t, c(r(t), d) represents the color value of the point r(t) from the camera origin o on the light line with viewing direction d, s(r(t), d) represents the semantic category probability of the point r(t) from the camera origin o on the light line with viewing direction d, σ(x) represents the probability density of termination at the position when the light passes through the position, Indicates that the light is emitted from the starting point t n To the launch end point t f Integrate the time t, T(t) represents the time from the light line c to the starting point t n To the end point of the light emission t f The cumulative transmittance of the image is obtained; thus, a two-dimensional synthetic human image and a synthetic semantic mask under any camera viewing angle are obtained, and C(r) represents the color value of the pixel.

6. The method according to claim 1, characterized in that S20 includes: the semantic category probability s and the density σ are rendered into a synthetic semantic mask by a semantic rendering module, and the color c and the density σ are rendered into a synthetic human image by a color rendering module; Alternatively, the S40 includes: obtaining the loss summary information of the synthetic semantic mask and the inverted semantic mask according to formula (1): If loss summary information If it exceeds the preset range, the parameters of G are adaptively adjusted; in, represents the loss function that ensures that the synthesized human image is similar to the source human image, represents the loss function that ensures that the synthetic semantic mask and the target action semantic mask are consistent, α and ε represent hyperparameters, and X s+1 represents the inverted semantic mask, X s represents the synthetic semantic mask, X a represents a synthesized human image, and I represents a source human image.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory and performs the steps of any one of the methods described in claims 1 to 6.

8. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the method for generating a human body motion migration image as described in any one of claims 1 to 6.

Images