3D Model Motion Transfer Method

Abstract

The present invention provides a three-dimensional model motion migration method, which is characterized by: step S1, generating an action network model, a posture network model and an action posture model migrated to a new action according to pre-acquired three-dimensional model data as real values ​​for training an action migration neural network; step S2, constructing a feature encoding network as an action feature extractor to extract action features of an action three-dimensional mesh model; step S3, constructing a feature migration network as a decoder to perform a migration operation on the posture three-dimensional mesh model based on the extracted action features, thereby obtaining a deformed action posture model; step S4, constructing an end-to-end action migration neural network framework; step S5, constructing a loss function; step S6, training the action migration neural network based on the loss function, wherein the action migration neural network includes an action feature encoding network and a feature migration neural network based on a spatial adaptive instance normalization method.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional computer vision, and specifically relates to a method for migrating the actions of a three-dimensional model using deep learning. Background Art

[0002] Deformation transfer technology has been attracting attention in recent years. It is an important technology in the field of computer graphics and computer vision. For example, in games or movies, one can easily transfer the motion of one 3D model to another 3D model. However, it is very challenging when the motion transfer occurs between two 3D models with very different shapes.

[0003] In recent years, related research work has attempted to solve such problems. In order to establish the connection between two 3D models, previous methods often require additional information, such as point correspondence, additional 3D models, information about points on manually labeled models, and cages outside the 3D model to assist deformation.

[0004] In the prior art, deformation transfer can be performed using methods such as affine transformation and optimization, but this method requires an additional three-dimensional model to calculate the transfer of motion, and requires manual annotation by humans, which makes this method lack of usability. It is also possible to generate a three-dimensional cage outside the three-dimensional model to assist in deformation. This method also requires an additional three-dimensional model and manual annotation. Lack of usability. A VC-GAN deep neural network method can also be designed to achieve motion transfer, but this method lacks adaptability to posture models that do not appear in the training set. Whenever a new posture is encountered, sufficient training data is required. If it is missing, effective motion transfer cannot be achieved. Summary of the invention

[0005] In order to solve the above problems, a 3D model motion migration method is provided, which is used to migrate the motion of the motion 3D mesh model to the body posture 3D mesh model, so as to obtain the motion body posture 3D mesh model with new motion and maintaining the original body posture. The present invention adopts the following technical solutions:

[0006] The present invention provides a three-dimensional model action migration method, which is characterized by: step S1, generating an action network model, a posture network model and an action posture model migrated to a new action and maintaining the original posture according to pre-acquired three-dimensional model data as real values ​​for training an action migration neural network; step S2, constructing a feature encoding network as an action feature extractor to extract action features of an action three-dimensional mesh model; step S3, constructing a feature migration neural network as a decoder to perform a migration operation on the posture three-dimensional mesh model based on the extracted action features, thereby obtaining a deformed action posture model; step S4, constructing an end-to-end action migration neural network framework, that is, given a posture three-dimensional mesh model and an action three-dimensional mesh model, obtaining action features from the action three-dimensional mesh model through an action feature encoding network, migrating the action to the posture three-dimensional mesh model through a decoder based on a spatial adaptive instance normalization method, and outputting a three-dimensional mesh model after the posture model is converted into an action; step S5, constructing a loss function; step S6, training the action migration neural network based on the loss function, wherein the action migration neural network includes an action feature encoding network and a feature migration neural network based on a spatial adaptive instance normalization method.

[0007] The three-dimensional model motion transfer method provided by the present invention is characterized in that the specific process of step S1 is as follows: using the SMPL model to generate three-dimensional model data, controlling the action parameters in the SMPL model to generate three-dimensional mesh models of different actions as the action mesh model, controlling the posture parameters in the SMPL model to generate three-dimensional mesh models of different postures as the posture three-dimensional mesh model, and using the action parameters and posture parameters of the SMPL model to generate a posture model after the conversion action as the corresponding action posture model required for training the action transfer neural network as the true value.

[0008] The three-dimensional model motion migration method provided by the present invention may also have the following characteristics, wherein the feature encoding network is a three-layer parameter-sharing convolutional network, each convolutional layer of the convolutional network uses an instance normalization layer and a rectified linear unit ReLU as an activation function, the input of the feature encoding network is the motion mesh model, and the output is a matrix of encoded motion features.

[0009] The three-dimensional model motion migration method provided by the present invention may also have the following characteristics, wherein the feature migration neural network is a four-layer convolutional network, the first three convolutional layers of the convolutional network are each followed by a spatially adaptive instance standardized residual module, each spatially adaptive instance standardized residual module follows the structure of the residual network module, the residual network module has three units, each unit includes a spatially adaptive instance standardized structure, a convolutional network and an activation function, each spatially adaptive instance standardized structure is a body posture three-dimensional mesh model respectively obtained through a layer of convolutional layer The weight and offset required for standardization are obtained, the weight is multiplied by the input feature and the offset is added to the output, and the output action body posture three-dimensional mesh model is further obtained through a layer of convolutional layer and a tanh activation function.

[0010] The three-dimensional model motion migration method provided by the present invention is characterized in that the specific process of step S1 is as follows: the two-norm Euclidean distance is used as the loss function to regress the spatial coordinates of each vertex in the mesh model to make it close to the real coordinates, and the edge length regularization is used to make all edges of the migrated action posture three-dimensional mesh model as short and uniform as possible, thereby making the surface smoother.

[0011] The three-dimensional model motion migration method provided by the present invention is characterized in that the specific process of step S1 is as follows: the grid model action grid three-dimensional model and the grid model posture three-dimensional grid model are used as inputs of the grid model action migration neural network to obtain the grid model action posture three-dimensional grid model with new action output by the action migration neural network, the loss value is calculated using the grid model loss function according to the grid model action posture three-dimensional grid model obtained through the grid model action migration neural network and the grid model action posture three-dimensional grid model used as the true value in step S1, and the grid model action migration neural network is trained according to the grid model loss value using the back propagation algorithm and the gradient descent algorithm.

[0012] Function and Effect of the Invention

[0013] According to the three-dimensional model motion migration method of the present invention, for an action three-dimensional mesh model that provides action, a multi-layer parameter-sharing convolutional network is designed to extract the motion features of the action three-dimensional mesh model; for a posture three-dimensional mesh model that provides posture, a decoder that uses spatial adaptive instance standardization is designed to migrate the posture three-dimensional mesh model based on the extracted motion features; under this framework, an end-to-end motion migration neural network is constructed, which can migrate the action of the action three-dimensional mesh model to the posture three-dimensional mesh model for a given posture three-dimensional mesh model and action three-dimensional mesh model, and output the corresponding action posture three-dimensional mesh model. This method does not require any additional information assistance, has a good generation effect for various posture models, and has strong robustness to the disruption of the point arrangement order and noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of a three-dimensional model action migration method according to an embodiment of the present invention;

[0015] Figure 2 is a schematic diagram of a system structure corresponding to the three-dimensional model action migration method according to an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of the structure of an action feature extractor according to an embodiment of the present invention;

[0017] Figure 4 is a schematic diagram of the structure of a spatially adaptable instance normalization module (SPAdaIN) according to an embodiment of the present invention;

[0018] Figure 5 4 is a schematic diagram of the structure of a spatially adaptive instance normalized residual module (SPAdaIN ResBlk) according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The specific implementation of the present invention is described below with reference to the accompanying drawings and embodiments.

[0020] <Example>

[0021] Figure 1 It is a flow chart of a 3D model action migration method according to an embodiment of the present invention. Figure 2 It is a schematic diagram of the system structure corresponding to the three-dimensional model action migration method according to an embodiment of the present invention.

[0022] like Figure 1 and Figure 2 As shown, this embodiment provides a 3D model action migration method for transferring the action of an action 3D mesh model to a posture 3D mesh model, thereby obtaining an action posture 3D mesh model, comprising the following steps:

[0023] Step S1, generating an action network model, a posture network model and an action posture model transferred to a new action as real values ​​for training an action transfer neural network according to pre-acquired three-dimensional model data.

[0024] The motion transfer neural network includes a motion feature encoding network and a feature transfer neural network based on a spatially adaptive instance normalization method.

[0025] In this embodiment, the SMPL model is used to generate three-dimensional model data. The three-dimensional mesh models of different actions are generated by controlling the action parameters in the SMPL model as the action mesh model, and the three-dimensional mesh models of different postures are generated by controlling the posture parameters in the SMPL model as the posture three-dimensional mesh model. The action parameters and posture parameters of the SMPL model are used to generate the posture model after the conversion action as the corresponding action posture model required for training the action transfer neural network as the true value.

[0026] Step S2, constructing a feature encoding network as an action feature extractor to extract action features of the action three-dimensional mesh model.

[0027] Figure 3 It is a structural diagram of an action feature extractor according to an embodiment of the present invention.

[0028] like Figure 3 As shown in the figure, the feature encoding network is a three-layer parameter-sharing convolutional network. After each convolutional layer of the convolutional network, an instance normalization layer and a rectified linear unit ReLU are used as activation functions.

[0029] The input of the feature encoding network is the action mesh model, and the output is a matrix of encoded action features.

[0030] Step S3, constructing a feature transfer neural network as a decoder to perform a transfer operation on the body posture three-dimensional mesh model based on the extracted action features, thereby obtaining a deformed action posture model.

[0031] Figure 4 It is a schematic diagram of the structure of a spatially adaptive instance normalization module (SPAdaIN) according to an embodiment of the present invention. Figure 5 4 is a schematic diagram of the structure of a spatially adaptive instance normalized residual module (SPAdaIN ResBlk) according to an embodiment of the present invention.

[0032] like Figure 4 and Figure 5 As shown in FIG. 1 , the feature transfer neural network is a four-layer convolutional network, and each of the first three convolutional layers of the convolutional network is followed by a spatially adaptive instance normalized residual module (SPAdaIN ResBlk).

[0033] Each spatially adaptive instance normalization residual module (SPAdaIN ResBlk) has three units, each of which includes a spatially adaptive instance normalization structure, a convolutional network, and an activation function.

[0034] The standardized structure of each spatial adaptability instance is that the posture 3D mesh model obtains the weights and offsets required for standardization through a layer of convolution layer, multiplies the weights with the input features and adds the offsets before outputting, and further passes through a layer of convolution layer and tanh activation function to obtain the output action posture 3D mesh model.

[0035] Step S4, constructing an end-to-end motion transfer neural network framework, that is, given a body 3D mesh model and a motion 3D mesh model, obtaining motion features of the motion 3D mesh model through a motion feature encoding network, migrating the motion to the body 3D mesh model through a decoder based on a spatial adaptive instance normalization method, and outputting a 3D mesh model after the body model is converted into motion.

[0036] Step S5, construct the loss function, the specific process is as follows:

[0037] The two-norm Euclidean distance (L2 loss) is used as the loss function to regress the spatial coordinates of each vertex in the mesh model to make it close to the real coordinates.

[0038] Edge length regularization (Edge loss function) is used to make all edges of the migrated action posture 3D mesh model as short and uniform as possible, making the surface smoother.

[0039] Step S6, training the action transfer neural network based on the loss function,

[0040] The grid model action grid 3D model and the grid model posture 3D grid model are used as inputs of the grid model action transfer neural network, and the grid model action posture 3D grid model with new action output by the action transfer neural network is obtained.

[0041] The loss value is calculated using the grid model loss function based on the grid model action posture three-dimensional grid model obtained through the grid model action transfer neural network and the grid model action posture three-dimensional grid model used as the true value in step S1.

[0042] The grid model action transfer neural network is trained using the back propagation algorithm and the gradient descent algorithm according to the grid model loss value.

[0043] After the training is completed, for a given posture 3D mesh model and an action 3D mesh model, the posture 3D mesh model can be made to perform actions provided by the action 3D mesh model.

[0044] By calculating the L2 loss to compare with the existing technology, it is found that its performance exceeds the existing methods. Through functional analysis, it is found that SPAdaIN makes the generated results accurate and maintains excellent results for various body shapes, and the Edge loss function makes the model surface smooth.

[0045] Example Function and Effect

[0046] The three-dimensional model motion migration method of this embodiment designs a multi-layer parameter-sharing convolutional network for extracting motion features of the action three-dimensional mesh model for the action three-dimensional mesh model that provides the action; designs a decoder that uses spatial adaptive instance standardization for the posture three-dimensional mesh model that provides the posture, and migrates the posture three-dimensional mesh model based on the extracted motion features; constructs an end-to-end motion feature neural network framework under this framework, which can migrate the motion of the action three-dimensional mesh model to the posture three-dimensional mesh model for a given posture three-dimensional mesh model and action three-dimensional mesh model, and output the corresponding action posture three-dimensional mesh model. This method does not require any additional information assistance, has a good generation effect for various posture models, and is highly robust to point arrangement order disruption and noise.

[0047] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A 3D model motion migration method, for migrating the motion of a motion 3D mesh model to a body posture 3D mesh model, thereby obtaining a motion body posture 3D mesh model with new motion, characterized in that: The steps include: Step S1, generating an action mesh model, a posture mesh model, and an action posture model transferred to a new action as real values ​​for training an action transfer neural network according to pre-acquired three-dimensional model data; Step S2, constructing a feature encoding network as an action feature extractor to extract action features of the action three-dimensional mesh model; Step S3, constructing a feature transfer neural network as a decoder to perform a transfer operation on the posture three-dimensional mesh model based on the extracted action features, thereby obtaining the deformed action posture model; Step S4, constructing an end-to-end framework of the action transfer neural network, that is, given a body posture 3D mesh model and an action 3D mesh model, obtaining action features of the action 3D mesh model through an action feature encoding network, transferring the action to the body posture 3D mesh model through a decoder based on a spatial adaptive instance normalization method, and outputting a 3D mesh model after the body posture model is converted into the action; Step S5, constructing a loss function; Step S6, training the action transfer neural network based on the loss function, Wherein, the action transfer neural network includes the action feature encoding network and the feature transfer neural network based on the spatial adaptive instance normalization method.

2. The three-dimensional model motion migration method according to claim 1, characterized in that: The specific process of step S1 is as follows: The three-dimensional model data is generated by using the SMPL model, the action parameters in the SMPL model are controlled to generate three-dimensional mesh models of different actions as the action mesh model, the posture parameters in the SMPL model are controlled to generate three-dimensional mesh models of different postures as the posture three-dimensional mesh model, and the action parameters and posture parameters of the SMPL model are used to generate a posture model after the conversion action as the corresponding action posture model required for training the action transfer neural network as the true value.

3. The three-dimensional model motion migration method according to claim 1, characterized in that: in, The feature encoding network is a three-layer parameter-sharing convolutional network, in which each convolutional layer uses an instance normalization layer and a rectified linear unit (ReLU) as an activation function. The input of the feature encoding network is the action grid model, and the output is the matrix of the encoded action features.

4. The three-dimensional model motion migration method according to claim 1, characterized in that: in, The feature transfer neural network is a four-layer convolutional network, in which the first three convolutional layers are each followed by a spatially adaptive instance normalized residual module. Each spatially adaptive instance normalization residual module follows the structure of a residual network module, which has three units, each of which includes a spatially adaptive instance normalization structure, a convolutional network, and an activation function. The standardized structure of each spatial adaptability instance is that the body posture three-dimensional mesh model obtains the weight and offset required for standardization through a layer of convolution layer, multiplies the weight with the input feature and adds the offset before outputting, and further passes through a layer of convolution layer and tanh activation function to obtain the output action body posture three-dimensional mesh model.

5. The three-dimensional model motion migration method according to claim 1, characterized in that: The specific process of step S5 is as follows: The two-norm Euclidean distance is used as the loss function to regress the spatial coordinates of each vertex in the mesh model to make it close to the real coordinates. Edge length regularization is used to make all edges of the migrated action posture three-dimensional mesh model as short and uniform as possible, thereby making the surface smoother.

6. The three-dimensional model motion migration method according to claim 1, characterized in that: The specific process of step S6 is as follows: The action mesh 3D model and the posture 3D mesh model are used as inputs of the action transfer neural network to obtain the action posture 3D mesh model with a new action output by the action transfer neural network. The loss value is calculated using the loss function according to the action posture three-dimensional mesh model obtained through the action transfer neural network and the action posture three-dimensional mesh model used as the true value in step S1, The action transfer neural network is trained according to the loss value using a back propagation algorithm and a gradient descent algorithm.

Images