Method and system for migrating any motion style of human body and storable medium
By using a multi-conditional denoising network to denoise Gaussian noise in the latent spatial diffusion model, combining content features, trajectory features and style features, the problem of foot slipping in the existing technology is solved, and the effect of transferring arbitrary motion styles and retaining the motion trajectory is achieved.
Patent Information
- Application Number
- CN202411081779.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2025-04-08
AI Technical Summary
The existing human body's movement style transfer methods are prone to foot slipping during the style transfer process, and cannot effectively retain the trajectory information of the movement.
In the latent spatial diffusion model, the Gaussian noise is denoised by guiding the content features, trajectory features and style features to obtain the stylized motion potential features, and restore it to the stylized motion through the pre-trained motion decoder.
It effectively avoids foot slippage in stylized movements, greatly retains the trajectory information of the movement, and achieves a good transfer of any movement style.
Smart Images
Figure CN119784906A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer animation and virtual reality, and in particular to a method, system and storable medium for transferring arbitrary human motion styles. Background Art
[0002] Human arbitrary motion style transfer is of great benefit to the virtual reality and computer animation industries. Animators often seek to create stylized motion to express the character's personality or emotion, making the character more lifelike. However, traditional motion capture methods rely on huge human labor and specific equipment. In order to generate realistic animations, a large number of studies have proposed arbitrary motion style transfer methods driven by the captured motion style. The main problem is how to effectively transfer the style while preserving the content of the motion to the greatest extent and accurately preserving the motion trajectory information.
[0003] In recent years, due to the rise of deep learning, some methods use neural networks to separate the content and style of motion, and then combine them to achieve style transfer. There are many ways to combine them. The more effective one is to change the style of motion by changing the data distribution of the content latent code, that is, to use adaptive instance normalization (AdaIN) with style weights to adjust the data distribution of the content latent code to achieve motion style transfer. There are also many ways to encode motion, such as one-dimensional time convolutional neural networks and graph convolutional neural networks. The main difference between the two is that graph convolutional neural networks can preserve the temporal and spatial information of motion, but the corresponding network cost is also high. However, although these data-driven motion style transfers have made progress for many years, there are still some problems in practice, mainly manifested in that the existing methods of style transfer are mostly targeted at defined styles and limited to limited style features. Although there are methods to achieve arbitrary motion style transfer, they still show poor style performance. In addition, previous methods directly copy trajectory information from content motion to stylized motion, which can cause stylized motion to slip.
[0004] Therefore, how to provide a method for transferring the style of arbitrary human motion so that stylized motion can avoid the phenomenon of foot slipping is a problem that technical personnel in this field need to solve urgently. Summary of the invention
[0005] In view of this, the present invention provides a method for transferring the style of arbitrary human motion, which can solve the problems existing in the prior art.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A method for transferring human body arbitrary motion style, comprising the following steps:
[0008] S1, collecting motion information and trajectory information of input content movement;
[0009] S2, input the motion information and trajectory information in S1 into the content coding network and the trajectory coding network respectively to obtain content features and trajectory features;
[0010] S3, collecting motion information of input style motion;
[0011] S4, input the motion information in S3 into the pre-trained style encoder for encoding to obtain style features;
[0012] S5, using content features, trajectory features and style features as conditions to guide the latent space diffusion model, using a multi-conditional denoising network to denoise Gaussian noise and obtain stylized motion latent features;
[0013] S6. Input the stylized motion latent features in S5 into the pre-trained motion decoder to restore the stylized motion.
[0014] In the above method, optionally, the content encoding network in S2 includes a pre-trained motion encoder and a style removal module.
[0015] In the above method, optionally, the trajectory encoding network in S2 is composed of a fully connected layer and a transformer network.
[0016] In the above method, optionally, the training process of the pre-trained style encoder in S4 aligns the motion latent space with the latent space of the image and text.
[0017] In the above method, optionally, the motion latent space is obtained by a pre-trained motion encoder and a pre-trained motion decoder.
[0018] In the above method, optionally, the pre-trained motion decoder is trained together with the pre-trained motion encoder to restore the latent space motion features into real motion.
[0019] In the above method, optionally, the training of the diffusion model in S5 includes a forward noise addition process and a reverse denoising process, and the noise addition process follows the Markov chain:
[0020]
[0021] Among them, z n 、z n-1 Represent the potential noise characteristics in the noise adding process in the nth step and the n-1th step respectively; α n It is a hyperparameter used to adjust the degree of noise addition, and I represents the identity matrix.
[0022] In the above method, optionally, the multi-condition denoising network in S5 is used for the denoising process of the diffusion model, and the multi-condition is to predict the final noise under the guidance of content features, trajectory features and style features.
[0023] A human body arbitrary motion style transfer system, which executes any one of the human body arbitrary motion style transfer methods described above, comprises an acquisition module, a content encoding module, a trajectory encoding module, a style encoding module, a multi-condition denoising module and a decoding module;
[0024] The output end of the acquisition module is connected to the input end of the content encoding module, the trajectory encoding module, and the style encoding module respectively; the input end of the multi-condition denoising module is connected to the input end of the content encoding module, the trajectory encoding module, and the style encoding module respectively; and the input end of the decoding module is connected to the output end of the multi-condition denoising module;
[0025] Acquisition module: used to acquire motion information and trajectory information of input content motion, and motion information of style motion;
[0026] Content encoding module: used to encode motion content information into content latent features and remove style information at the same time;
[0027] Trajectory encoding module: used to encode motion trajectory information to obtain trajectory potential features;
[0028] Style encoding module: used to encode motion style information to obtain style potential features;
[0029] Multi-conditional denoising module: used to implement the denoising process of the latent diffusion model under the conditional guidance of the obtained content latent features, trajectory latent features and style latent features, so as to obtain stylized latent motion features;
[0030] Decoding module: used to restore the obtained stylized latent motion features into stylized motion.
[0031] A computer storable medium stores a computer program, which, when executed by a processor, implements the steps of any of the above methods for transferring the style of arbitrary human motion.
[0032] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a method, system and storage medium for transferring the style of arbitrary human motion, which has the following beneficial effects: the present invention can finally obtain the stylized motion after arbitrary style transfer by inputting the style motion, content motion and the trajectory information of the content motion through the above solution, and at the same time, due to the addition of the motion trajectory information as an additional condition, the obtained stylized motion can greatly retain the trajectory of the motion and avoid the problem of slipping. Extensive experimental results and comprehensive evaluations have confirmed that the method for transferring the style of arbitrary human motion of the present invention can well transfer the style of arbitrary motion while retaining the content and trajectory of the motion. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 A flow chart of a method for transferring human body arbitrary motion style disclosed in the present invention;
[0035] Figure 2 A schematic diagram of stylized motion disclosed in an embodiment of the present invention;
[0036] Figure 3 This is a structural diagram of a human body arbitrary motion style transfer system disclosed in the present invention. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0039] Reference Figure 1 As shown, the present invention discloses a method for transferring the style of arbitrary human body motion, comprising the following steps:
[0040] S1, collecting motion information and trajectory information of input content movement;
[0041] S2, input the motion information and trajectory information in S1 into the content coding network and the trajectory coding network respectively to obtain content features and trajectory features;
[0042] S3, collecting motion information of input style motion;
[0043] S4, input the motion information in S3 into the pre-trained style encoder for encoding to obtain style features;
[0044] S5, using content features, trajectory features and style features as conditions to guide the latent space diffusion model, using a multi-conditional denoising network to denoise Gaussian noise and obtain stylized motion latent features;
[0045] S6. Input the stylized motion latent features in S5 into the pre-trained motion decoder to restore the stylized motion.
[0046] Furthermore, the content encoding network in S2 includes a pre-trained motion encoder and a style removal module.
[0047] Furthermore, the trajectory encoding network in S2 consists of a fully connected layer and a transformer network.
[0048] Furthermore, the training process of the pre-trained style encoder in S4 aligns the motion latent space with the latent space of images and text.
[0049] Furthermore, the motion latent space is obtained by a pre-trained motion encoder and a pre-trained motion decoder.
[0050] Furthermore, the pre-trained motion decoder is trained together with the pre-trained motion encoder to restore the latent space motion features to real motion.
[0051] Furthermore, the training of the diffusion model in S5 includes a forward noise addition process and a reverse denoising process, and the noise addition process follows the Markov chain:
[0052]
[0053] Among them, z n 、z n-1 Represent the potential noise characteristics in the noise adding process in the nth step and the n-1th step respectively; α n It is a hyperparameter used to adjust the degree of noise addition, and I represents the identity matrix.
[0054] Furthermore, the multi-condition denoising network in S5 is used for the denoising process of the diffusion model, and the multi-conditions are used to predict the final noise under the guidance of content features, trajectory features, and style features.
[0055] Specifically, in a specific embodiment, the method is compared with the most advanced high-performance human arbitrary motion style transfer method, see Figure 2 The following is a schematic diagram of stylized motion. For data and structure, see Table 1.
[0056] Table 1 evaluates the motion quality, content preservation, style expression and foot slippage respectively.
[0057]
[0058]
[0059] Table 1 evaluates motion quality, content preservation, style expression, and foot slippage, respectively, and shows the performance of the state-of-the-art methods: Deep-Motion-Editing, Diverse-Style-Stylization, and Motion-Puzzle.
[0060] and Figure 1 Correspondingly, the present invention also discloses a human body arbitrary motion style transfer system, which executes any of the above human body arbitrary motion style transfer methods. The specific structural block diagram is shown in Figure 3 As shown, it includes an acquisition module, a content encoding module, a trajectory encoding module, a style encoding module, a multi-condition denoising module and a decoding module;
[0061] The output end of the acquisition module is connected to the input end of the content encoding module, the trajectory encoding module, and the style encoding module respectively; the input end of the multi-condition denoising module is connected to the input end of the content encoding module, the trajectory encoding module, and the style encoding module respectively; and the input end of the decoding module is connected to the output end of the multi-condition denoising module;
[0062] Acquisition module: used to acquire motion information and trajectory information of input content motion, and motion information of style motion;
[0063] Content encoding module: used to encode motion content information into content latent features and remove style information at the same time;
[0064] Trajectory encoding module: used to encode motion trajectory information to obtain trajectory potential features;
[0065] Style encoding module: used to encode motion style information to obtain style potential features;
[0066] Multi-conditional denoising module: used to implement the denoising process of the latent diffusion model under the conditional guidance of the obtained content latent features, trajectory latent features and style latent features, so as to obtain stylized latent motion features;
[0067] Decoding module: used to restore the obtained stylized latent motion features into stylized motion.
[0068] A computer storable medium stores a computer program, which, when executed by a processor, implements the steps of any of the above methods for transferring the style of arbitrary human motion.
[0069] The embodiment of the present invention discloses a method, system and storage medium for transferring the style of arbitrary human motion. The present invention inputs the motion information and trajectory information of the content motion, and passes them through the content encoding network and the trajectory encoding network respectively to obtain the content features and the trajectory features; inputs the motion information of the style motion, and encodes it through the pre-trained style encoder to obtain the style features; uses the obtained content, trajectory and style features as conditions to guide the denoising process of Gaussian noise in the potential space diffusion model, and the denoising process is realized by a multi-condition denoising network, and finally obtains the stylized motion potential features; inputs the stylized motion potential features into the motion decoder, and restores them into the final stylized motion. Through the above scheme, the present invention can finally obtain the stylized motion after arbitrary style transfer by inputting the style motion, content motion and the trajectory information of the content motion, and at the same time, due to the addition of the motion trajectory information as an additional condition, the obtained stylized motion can greatly retain the trajectory of the motion and avoid the problem of slipping. Extensive experimental results and comprehensive evaluations have confirmed that our method for transferring the style of arbitrary human motion can well transfer the style of arbitrary motion while retaining the content and trajectory of the motion.
[0070] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.
[0071] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for transferring human body arbitrary motion style, characterized in that: The following steps are involved: S1, collecting motion information and trajectory information of input content movement; S2, input the motion information and trajectory information in S1 into the content coding network and the trajectory coding network respectively to obtain content features and trajectory features; S3, collecting motion information of input style motion; S4, input the motion information in S3 into the pre-trained style encoder for encoding to obtain style features; S5, using content features, trajectory features and style features as conditions to guide the latent space diffusion model, using a multi-conditional denoising network to denoise Gaussian noise and obtain stylized motion latent features; S6. Input the stylized motion latent features in S5 into the pre-trained motion decoder to restore the stylized motion.
2. A method for transferring human body arbitrary motion style according to claim 1, characterized in that: The content encoding network in S2 includes a pre-trained motion encoder and a style removal module.
3. The method for transferring human body arbitrary motion style according to claim 1, characterized in that: The trajectory encoding network in S2 consists of a fully connected layer and a transformer network.
4. The method for transferring human body arbitrary motion style according to claim 1, characterized in that: The training process of the pre-trained style encoder in S4 aligns the motion latent space with the latent space of images and text.
5. A method for transferring human body arbitrary motion style according to claim 4, characterized in that: The motion latent space is obtained by a pre-trained motion encoder and a pre-trained motion decoder.
6. A method for transferring human body arbitrary motion style according to claim 5, characterized in that: The pre-trained motion decoder is trained together with the pre-trained motion encoder to restore the latent space motion features to real motion.
7. The method for transferring human body arbitrary motion style according to claim 1, characterized in that: The training of the diffusion model in S5 includes a forward noise addition process and a reverse denoising process, and the noise addition process follows the Markov chain: Among them, z n 、z n-1 Represent the potential noise characteristics in the noise adding process in the nth step and the n-1th step respectively; α n It is a hyperparameter used to adjust the degree of noise addition, and I represents the identity matrix.
8. The method for transferring human body arbitrary motion style according to claim 1, characterized in that: The multi-condition denoising network in S5 is used for the denoising process of the diffusion model. The multi-condition is to predict the final noise under the guidance of content features, trajectory features and style features.
9. A human body arbitrary motion style transfer system, characterized in that: A method for transferring human body arbitrary motion style according to any one of claims 1 to 8 is implemented, comprising an acquisition module, a content encoding module, a trajectory encoding module, a style encoding module, a multi-condition denoising module and a decoding module; The output end of the acquisition module is connected to the input end of the content encoding module, the trajectory encoding module, and the style encoding module respectively; the input end of the multi-condition denoising module is connected to the input end of the content encoding module, the trajectory encoding module, and the style encoding module respectively; and the input end of the decoding module is connected to the output end of the multi-condition denoising module; Acquisition module: used to acquire motion information and trajectory information of input content motion, and motion information of style motion; Content encoding module: used to encode motion content information into content latent features and remove style information at the same time; Trajectory encoding module: used to encode motion trajectory information to obtain trajectory potential features; Style encoding module: used to encode motion style information to obtain style potential features; Multi-conditional denoising module: used to implement the denoising process of the latent diffusion model under the conditional guidance of the obtained content latent features, trajectory latent features and style latent features, so as to obtain stylized latent motion features; Decoding module: used to restore the obtained stylized latent motion features into stylized motion.
10. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for transferring the style of arbitrary human motion as described in any one of claims 1 to 8 are implemented.