A method and system for completing keyframe animations of complex motion based on neural networks

Abstract

This invention discloses a method and system for keyframe animation completion of complex motion based on neural networks. By providing precise footwork control through different foot contact information, this invention offers a stable and high-quality method for keyframe animation completion of complex movements. It eliminates the slippage problem during motion generation and exhibits strong robustness. This solves the problem in existing technologies that utilize deep learning to perform complex motion generation tasks, which suffer from severe slippage and ghosting issues, leading to significant manpower and resources required for subsequent motion correction.

Description

Technical Field

[0001] This invention relates to the field of animation production, and in particular to a method and system for completing complex motion keyframe animations based on neural networks. Background Technology

[0002] In animation technology, human body animation is one of the most challenging topics. It has broad application prospects and plays an important role in many fields such as simulation training, film and television production, entertainment games, and virtual reality. Because human movement involves multiple degrees of freedom, how to set the degrees of freedom of human movement on animation keyframes to generate highly realistic 3D human body animation has become a research hotspot in the field of computer science. To obtain more realistic human movements, the most direct method is to use motion capture data to drive human body animation, as the captured data has high fidelity and real-time processing capabilities, making it the most widely used method. However, captured data has drawbacks such as high data cost, large data volume, lack of structured information, and poor reusability of motion data. Therefore, for the production of video games and animations, filling in motion on given keyframes is a simple and efficient method. Traditional methods generate motion sequences through interpolation based on the motion time function of several keyframes. Human movement is inherently complex and random. Simple interpolation methods are insufficient to generate reasonable and high-quality movements, often requiring complex post-processing algorithms.

[0003] However, in modern video game and animation production, designing animations for characters with rich behaviors using motion capture and traditional keyframe animation is extremely time-consuming. With the development of deep learning, a common trend is to utilize deep learning to assist traditional production methods. Currently, end-to-end generative models achieve good results with simple, periodic movements, but often fall short for complex and varied actions. In diverse and complex motion generation tasks such as dance and martial arts, deep learning-based methods often produce animations with effects like sliding and afterimages. Consequently, even end-to-end generative models still require numerous and cumbersome post-processing algorithms to correct the generated animations, significantly limiting the application of these algorithms.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for completing keyframe animations of complex motion based on neural networks, in order to address the above-mentioned defects of the prior art. The aim is to solve the problem that the existing technology uses deep learning to perform complex motion generation tasks, which has serious problems of sliding and afterimages, resulting in a large amount of manpower and resources to correct the motion in the later stage.

[0006] The technical solution adopted by this invention to solve the problem is as follows:

[0007] In a first aspect, embodiments of the present invention provide a method for completing complex motion keyframe animations based on neural networks, wherein the method includes:

[0008] Acquire keyframe information and foot contact information of the motion sequence, and determine a hybrid embedding representation based on the keyframe information and the foot contact information, wherein the hybrid embedding representation is used to reflect the summation result of the keyframe information and the foot contact information after they are respectively embedded in a high-dimensional space;

[0009] Interpolation padding is applied between keyframes to obtain the padded motion sequence;

[0010] The motion sequence after the hybrid embedding representation and padding is input into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted by the spatiotemporal self-attention network.

[0011] The motion posture and the foot contact state are input into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state;

[0012] The keyframes are completed based on the motion poses between keyframes and the reconstructed root trajectory.

[0013] In one implementation, determining a hybrid embedding representation based on the keyframe information and the footstep contact information includes:

[0014] Obtain a preset keyframe encoding matrix, and embed the keyframe information into a high-dimensional space according to the keyframe encoding matrix to obtain a keyframe embedding representation.

[0015] Obtain a preset foot contact encoding matrix, and embed the foot contact embedding information into a high-dimensional space according to the foot contact encoding matrix to obtain a foot contact embedding representation;

[0016] The keyframe embedding representation and the footstep contact embedding representation are summed to obtain the hybrid embedding representation.

[0017] In one embodiment, the interpolation fill is a spherical linear interpolation fill.

[0018] In one embodiment, the spatiotemporal self-attention network includes:

[0019] A spatiotemporal encoder module is used to determine the spatiotemporal coding features of the motion sequence based on the padded motion sequence and the hybrid embedding representation;

[0020] The decoder module is used to decode the spatiotemporal encoded features to obtain the motion posture and the foot contact state.

[0021] In one embodiment, the spatiotemporal encoder module includes:

[0022] A spatial self-attention module is used to extract spatial coding features of the motion sequence based on the filled motion sequence and a preset spatial position embedding representation, wherein the spatial position embedding representation is used to distinguish different joints of the human skeleton.

[0023] The temporal self-attention module is used to acquire a merged data representation based on the spatial coding features, a preset temporal position embedding representation, and the hybrid embedding representation, wherein the temporal position embedding representation is used to distinguish different frames of the motion sequence; and to extract the spatiotemporal coding features of the motion sequence based on the merged data representation.

[0024] In one implementation, the method for generating the spatial location embedding representation includes:

[0025] Obtain the joint markers corresponding to each joint of the human skeleton;

[0026] The joint markers are embedded into a high-dimensional space using a preset matrix to obtain the spatial position embedding representation.

[0027] In one embodiment, the decoder module includes:

[0028] An attitude decoder is used to decode the spatiotemporal encoded features to obtain the motion attitude;

[0029] A footstep state decoder is used to decode the spatiotemporal encoded features to obtain the footstep contact state.

[0030] In one implementation, the method for determining the loss function used by the spatiotemporal self-attention network during training includes:

[0031] Obtain the reconstruction loss, wherein the reconstruction loss is used to converge the gap between the true motion pose and the predicted motion pose;

[0032] Obtain the foot contact loss, wherein the foot contact loss is used to converge the difference between the actual foot contact state and the predicted foot contact state;

[0033] Obtain the root locus loss, wherein the root locus loss is used to converge the difference between the true root locus and the reconstructed root locus;

[0034] Obtain the joint position loss, wherein the joint position loss is used to converge the difference between the true joint position and the predicted joint position calculated based on the predicted motion posture.

[0035] The loss function is determined based on the reconstruction loss, the foot contact loss, the root trajectory loss, and the joint position loss.

[0036] In one implementation, the constraints of the root trajectory reconstruction algorithm are determined based on the motion posture and the foot contact state. The operation principle of the root trajectory reconstruction algorithm is to determine the position of the root node in the (t+1)th frame by subtracting the displacement of the touching foot in the tth frame from the position of the root node in the tth frame.

[0037] In one embodiment, the method further includes:

[0038] The completion action data is stored as BVH action data, and the BVH action data is imported into the Blender skeleton model for display.

[0039] Obtain several preset quantitative evaluation indicators, and evaluate the completion action data according to each of the quantitative evaluation indicators.

[0040] Secondly, embodiments of the present invention also provide a complex motion keyframe animation completion system based on a neural network, wherein the system includes:

[0041] The definition and embedding module is used to acquire keyframe information and foot contact information of the motion sequence, and to determine a hybrid embedding representation based on the keyframe information and the foot contact information. The hybrid embedding representation is used to reflect the summation result of embedding the keyframe information and the foot contact information into a high-dimensional space respectively.

[0042] The filling and prediction module is used to interpolate and fill the keyframes to obtain the filled motion sequence; the hybrid embedding representation and the filled motion sequence are input into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted based on the spatiotemporal self-attention network.

[0043] The reconstruction and completion module is used to input the motion posture and the foot contact state into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state; and to complete the keyframes based on the motion posture and the reconstructed root trajectory between keyframes.

[0044] In one implementation, the reconstructed root trajectory is a loss term of the loss function of the spatiotemporal self-attention network, the constraints of the root trajectory reconstruction algorithm are determined based on the motion posture and the foot contact state, and the operation principle of the root trajectory reconstruction algorithm is to determine the position of the root node in the (t+1)th frame by subtracting the displacement of the touching foot in the tth frame from the position of the root node in the tth frame.

[0045] In one embodiment, the system further includes:

[0046] The display and evaluation module is used to store the completion action data as BVH action data, import the BVH action data into the Blender skeleton model for display, obtain a number of preset quantitative evaluation indicators, and evaluate the completion action data according to each of the quantitative evaluation indicators.

[0047] Thirdly, embodiments of the present invention also provide a terminal, wherein the terminal includes a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the neural network-based complex motion keyframe animation completion method as described above; and the processor is used to execute the programs.

[0048] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the steps of any of the above-described neural network-based complex motion keyframe animation completion methods.

[0049] The beneficial effects of this invention are as follows: By providing different foot contact information, this invention achieves precise foot control, providing a stable and high-quality method for keyframe animation completion of complex movements. It can eliminate the slippage problem during motion generation and has strong robustness. It solves the problem in existing technologies that use deep learning to perform complex motion generation tasks, which suffer from severe slippage and ghosting issues, leading to the need for significant manpower and resources for motion correction in later stages. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the basic process of the complex motion keyframe animation completion method based on neural networks provided in the embodiments of the present invention.

[0052] Figure 2 This is a schematic diagram illustrating the working principle of the spatiotemporal self-attention network provided in an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the structure of the self-attention module and a schematic diagram of the generation of hybrid embedding representation provided in the embodiments of the present invention.

[0054] Figure 4 This is a schematic diagram of the displacement of the ground contact foot and the displacement of the root joint provided in an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of motion transition generation under different grounding foot state control conditions in the same keyframe provided by an embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram of motion transition generation under different keyframe control conditions with the same grounding foot state provided in an embodiment of the present invention.

[0057] Figure 7 This is a schematic diagram of motion generation when both feet touch the ground, provided in an embodiment of the present invention.

[0058] Figure 8 This is a comparative experimental diagram of motion generation with irrelevant node position loss supervision provided in an embodiment of the present invention.

[0059] Figure 9 This is a comparative experimental diagram of motion generation with and without hybrid embedding representation provided in an embodiment of the present invention.

[0060] Figure 10 This is a schematic diagram of the human skeleton structure provided in an embodiment of the present invention.

[0061] Figure 11 This is a schematic diagram of the basic modules of the complex motion keyframe animation completion system based on neural networks provided in this embodiment of the invention.

[0062] Figure 12 This is a schematic diagram of the terminal provided in the embodiment of the present invention. Detailed Implementation

[0063] This invention discloses a method and system for completing keyframe animations of complex motion based on neural networks. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0064] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0065] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0066] To address the aforementioned deficiencies in existing technologies, this invention provides a method for completing keyframe animations of complex motions based on neural networks. The method includes: acquiring keyframe information and foot contact information of a motion sequence; determining a hybrid embedding representation based on the keyframe information and the foot contact information, wherein the hybrid embedding representation reflects the summation result of embedding the keyframe information and the foot contact information into a high-dimensional space; interpolating and padding between keyframes to obtain the padded motion sequence; inputting the hybrid embedding representation and the padded motion sequence into a pre-trained spatiotemporal self-attention network to obtain motion posture and foot contact state predicted by the spatiotemporal self-attention network; inputting the motion posture and the foot contact state into a preset root trajectory reconstruction algorithm to obtain a reconstructed root trajectory corresponding to the foot contact state; and completing the keyframes based on the motion posture and the reconstructed root trajectory. This invention achieves precise foot control by providing different foot contact information, providing a stable and high-quality method for completing keyframe animations of complex movements. It can eliminate the slippage problem during motion generation and has strong robustness. This solves the problem that existing technologies using deep learning to perform complex motion generation tasks suffer from severe slippage and ghosting issues, which lead to the need for significant manpower and resources for motion correction in the later stages.

[0067] like Figure 1 As shown, the method includes:

[0068] Step S100: Obtain keyframe information and foot contact information of the motion sequence, and determine a hybrid embedding representation based on the keyframe information and the foot contact information, wherein the hybrid embedding representation is used to reflect the summation result of the keyframe information and the foot contact information after they are respectively embedded in a high-dimensional space;

[0069] Step S200: Interpolate and fill the keyframes to obtain the filled motion sequence;

[0070] Step S300: Input the motion sequence after the hybrid embedding representation and filling into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted based on the spatiotemporal self-attention network;

[0071] Step S400: Input the motion posture and the foot contact state into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state;

[0072] Step S500: Complete the keyframes according to the motion poses between keyframes and the reconstructed root trajectory.

[0073] Specifically, the motion sequence in this embodiment is the keyframe animation to be completed. The control conditions and parameters required for the completion process include keyframe information and foot contact information. Furthermore, a hybrid embedding composed of keyframe information and foot contact information is needed to reflect the summation result of embedding the two types of information into a high-dimensional space. To achieve keyframe animation completion, this embodiment pre-constructs and trains a spatiotemporal self-attention network. The input to the spatiotemporal self-attention network is the interpolation filling result between keyframes, and the network can predict the motion posture and foot contact state. Then, the motion posture and foot contact state are input into a preset root trajectory reconstruction algorithm, ultimately obtaining the motion posture between keyframes and the reconstructed root trajectory that matches the corresponding foot contact state. By using the motion posture between keyframes and the reconstructed root trajectory, keyframe motion completion can be achieved, eliminating the sliding phenomenon and presenting realistic and natural movements.

[0074] In one implementation, the keyframe information is used to reflect the frame type of each frame. For example, K E Keyframe information is identified by one-hot encoding. Any keyframe can be represented as [1,0], and intermediate frames can be represented as [0,1]. For a sequence of length T, the keyframe information K... E The dimension can be represented as T×2.

[0075] In one implementation, the foot contact information is used to reflect the foot joint closest to the ground in each frame. For example, S cThis represents the footstep contact information of a given sequence. Figure 10 The human skeletal structure is illustrated. In this embodiment, the foot joint is selected as the second-to-last joint of the leg in the skeleton, and the contacting foot is chosen. The foot contact determination criterion is the foot joint closest to the ground in the world coordinate system. For any frame, left foot contact can be represented as [0,1], and right foot contact can be represented as [1,0]. For a sequence of length T, the foot contact information S... c The dimension can be represented as T×2. Figure 5 The motion transition generation is shown under different foot touch state control conditions in the same keyframe. (a) is the foot touch control signal, (b) is the generated motion, and (c) is the root trajectory corresponding to the generated motion. Figure 6 The motion transition generation is shown under different keyframe control conditions with the same ground foot state. (a) and (b) are the generated motions, and (c) is the root trajectory corresponding to the generated motion. Figure 7 The motion generation when both feet are on the ground is shown. (a) is the true motion, (b) is the motion generated assuming the left foot is on the ground, (c) is the motion generated assuming the right foot is on the ground, and (d) is the root locus corresponding to the motion.

[0076] In one implementation, the interpolation fill is a spherical linear interpolation fill.

[0077] For example, K1 and K2 represent the quaternions of all joint rotations in the beginning and end keyframes, respectively. The result of its spherical linear interpolation is given for a skeleton structure with J joints in frame T. The dimensions are T×J×4.

[0078] In one implementation, determining the hybrid embedding representation based on the keyframe information and the footstep contact information includes:

[0079] Step S101: Obtain a preset keyframe encoding matrix, and embed the keyframe information into a high-dimensional space according to the keyframe encoding matrix to obtain a keyframe embedding representation.

[0080] Step S102: Obtain a preset foot contact encoding matrix, and embed the foot contact embedding information into a high-dimensional space according to the foot contact encoding matrix to obtain a foot contact embedding representation.

[0081] Step S103: Sum the keyframe embedding representation and the footstep contact embedding representation to obtain the hybrid embedding representation.

[0082] Specifically, assuming the high-dimensional space is a 1024-dimensional space, for the keyframe information K... E Contact information with footsteps S c Its dimension can be represented as T×2. The keyframe encoding matrix Wkf and foot contact coding matrix W fc Keyframe information K E Contact information with footsteps S c The keyframe embedding representation E is obtained by embedding into a 1024-dimensional space. kf Footprint contact embedding representation E fc .like Figure 3 As shown, the hybrid embedding characterization E mix Then through E kf With E fc The sum of these elements constitutes a structure with dimensions T×1024.

[0083] In one implementation, the completed motion data includes the motion posture and foot contact state of the completed motion, such as... Figure 2 As shown, the spatiotemporal self-attention network includes:

[0084] A spatiotemporal encoder module is used to determine the spatiotemporal coding features of the motion sequence based on the padded motion sequence and the hybrid embedding representation;

[0085] The decoder module is used to decode the spatiotemporal encoded features to obtain the motion posture and the foot contact state.

[0086] Specifically, the spatiotemporal self-attention network in this embodiment mainly includes two parts: a spatiotemporal encoder module and a decoder module. The interpolation padding results between keyframes serve as the input to the spatiotemporal encoder module. The spatiotemporal encoded features of the input sequence are extracted by the spatiotemporal encoder module, and then the spatiotemporal encoded features are input to the decoder module to decode the motion posture and foot contact state of the completed motion.

[0087] In one implementation, such as Figure 3 As shown, both the spatiotemporal encoder module and the decoder module are composed of self-attention modules. Each self-attention module consists of a multi-head self-attention (MHSA) module and a feed-forward network (FFN) module. The MHSA module consists of multiple self-attention modules and is used to adaptively establish the relationship between input labels, while the FFN module consists of two fully connected (FC) layers and is used for feature transformation. Residual addition is performed after both the MHSA and FFN modules.

[0088] In one implementation, the spatiotemporal encoder module includes:

[0089] A spatial self-attention module is used to extract spatial coding features of the motion sequence based on the filled motion sequence and a preset spatial position embedding representation, wherein the spatial position embedding representation is used to distinguish different joints of the human skeleton.

[0090] The temporal self-attention module is used to acquire a merged data representation based on the spatial coding features, a preset temporal position embedding representation, and the hybrid embedding representation, wherein the temporal position embedding representation is used to distinguish different frames of the motion sequence; and to extract the spatiotemporal coding features of the motion sequence based on the merged data representation.

[0091] In simple terms, the spatiotemporal encoder module in this embodiment is composed of a multi-layer spatial self-attention network and a multi-layer temporal self-attention network (e.g., a stack of 4 spatial Transformer layers and 4 temporal Transformer layers). Therefore, it is divided into two main modules: a spatial self-attention module and a temporal self-attention module. The spatial self-attention module learns the relationships between joints, while the temporal self-attention module learns the relationships between different frames of the sequence. The input to the spatial self-attention module is the interpolation padding results between keyframes and a learnable spatial location embedding representation, used to extract the spatial coding features of the input sequence. Then, the spatial coding features output by the spatial self-attention module are flattened, and the learnable temporal location embedding representation and the hybrid embedding representation are merged as the input to the temporal self-attention module, ultimately obtaining the spatiotemporal coding features of the input sequence. In one implementation, the method for generating the spatial location embedding representation includes: obtaining joint markers corresponding to each joint of the human skeleton; embedding each joint marker into a high-dimensional space using a preset matrix to obtain the spatial location embedding representation. This embodiment adds a spatial location embedding representation to the input of the spatial self-attention module. Due to the permutation invariance of the Transformer structure, adding the spatial location embedding representation can distinguish different joints of the human skeleton. Add temporal position embedding representation and hybrid embedding representation to the input of the temporal self-attention module. The temporal position embedding representation can distinguish different frames of the motion sequence, and the hybrid embedding includes keyframe embedding and foot contact embedding to distinguish different frame markers and foot contact states. Figure 9 The experiment diagram shows a comparison of motion generation with and without hybrid embedding representation.

[0092] For example, each joint of a 32-joint skeleton is considered a marker, and a preset matrix W is used to define the marker. s ∈R 32*4 Embed them into a 32-dimensional space, and then add a learnable spatial location to embed E. s ∈R 32×32This will serve as the input to the spatial Transformer. The MHSA module has a head size of 8 and an embedding dimension of 32. The final result is the spatial encoded features X of the input sequence. So ∈R T×32×32 Flatten the output of the spatial Transformer while embedding the learnable temporal location into E. T ∈R T×1024 and hybrid embedded E mix ∈R T×1024 The MHSA modules are merged and used as input to the temporal Transformer, which has a head size of 8 and an embedding dimension of 1024. The temporal Transformer module follows the same steps as the spatial Transformer, ultimately yielding the spatiotemporal encoded features X of the input sequence. To ∈R T×1024 .

[0093] In one implementation, the decoder module includes:

[0094] An attitude decoder is used to decode the spatiotemporal encoded features to obtain the motion attitude;

[0095] A footstep state decoder is used to decode the spatiotemporal encoded features to obtain the footstep contact state.

[0096] Specifically, the decoder module in this embodiment consists of a posture decoder and a foot state decoder. The two decoders decode the spatiotemporal encoded features to obtain the motion posture and foot contact state of the completed motion. In one implementation, the posture decoder and the foot state decoder each consist of two linear layers. The posture decoder reduces the spatiotemporal features from 1024 dimensions to 128 dimensions, and the foot contact state decoder reduces the spatiotemporal features from 1024 dimensions to 2 dimensions.

[0097] For ease of understanding, this embodiment provides a data processing procedure for a spatiotemporal self-attention network. The spatiotemporal self-attention network outputs data in a non-autoregressive manner. For the transition frame of the completed motion intermediate frame T, spherical linear interpolation is first used to fill the key frame to frame T, which is then input into the spatiotemporal self-attention network. The spatiotemporal self-attention network calculates the motion sequence of length T of the completed frame through one forward propagation. Finally, the motion posture and foot contact state of the completed frame T are decoded and output by the attitude decoder and foot contact decoder.

[0098] In one implementation, the method for determining the loss function used by the spatiotemporal self-attention network during training includes:

[0099] Step S1: Obtain the reconstruction loss, wherein the reconstruction loss is used to converge the gap between the true motion pose and the predicted motion pose;

[0100] Step S2: Obtain the foot contact loss, wherein the foot contact loss is used to converge the difference between the actual foot contact state and the predicted foot contact state.

[0101] Step S3: Obtain the root locus loss, wherein the root locus loss is used to converge the difference between the true root locus and the reconstructed root locus;

[0102] Step S4: Obtain the joint position loss, wherein the joint position loss is used to converge the difference between the actual joint position and the predicted joint position calculated based on the predicted motion posture.

[0103] Step S5: Determine the loss function based on the reconstruction loss, the foot contact loss, the root trajectory loss, and the joint position loss.

[0104] Specifically, the spatiotemporal self-attention network in this embodiment is pre-trained. During training, a predefined loss function is used to converge the gap between the real and predicted data. The loss function in this embodiment includes multiple loss terms: reconstruction loss, foot contact loss, root trajectory loss, and joint position loss. The reconstruction loss penalizes the joint rotation quaternion at each time step, thereby converging the gap between the real and predicted motion postures. The foot contact loss penalizes the predicted foot contact state at each time step, thereby converging the gap between the real and predicted foot contact states. The root trajectory loss and joint position loss are used to converge the gap between the real and reconstructed root trajectories, and the gap between the real and predicted joint positions, respectively. Therefore, the root trajectory loss and joint position loss ensure that the model predicts the joint nodes in the correct positions, optimizing the root trajectory and the position of the end joints during the motion process.

[0105] For example, based on the structure and objective of the spatiotemporal self-attention network, multiple loss terms are defined, including reconstruction loss. Foot contact loss Root locus loss Joint position loss

[0106] Reconstruction losses The mean squared error (MSE) is used to calculate the reconstruction loss, which enables the spatiotemporal self-attention network to generate motion sequences consistent with real data. The reconstruction loss is shown in the formula:

[0107]

[0108] Where T is the sequence length, For Ground Truth, Y t To predict motion posture, the reconstruction loss penalizes the joint rotation quaternion at each time step.

[0109] Foot contact loss The cross-entropy loss is used to ensure that the spatiotemporal self-attention network predicts the correct foot contact state. The foot contact loss is shown in the formula:

[0110]

[0111] in, For the Ground Truth in the foot contact state, S c,t Predicted foot contact state; foot contact loss penalizes the predicted foot contact state at each time step;

[0112] Root locus loss and joint position loss Using mean squared error in calculations ensures that the joint nodes predicted by the spatiotemporal self-attention network are in the correct positions, and the root joint loss... and joint position loss As shown in the formula:

[0113]

[0114]

[0115] in, For the root locus Ground Truth, R t For the root trajectory we reconstructed, For the key point location Ground Truth, P t The network predicts the motion posture and then calculates the position coordinates. Figure 8 The experiment diagram shows a comparison of motion generation with and without node location-independent loss supervision, highlighting the effectiveness of root trajectory loss. and joint position loss It greatly optimizes the root trajectory and the position of the end joints during the motion process.

[0116] The total loss function of the spatiotemporal self-attention network is shown in the following formula:

[0117]

[0118] Here, λ balances the weights of the main loss and the auxiliary loss, and through extensive experiments, it has been empirically set to le-5.

[0119] For ease of understanding, this embodiment provides an actual training process for a spatiotemporal self-attention network as a reference: This embodiment uses Python as the programming language, an Intel(R) i7 8700K CPU (4.8GHz, 8 cores) and 32GB RAM, running on a Windows 10 64-bit system. This embodiment uses three different datasets. The dance dataset contains 123 modern dance segments; 80% of these segments, totaling 98 dance segments and 93,347 frames, were selected for training, and the remaining 25 dance segments, totaling 20,897 frames, were used for testing. The LaFAN1 dataset used 30Hz MOCAP sampling, recording 496,672 motion frames. Motion frames from subject 5 were selected as the test set, and the other subjects were selected as the training set. The martial arts dataset consisted of 10 actors performing contemporary martial arts, resulting in 10 segments with a total of 29,183 motion frames. In the experiment, the last 9 participants (23,066 frames) were used as the training set, and the first participant (6,117 frames) were used as the test set. The training and test datasets were divided into segments of different lengths using a sliding window, with a sliding window length of 50 frames and a sliding interval of 10 frames. During the training phase, the network was trained with fixed training lengths of 10 frames, 30 frames, and 50 frames. The spatiotemporal self-attention network was trained using the Adam optimizer with a learning rate of 0.001 and a maximum training epoch of 1000.

[0120] In one implementation, the constraints of the root trajectory reconstruction algorithm are determined based on the motion posture and the foot contact state. The operation principle of the root trajectory reconstruction algorithm is to determine the position of the root node in the (t+1)th frame by subtracting the displacement of the touching foot in the tth frame from the position of the root node in the tth frame.

[0121] Specifically, this embodiment obtains the reconstructed root trajectory through a preset root trajectory reconstruction algorithm. Since the root trajectory reconstruction algorithm uses the motion posture and foot contact state output by the spatiotemporal self-attention network as constraints, it can reconstruct the matching root trajectory through reasoning, and use the reconstructed root trajectory as a loss term in the loss function to constrain the spatiotemporal self-attention network. The principle of the root trajectory reconstruction algorithm is actually to determine the position of the root node in frame t+1 by subtracting the displacement of the touching foot in frame t from the position of the root node in frame t. That is, by moving the origin of the relative coordinate system, the displacement of the touching foot in the relative coordinate system is compensated. Figure 10 As shown, the root node is located at the geometric center of the human body model.

[0122] For example, the root trajectory reconstruction algorithm first uses forward kinematics (FK) to calculate the joint positions in a relative coordinate system. The relative coordinate system is defined as a coordinate system with the root joint as its origin. The calculation of the joint position coordinates can be described as follows:

[0123] P = FK(O r ,Y);

[0124] Where Or represents the origin of the relative coordinate system, P represents the inferred position of all joints in the relative coordinate system except the root joint, and Y is the quaternion of the network's predicted motion posture.

[0125] The displacement of the contact foot in each frame is calculated using the joint position P in the relative coordinate system. The displacement of the contact foot in the relative coordinate system can be defined as follows:

[0126] ΔP c,t =P c,t+1 -P c,t ;

[0127] Among them, P c，t+1 and P c，t The positions of the grounding feet in frame t+1 and frame t are respectively, and ΔP c，t Let be the displacement of the touching foot in frame t. The touching foot is determined as the foot joint closest to the ground in the world coordinate system.

[0128] Finally, the displacement ΔR of the root joint is reconstructed. t Since the contact foot is immovable, based on the relativity of motion, the displacement of the contact foot in the relative coordinate system can be compensated by moving the origin of the relative coordinate system, thus completing the transition from the relative coordinate system to the world coordinate system. Therefore, root locus reconstruction can actually be described as:

[0129] R t+1 =R t -ΔP c,t ;

[0130] Where Rt is the position of the root node in frame t, and ΔP c，t Let R be the displacement of the grounding foot in frame t. t+1 The position of the root node at frame t+1 (e.g.) Figure 4 (As shown).

[0131] In one implementation, the method further includes:

[0132] Step S400: Store the completion action data as BVH action data, and import the BVH action data into the Blender skeleton model for display;

[0133] Step S500: Obtain several preset quantitative evaluation indicators, and evaluate the completion action data according to each of the quantitative evaluation indicators.

[0134] Specifically, the completion motion data output by the spatiotemporal self-attention network is actually the rotation quaternion information of the joints. This information is converted into Euler angle information of the joints and then stored as motion data in BVH format. A Python script then imports the BVH motion data into the Blender skeletal model to display the final result. The skeletal model is a character model built in Blender and its bones are bound together. Simultaneously, a predefined quantitative evaluation metric is used to evaluate the final result.

[0135] In one implementation, the quantitative evaluation metrics include L2P, L2Q, and NPSS evaluation metrics.

[0136] The calculation formulas for L2P and L2Q evaluation indicators are shown below:

[0137]

[0138]

[0139] Where s is the motion sequence of the test set D, and T is the transition length. For the joint quaternion Ground Truth, Y t Predict joint quaternions for the network, Joint spatial position coordinates Ground Truth, P t Predict joint spatial coordinates for the network.

[0140] The calculation formula for the NPSS evaluation index is as follows:

[0141]

[0142]

[0143]

[0144] Where, X i，j [f] and Y i，j [f] is the squared magnitude spectrum of the discrete Fourier transform coefficients of the i-th sequence with feature dimension j.

[0145] Compared with the prior art, the present invention has the following advantages:

[0146] 1. This invention synthesizes high-quality motion segments on complex motion datasets such as dance and martial arts, achieving better results on various evaluation criteria and resulting in higher quality character animations.

[0147] 2. This invention achieves precise control of complex dance movements through precise foot contact control.

[0148] 3. This invention can eliminate the sliding phenomenon in movement, and not only achieves better results in complex movements, but can also generate realistic and natural simple movements, such as walking and running. The technology has high robustness.

[0149] Based on the above embodiments, the present invention also provides a complex motion keyframe animation completion system based on neural networks, such as... Figure 11 As shown, the system includes:

[0150] The definition and embedding module 01 is used to acquire keyframe information and foot contact information of the motion sequence, and determine a hybrid embedding representation based on the keyframe information and the foot contact information, wherein the hybrid embedding representation is used to reflect the summation result of the keyframe information and the foot contact information after they are respectively embedded in a high-dimensional space;

[0151] The filling and prediction module 02 is used to interpolate and fill the keyframes to obtain the filled motion sequence; the hybrid embedding representation and the filled motion sequence are input into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted by the spatiotemporal self-attention network.

[0152] The reconstruction and completion module 03 is used to input the motion posture and the foot contact state into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state; and to complete the key frames according to the motion posture and the reconstructed root trajectory between key frames.

[0153] In one implementation, the reconstructed root trajectory is a loss term of the loss function of the spatiotemporal self-attention network, wherein the constraints of the root trajectory reconstruction algorithm are determined based on the motion posture and the foot contact state, and the operation principle of the root trajectory reconstruction algorithm is to determine the position of the root node in the (t+1)th frame by subtracting the displacement of the touching foot in the tth frame from the position of the root node in the tth frame.

[0154] In one implementation, the system further includes:

[0155] The display and evaluation module 04 is used to store the completion action data as BVH action data, import the BVH action data into the Blender skeleton model for display, obtain a number of preset quantitative evaluation indicators, and evaluate the completion action data according to each of the quantitative evaluation indicators.

[0156] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 12As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a neural network-based method for completing complex motion keyframe animations. The display screen can be an LCD screen or an e-ink screen.

[0157] Those skilled in the art will understand that Figure 11 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors. The programs include instructions for performing a neural network-based method for completing complex motion keyframe animations.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0160] In summary, this invention discloses a method and system for keyframe animation completion of complex motion based on neural networks. The method includes: acquiring keyframe information and foot contact information of a motion sequence; determining a hybrid embedding representation based on the keyframe information and the foot contact information, wherein the hybrid embedding representation reflects the summation result of embedding the keyframe information and the foot contact information into a high-dimensional space; interpolating and filling the keyframes to obtain the filled motion sequence; inputting the hybrid embedding representation and the filled motion sequence into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted by the spatiotemporal self-attention network; inputting the motion posture and the foot contact state into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state; and completing the keyframes based on the motion posture and the reconstructed root trajectory between the keyframes. This invention achieves precise foot control by providing different foot contact information, providing a stable and high-quality method for keyframe animation completion of complex movements, eliminating the slippage problem in the motion generation process, and exhibiting strong robustness. This solves the problem that existing technologies using deep learning to perform complex motion generation tasks suffer from severe slippage and ghosting issues, which lead to the need for significant manpower and resources for motion correction in the later stages.

[0161] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A neural network-based complex motion keyframe animation completion method, characterized in that, The method includes: Acquire keyframe information and foot contact information of the motion sequence, and determine a hybrid embedding representation based on the keyframe information and the foot contact information, wherein the hybrid embedding representation is used to reflect the summation result of the keyframe information and the foot contact information after they are respectively embedded in a high-dimensional space; Interpolation padding is applied between keyframes to obtain the padded motion sequence; The motion sequence after the hybrid embedding representation and padding is input into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted by the spatiotemporal self-attention network. The motion posture and the foot contact state are input into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state; The keyframes are completed based on the motion poses between keyframes and the reconstructed root trajectory.

2. The neural network-based complex motion keyframe animation completion method of claim 1, wherein, Determining a hybrid embedding representation based on the keyframe information and the footstep contact information includes: Obtain a preset keyframe encoding matrix, and embed the keyframe information into a high-dimensional space according to the keyframe encoding matrix to obtain a keyframe embedding representation. Obtain a preset foot contact encoding matrix, and embed foot contact embedding information into a high-dimensional space according to the foot contact encoding matrix to obtain a foot contact embedding representation; The keyframe embedding representation and the footstep contact embedding representation are summed to obtain the hybrid embedding representation.

3. The neural network-based complex motion keyframe animation completion method of claim 1, wherein, The interpolation fill is a spherical linear interpolation fill.

4. The neural network-based complex motion keyframe animation completion method of claim 1, wherein, The spatiotemporal self-attention network includes: A spatiotemporal encoder module is used to determine the spatiotemporal coding features of the motion sequence based on the padded motion sequence and the hybrid embedding representation; The decoder module is used to decode the spatiotemporal encoded features to obtain the motion posture and the foot contact state.

5. The neural network-based complex motion keyframe animation completion method of claim 4, wherein, The spatiotemporal encoder module includes: A spatial self-attention module is used to extract spatial coding features of the motion sequence based on the filled motion sequence and a preset spatial position embedding representation, wherein the spatial position embedding representation is used to distinguish different joints of the human skeleton. The temporal self-attention module is used to acquire a merged data representation based on the spatial coding features, a preset temporal position embedding representation, and the hybrid embedding representation, wherein the temporal position embedding representation is used to distinguish different frames of the motion sequence; and to extract the spatiotemporal coding features of the motion sequence based on the merged data representation.

6. The neural network-based complex motion keyframe animation completion method of claim 5, wherein, The method for generating the spatial location embedding representation includes: Obtain the joint markers corresponding to each joint of the human skeleton; The joint markers are embedded into a high-dimensional space using a preset matrix to obtain the spatial position embedding representation.

7. The method for completing complex motion keyframe animations based on neural networks according to claim 4, characterized in that, The decoder module includes: An attitude decoder is used to decode the spatiotemporal encoded features to obtain the motion attitude; A footstep state decoder is used to decode the spatiotemporal encoded features to obtain the footstep contact state.

8. The method for completing complex motion keyframe animations based on neural networks according to claim 4, characterized in that, The method for determining the loss function used during training of the spatiotemporal self-attention network includes: Obtain the reconstruction loss, wherein the reconstruction loss is used to converge the gap between the true motion pose and the predicted motion pose; Obtain the foot contact loss, wherein the foot contact loss is used to converge the difference between the actual foot contact state and the predicted foot contact state; Obtain the root locus loss, wherein the root locus loss is used to converge the difference between the true root locus and the reconstructed root locus; Obtain the joint position loss, wherein the joint position loss is used to converge the difference between the true joint position and the predicted joint position calculated based on the predicted motion posture. The loss function is determined based on the reconstruction loss, the foot contact loss, the root trajectory loss, and the joint position loss.

9. The method for completing complex motion keyframe animations based on neural networks according to claim 8, characterized in that, The constraints of the root trajectory reconstruction algorithm are determined based on the motion posture and the foot contact state. The operation principle of the root trajectory reconstruction algorithm is to determine the position of the root node in the (t+1)th frame by subtracting the displacement of the touching foot in the tth frame from the position of the root node in the tth frame.

10. The method for completing complex motion keyframe animations based on neural networks according to claim 1, characterized in that, The method further includes: The completion motion data is stored as BVH motion data, and the BVH motion data is imported into the Blender skeleton model for display. Obtain several preset quantitative evaluation indicators, and evaluate the completion action data according to each of the quantitative evaluation indicators.

11. A complex motion keyframe animation completion system based on neural networks, characterized in that, The system includes: The definition and embedding module is used to acquire keyframe information and foot contact information of the motion sequence, and to determine a hybrid embedding representation based on the keyframe information and the foot contact information. The hybrid embedding representation is used to reflect the summation result of embedding the keyframe information and the foot contact information into a high-dimensional space respectively. The filling and prediction module is used to interpolate and fill the keyframes to obtain the filled motion sequence; the hybrid embedding representation and the filled motion sequence are input into a pre-trained spatiotemporal self-attention network to obtain the motion posture and foot contact state predicted based on the spatiotemporal self-attention network. The reconstruction and completion module is used to input the motion posture and the foot contact state into a preset root trajectory reconstruction algorithm to obtain the reconstructed root trajectory corresponding to the foot contact state; and to complete the keyframes based on the motion posture and the reconstructed root trajectory between keyframes.

12. The complex motion keyframe animation completion system based on neural networks according to claim 11, characterized in that, The reconstructed root trajectory is a loss term of the loss function of the spatiotemporal self-attention network. The constraints of the root trajectory reconstruction algorithm are determined based on the motion posture and the foot contact state. The operation principle of the root trajectory reconstruction algorithm is to determine the position of the root node in the (t+1)th frame by subtracting the displacement of the touching foot in the tth frame from the position of the root node in the tth frame.

13. The complex motion keyframe animation completion system based on neural networks according to claim 11, characterized in that, The system also includes: The display and evaluation module is used to store the completion action data as BVH action data, import the BVH action data into the Blender skeleton model for display, obtain a number of preset quantitative evaluation indicators, and evaluate the completion action data according to each of the quantitative evaluation indicators.

14. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the neural network-based complex motion keyframe animation completion method as described in any one of claims 1-10; the processors are used to execute the programs.

15. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are applicable to be loaded and executed by a processor to implement the steps of the neural network-based complex motion keyframe animation completion method as described in any one of claims 1-10.

Images