Animation generation method and system based on contrastive learning

Abstract

The present invention discloses an animation generation method and system based on contrastive learning, which relates to the technical field of human animation generation, including: contrastive learning uses meta-animation with strong feature semantics to encode cross-modal semantics, informative geometric features and movement principles in postures; parameterizes the latent space through VAE with spatiotemporal representation, and generates transitions of coherent animations without supervision. The present invention matches mesh animations with continuous semantic labels through context-related clues of adjacent frames, and guides the alignment between semantics and mesh animations through strong geometric priors. Features are extracted through a semantically guided encoder to improve the intra-class compactness and inter-class separability of animations. By learning the latent space distribution to simultaneously represent the spatial and temporal clues in the animation, realistic animations with high flexibility are generated under the control of spatiotemporal semantics.

Description

Technical Field

[0001] The present invention relates to the technical field of human body animation generation, and in particular to an animation generation method and system based on contrast learning. Background Art

[0002] Animation synthesis is a long-standing task in computer graphics and computer animation. In particular, generating semantically controlled sequences with poses and shapes is a challenging task. In recent years, a large number of research works have found that there are four forms of animation representation, including partial pose sequences, meshes (SMPL), fixed human models, and implicit functions. Meshes are limited in their expressiveness due to the limited scale of vertices. Implicit functions require high computational costs. Although fixed human model methods can synthesize clothed people with flexible vertex scales. However, these methods have low generalization ability in generating animations in datasets. Using primitive numerical analysis to model human motion and then tracking 3D human motion through the model, such methods have difficulty in generating a variety of actions. Network flows learn rich and varied latent spaces to encode motion, but these methods are not directly applicable to learning controllable generative models. Frontier research on diffusion models also has the ability to generate models. These techniques can provide higher quality samples, but they are inflexible in terms of computational time and high dimensionality of the latent code and cannot be actively edited and controlled as required by the user.

[0003] Therefore, how to provide a method for generating human body animation that can be generated by semantic control and has spatiotemporal representation is an urgent problem that those skilled in the art need to solve. Summary of the invention

[0004] In view of this, the present invention proposes an animation generation method and system based on contrastive learning. By approximating the continuous mapping function between semantics and animation through a contrastive learning framework, human body animation can be generated by semantic control, and human body animation with spatiotemporal representation can be generated. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0005] An animation generation method based on contrastive learning, comprising:

[0006] Step (1): obtain input text and obtain text features through a text encoder; obtain input meta-animation and obtain geometric features through a geometric encoder; obtain joint point information of human body motion and obtain motion features through a motion encoder;

[0007] Step (2): concatenate the obtained text features, geometric features and motion features, and extract features through a semantically guided encoder;

[0008] Step (3): The features extracted in step (2) are input into a novel temporal position encoder connected with the GRU to retain the semantics related to high-frequency action cues;

[0009] Step (4): Parameterize the latent space through a VAE with spatiotemporal representation to generate coherent animation transitions without supervision.

[0010] Optionally, in step (1), the text encoder searches for corresponding actions by setting a learnable semantic embedding vector and using the label as an index.

[0011] Optionally, in step (1), the geometric encoder includes converting the meta-animation into a feature vector through a nonlinear mapping function R(), encoding the vertex information of the meta-animation into a tensor, converting the mesh motion into a feature space, and aligning it with the semantic label;

[0012]

[0013] in, is a LSTM long short-term memory neural network, V represents the vertex of the meta-animation, ψ(V) is a tensor, is the function value at point V, represents the SMPL parameters of the meta-animation, Ω represents the weight of the bone binding, and (β,θ) is the pose.

[0014] Optionally, in step (1), the motion features of the human skeleton, including the motion tree, joint coordinates and vertex coordinates, are extracted through a motion encoder, the posture (β, θ) is encoded into the feature space through an MLP network, and the generation of the animation is controlled to encode the continuous posture φ(P) by modifying the size P of the key points of the body;

[0015]

[0016] in, is the posture function.

[0017] Optionally, the step of extracting features in step (2) includes inputting the spliced ​​information into the encoding module network for training until the model converges, decomposing it into semantic flow and animation flow, and obtaining the time change law of semantic objects with interactive relationships.

[0018] Optionally, in step (3), the new temporal position encoder includes encoding the temporal order clues of the animation through position encoding of a periodic sine function, first providing the new temporal position encoder with a cross-modal data stream including posture, semantics and meta-animation, and then processing the position encoding module between the geometry-kinematic fusion layer and the animation encoder layer to calculate the extension calculated from the previous position encoding to make temporally smooth frames with semantic labels.

[0019] Optionally, the loss function including contrastive learning in step (3) is constructed as:

[0020]

[0021] Among them, sim represents the cosine similarity of the product of two feature points, P represents the skeleton key point information, M represents the geometric information of the mesh, and the symbol τ is a network parameter.

[0022] Optionally, in step (4), the spatiotemporal representation is generated through long-term motion under multi-semantic conditions, spatial semantics and temporal semantics are integrated, spatial semantics are used to control the simultaneous generation of multiple actions, and temporal semantics are refined into multiple actions.

[0023] Optionally, an animation generation system based on contrastive learning, comprising:

[0024] Acquisition module: obtain multimodal information;

[0025] Information acquisition module: obtain text information, meta-animation information, and joint point information;

[0026] The first building block: used to build the coding module;

[0027] The first training module: input multimodal information into the encoding module network for training until the model converges and decomposes it into semantic flow and animation flow, and obtains the temporal change law of semantic objects with interactive relationships;

[0028] The first processing module: The features extracted from the multimodal information are concatenated and input into a novel temporal position encoder connected with the GRU to retain the semantics related to high-frequency action cues;

[0029] The second building block: used to build a model of human motion;

[0030] The second training module is used for long-term motion generation under multi-semantic conditions. It integrates spatial semantics and temporal semantics, uses spatial semantics to control the simultaneous generation of multiple actions, refines temporal semantics into multiple actions, and improves the conversion between multiple semantics.

[0031] Human animation generation module: Generates the interactions between people and objects from the latent variables of information, generating animations with semantically reasonable and physically plausible deformations and kinematics.

[0032] It can be seen from the above technical solutions that, compared with the prior art, the present invention proposes an animation generation method and system based on contrastive learning. The present invention inputs multimodal information into the encoding module network for training until the model converges, decomposes it into semantic streams and animation streams, obtains the temporal change law of semantic objects with interactive relationships, and generates long-term motion under multi-semantic conditions. In order to generate semantically coherent animations, the fusion of spatial semantics and temporal semantics is used, that is, spatial semantics is used to control the simultaneous generation of multiple actions, while temporal semantics is refined into individual actions, improving the conversion between multiple semantics.

[0033] Semantic information is used to improve the fidelity of animations in order to mimic real human behaviors. To fill the gap between animation and semantics, a new data-driven paradigm is proposed: spatiotemporal representation based on contrastive learning. The whole method can be decomposed into two subtasks: cross-modal contrastive learning and spatiotemporal representation. Cross-modal contrastive learning matches mesh animations with continuous semantic labels through context-related clues of adjacent frames. To strengthen the geometric details, a series of meta-animations are designed as strong geometric priors to guide the alignment between semantics and mesh animations.

[0034] In order to maintain the high discriminative power of cross-modal features (including posture, geometric details and semantics), the present invention extracts features through a semantically guided encoder, thereby improving the intra-class compactness and inter-class separability of animations, and can simultaneously represent the spatial and temporal clues in animations by learning latent space distribution. It achieves the generation of realistic animations with high flexibility under the control of spatiotemporal semantics, and can also generate new categories of animations that are not in the dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0037] Figure 2 An animated schematic diagram is generated for the human body of the present invention.

[0038] Figure 3 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Embodiment 1 of the present invention discloses an animation generation method based on contrastive learning, such as Figure 1 As shown, including:

[0041] Step (1): Get the input text and obtain the text information through the text encoder; propose a learnable semantic embedding vector and use the label as an index to find the corresponding action. Therefore, the generation of different categories of actions can be controlled.

[0042] Step (2): Obtain the input primitive animation and obtain geometric information through a geometric encoder; the geometric encoding includes: proposing a geometric encoder to convert the primitive animation into a feature vector in a nonlinear manner. The primitive animation is converted into a feature vector through a nonlinear mapping function R(). In order to obtain the features of the mesh animation, the vertex V information of the primitive animation is encoded into a tensor.

[0043]

[0044] Among them, the long short-term memory (LSTM) neural network is used The mesh motion is further transformed into feature space to adaptively align with the semantic labels. V represents the vertices of the meta-animation, Represents the SMPL parameters of the meta-animation, and Ω represents the weight of the bone binding.

[0045] Step (3): Obtain the joint point information of human motion and obtain motion information through the motion encoder; a kinematic encoder is proposed to extract the motion features of human skeleton, including motion tree, joint coordinates and vertex coordinates. Similar to the geometric encoder, the pose (β, θ) is encoded into the feature space through the MLP network and the generation of animation is controlled. In order to obtain kinematic features, the size P of the body key points is modified to encode the continuous pose φ(P):

[0046]

[0047] By applying the posture function Get the key point information of the human body.

[0048] Step (4): The obtained text features, geometric features, and motion features are concatenated and features are extracted through a semantically guided encoder. The novel temporal position encoder includes encoding the temporal order clues of the animation through position encoding of a periodic sine function. The purpose is to retain the semantic expression part of the animation. First, a cross-modal data stream including posture, semantics, and meta-animation is provided to the novel temporal position encoder. In addition, in order to maintain semantic fluency in the temporal dimension, it is recommended to smooth the frames with semantic labels in time. Specifically, the position encoding module is processed between the geometry-kinematic fusion layer and the animation encoder layer to calculate the extension calculated from the previous position encoding.

[0049] When encoding action sequences, a new temporal position encoder is introduced. Sequential cues are the key to expressing action sequences. The timing cues of animation are encoded through the position encoding of periodic sine functions, thereby retaining the semantic expression components of animation. First, a cross-modal data stream is provided to the temporal position encoder, including posture P, semantics S, and vertices V in meta-animation. At the same time, in order to maintain the fluency of semantics in the temporal dimension, frames with semantic labels are smoothed in time by selecting. That is, the position encoding module is processed between the geometry-kinematics fusion layer and the animation encoder layer, and the position encoding is expanded to:

[0050] G t (x t )=ξ(q·[x t-1 ,x t ]),;

[0051]

[0052] Among them, ξ(,) represents the sigmoid function, q represents the weight of the temporal encoder, [,] represents the vector connection operation, G t (,) represents the forget gate in GRU, x t The representation is initialized from the cross-modal positional encoding. In this way, semantically relevant temporal features and coherent animations can be obtained globally. The latent space is guided by the temporal position encoder network to encode the temporal and spatial cues of the animation, which can fully preserve the semantically relevant high-frequency details of the animation.

[0053] Step (5): The extracted features are input into a novel temporal position encoder connected with the position encoder GRU to retain the semantics related to high-frequency action cues. The loss function of contrastive learning is constructed as:

[0054]

[0055] Among them, sim represents the cosine similarity of the product of two feature points, P represents the skeleton key point information, M represents the geometric information of the mesh, and the symbol τ is a network parameter.

[0056] Step (6): In order to generate coherent animations, the latent space is parameterized through VAE with spatiotemporal representation, so as to generate coherent animation transitions without supervision. Spatiotemporal representation is the long-term motion generation under multi-semantic conditions. In order to generate semantically coherent animations, the fusion of spatial semantics and temporal semantics is used at the same time. In other words, spatial semantics is used to control the simultaneous generation of multiple actions, while temporal semantics is refined into individual actions, and semantics improves the conversion between multiple semantics.

[0057] In order to compare the adopted approach with other high-performance human shape estimation methods in a quantitative manner, the performance of the state-of-the-art methods is shown: Motion CLIP, Actor and Avatar CLIP. The data and results are shown in Table 1.

[0058] Table 1 Error comparison results of animation vertices

[0059] method FID Acc Div Multimodal MotionCLIP 0.13 95.1 6.78 2.54 Actors 0.13 94.9 6.83 2.56 AvatarCLIP 0.10 96.0 4.83 1.46 Ours 0.10 96.1 6.85 6.85

[0060] Embodiment 2 of the present invention discloses an animation generation system based on contrastive learning, such as Figure 3 As shown, including:

[0061] Acquisition module: obtain multimodal information.

[0062] Information acquisition module: obtain text information, geometric information, and motion information.

[0063] The first building block: used to build the encoding module.

[0064] The first training module: By inputting multimodal information into the encoding module network for training until the model converges, it is decomposed into semantic flow and animation flow to obtain the time change law of semantic objects with interactive relationships.

[0065] The first processing module: The feature input position encoding extracted from multimodal information is connected to a novel temporal position encoder connected to GRU to retain the semantics related to high-frequency action cues.

[0066] The second building block: used to build a model of human motion.

[0067] The second training module is used for long-term motion generation under multi-semantic conditions. In order to generate semantically coherent animations, the fusion of spatial semantics and temporal semantics is used at the same time. In other words, spatial semantics is used to control the simultaneous generation of multiple actions, while temporal semantics is refined into individual actions. Semantics improves the conversion between multiple semantics.

[0068] Human animation generation module: Generates interactions between people and objects from latent variables of information, with semantically and physically plausible deformation and kinematic animations.

[0069] Embodiment 3 of the present invention discloses a computer storable medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the human body animation generation method described above are implemented.

[0070] The embodiment of the present invention discloses a method and system for generating animation based on contrastive learning. The present invention approximates a continuous mapping function between semantics and animation through a contrastive learning framework. The contrastive learning utilizes meta-animation with strong feature semantics, which encodes cross-modal semantics, informative geometric features, and motion principles in postures; at the same time, in order to generate coherent animations, the latent space is parameterized by VAE with spatiotemporal representation, which generates transformations of coherent animations without supervision. The present invention matches mesh animations with continuous semantic labels through context-related clues of adjacent frames. In order to enhance geometric details, a series of meta-animations are designed as strong geometric priors to guide the alignment between semantics and mesh animations. In addition, in order to maintain the high discriminative power of cross-modal features (including postures, geometric details, and semantics), a new spatiotemporal feature representation method is proposed, that is, features are extracted through a semantically guided encoder to improve the intra-class compactness and inter-class separability of animations. The model learns latent space distribution to simultaneously represent spatial and temporal clues in animations to produce realistic animations with high flexibility under spatiotemporal semantic control. Extensive experimental results and comprehensive evaluations show that the proposed method can produce realistic animations with high flexibility under spatiotemporal semantic control, and is also capable of generating new categories of animations that are not in the dataset.

[0071] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0072] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An animation generation method based on contrastive learning, characterized in that: include: Step (1): obtain input text and obtain text features through a text encoder; obtain input meta-animation and obtain geometric features through a geometric encoder; Obtain the joint point information of human body movement and obtain the movement characteristics through the motion encoder; Step (2): Merge the obtained text features, geometric features and motion features, and extract features through a semantically guided encoder; Step (3): The features extracted in step (2) are input into a novel temporal position encoder connected with the GRU to retain the semantics related to high-frequency action cues; When encoding action sequences, a new temporal position encoder is introduced. Sequential clues are the key to expressing action sequences. The timing clues of animation are encoded through the position encoding of periodic sine functions, thereby retaining the semantic expression components of animation. First, a cross-modal data stream is provided to the temporal position encoder, including posture P, semantics S, and vertices V in meta-animation. At the same time, in order to maintain the fluency of semantics in the temporal dimension, frames with semantic labels are smoothed in time, that is, the position encoding module is processed between the geometry-kinematics fusion layer and the animation encoder layer, and the position encoding is expanded as follows: G t (x t )=ξ(q·[x t-1 ,x t ]); Among them, ξ(,) represents the sigmoid function, q represents the weight of the temporal encoder, [,] represents the vector connection operation, G t (,) represents the forget gate in GRU, x t The representation is initialized from the cross-modal position encoding, semantically relevant temporal features and coherent animations are obtained globally, and the latent space is guided by the temporal position encoder network to encode the temporal and spatial clues of the animation, fully preserving the high-frequency details related to the semantics of the animation; Step (4): Parameterize the latent space through a VAE with spatiotemporal representation to generate coherent animation transitions without supervision.

2. The animation generation method based on contrastive learning according to claim 1, characterized in that: In step (1), the text encoder sets a learnable semantic embedding vector and uses the label as an index to find the corresponding action.

3. The animation generation method based on contrastive learning according to claim 1, characterized in that: In step (1), the geometric encoder includes converting the meta-animation into a feature vector through a nonlinear mapping function R(), encoding the vertex information of the meta-animation into a tensor, converting the mesh motion into a feature space, and aligning it with the semantic label; in, is a LSTM long short-term memory neural network, V represents the vertex of the meta-animation, ψ(V) is a tensor, is the function value at point V, represents the SMPL parameters of the meta-animation, Ω represents the weight of the bone binding, and (β, θ) is the pose.

4. The animation generation method based on contrastive learning according to claim 1, characterized in that: In step (1), the motion features of human skeleton, including motion tree, joint coordinates and vertex coordinates, are extracted through the motion encoder, the posture (β, θ) is encoded into the feature space through the MLP network, and the generation of animation is controlled by modifying the size P of the key points of the body to encode the continuous posture φ(P); in, is the posture function.

5. The method for generating animation based on contrastive learning according to claim 1, characterized in that: The step of extracting features in step (2) includes inputting the merged information into the encoding module network for training until the model converges, decomposing it into semantic flow and animation flow, and obtaining the time change law of semantic objects with interactive relationships.

6. The animation generation method based on contrastive learning according to claim 1, characterized in that: In step (3), the novel temporal position encoder includes encoding the temporal order clues of the animation through position encoding of a periodic sine function, first providing the novel temporal position encoder with a cross-modal data stream including posture, semantics and meta-animation, and then processing the position encoding module between the geometry-kinematics fusion layer and the animation encoder layer to calculate the extension calculated from the previous position encoding to make the frames with semantic labels smooth in time.

7. The method for generating animation based on contrastive learning according to claim 6, characterized in that: The loss function including contrastive learning in step (3) is constructed as: Among them, som represents the cosine similarity of the product of two feature points, P represents the skeleton key point information, M represents the geometric information of the mesh, and the symbol τ is a network parameter.

8. The method for generating animation based on contrastive learning according to claim 1, characterized in that: In step (4), the spatiotemporal representation is generated through long-term motion under multi-semantic conditions, spatial semantics and temporal semantics are integrated, spatial semantics are used to control the simultaneous generation of multiple actions, and temporal semantics are refined into multiple actions.

9. An animation generation system based on contrastive learning, characterized in that: include: Acquisition module: obtain multimodal information; Information acquisition module: obtain text information, meta-animation information, and joint point information; The first building block: used to build the coding module; The first training module: input multimodal information into the encoding module network for training until the model converges and decomposes it into semantic flow and animation flow, and obtains the temporal change law of semantic objects with interactive relationships; The first processing module: The features extracted from the multi-modal information are merged and input into a new temporal position encoder connected with the GRU to retain the semantics related to high-frequency action cues; The second building block: used to build a model of human motion; The second training module is used for long-term motion generation under multi-semantic conditions. It integrates spatial semantics and temporal semantics, uses spatial semantics to control the simultaneous generation of multiple actions, refines temporal semantics into multiple actions, and improves the conversion between multiple semantics. Human animation generation module: The interaction between people and objects is generated from the latent variables of information, generating animations with semantically reasonable and physically plausible deformations and kinematics.

Images