Motion level decoupling representation and analysis method based on visual language large model and application
By employing a motion hierarchy decoupling representation method based on a large visual language model, and utilizing skeleton graph structure and high-order Bézier curve parameterization modeling, the problems of error accumulation and motion distortion in the generation of motion sequences from text description are solved, achieving higher accuracy and smoother motion generation.
Patent Information
- Application Number
- CN202511031254.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-25
- Publication Date
- 2025-11-07
AI Technical Summary
Existing text-to-motion sequence generation models suffer from the problems of accumulated generation errors and motion distortion when faced with the complexity and variability of motion sequences, especially in autoregressive and direct generation methods, which struggle to generate natural and realistic human motion.
A motion hierarchy decoupling representation method based on a large visual language model is adopted. By constructing a skeleton graph structure constraint, the motion sequence is decomposed into pivot joints and end joints of the body region. Then, high-order Bézier curves are used for parametric modeling, and the motion sequence is generated by combining the graph structure and Bézier decoder.
It effectively reduces the complexity of motion generation, improves generation accuracy and smoothness, ensures the coordination of motion sequences in the temporal and spatial domains, and generates more natural and smooth motion.
Smart Images
Figure CN120912844A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of deep learning and multi-modal learning, and particularly relates to a motion level decoupling representation and analysis method based on a visual language large model and an application thereof. BACKGROUND
[0002] Accurate modeling of motion is crucial for understanding human and animal behavior, improving human-computer interaction, making movies and games, and developing intelligent systems that can predict and respond to human motion.
[0003] Generating motion sequences based on text descriptions as an important part of motion modeling aims to understand the action content of the text description and generate the corresponding motion sequence. This requires effective understanding of both the semantic features of natural language and motion sequences.
[0004] Although large language models and human motion analysis have developed rapidly in recent years, due to the inherent complexity and variability of motion sequences, generating natural and realistic human motion based on text descriptions remains challenging.
[0005] The current mainstream text description-motion sequence generation model adopts the following structure: first, a vector quantization variational autoencoder (VQ-VAE) is trained to align the discrete hidden features of text and motion sequences, and then a generative model is used to learn the mapping of the multi-modal aligned hidden feature space to the motion sequence, thereby obtaining a generative model from text to motion sequence.
[0006] Existing methods often use autoregressive-based generative models to iteratively generate or use Diffusion and other generative models to directly generate when learning the mapping of discrete hidden features to motion sequences. However, due to the spatial and temporal complexity of motion sequences, autoregressive generation often faces the problem of cumulative generation error, while direct generation methods need to face the problem of motion distortion caused by the large solution space of motion sequences, which has a great impact on motion synthesis based on text descriptions. SUMMARY
[0007] To solve the above problems of existing cross-domain pose estimation models, the application provides a motion level decoupling representation and analysis method based on a visual language large model, which is a cross-domain pose estimation method based on skeleton graph structure constraints.
[0008] The motion level decoupling representation and analysis method based on a visual language large model includes the following steps: S1: constructing a text encoder based on a large language model, a motion sequence feature extraction model, and performing visual-language multi-modal feature learning and feature space alignment learning; S2: Construct a hierarchical representation of the motion sequence, and perform Bezier parameterization representation on the time sequence motion trajectory of the body pivot joint; S3: Model the skeleton as a graph structure, which models the generation process of the skeleton as a graph node feature transformation process under the condition of edge connection determination, and then construct a triple sample pair of text description-skeleton graph structure sequence-pivot joint trajectory Bezier parameter; S4: Construct a motion sequence decoding model, including a trajectory generation model based on high-order Bezier curve representation and a skeleton generator based on graph structure, and train the model.
[0009] The step S1 specifically comprises: S11: Construct a text description-motion sequence training sample pair; S12: Construct a text encoder based on a large language model, and a visual-linguistic modal feature extraction alignment model; S13: Train the cross-modal feature alignment model based on the above training sample pair.
[0010] The visual-linguistic modal feature extraction alignment model in S12 includes a text encoder, a motion sequence encoder, and a hidden feature quantization module.
[0011] The hidden feature quantization module obtains the corresponding quantized hidden feature z e [i] by searching for the nearest element from the learnable codebook C for each feature vector z[i].
[0012] The objective function for training the cross-modal feature alignment model based on the above training sample pair in S13 is: Wherein α is the weight, and sg[*] represents gradient stop.
[0013] The step S2 specifically comprises: S21: Divide the joints into different regions according to the body parts, including the trunk part and the four limb parts; S22: According to the motion characteristics of each joint, the joints in each region are divided into a pivot joint and a plurality of end joints. The pivot joint represents the connection point between the region and the body trunk part; S23: Parameterize the motion trajectory of the pivot joint of each part of the body based on high-order Bezier curve, and represent it as a set of Bezier control points.
[0014] The joint trajectory parameterization modeling based on high-order Bezier curve specifically refers to parameterizing the motion trajectory t = {t1, t2, … tl The parametric modeling specifically includes: first initializing the Bezier control points, then iteratively calculating the gradient until convergence, and updating the Bezier control points
[0015] The gradient calculation method in the iterative gradient calculation is: Where B(u) represents the n-order Bezier control point set based on The reconstructed joint trajectory curve:
[0016] The step S3 specifically includes: S31: constructing a skeleton graph model based on the topological connection relationship of the key points; S32: constructing a ternary training sample according to the text-motion sequence dataset, including a text description, a skeleton graph structure sequence, and a high-order Bezier parameter of a pivotal joint time sequence motion trajectory; S33: loading a pre-trained text-motion feature extraction alignment model, and decoding the training motion sequence according to the constructed ternary sample.
[0017] The construction of the skeleton graph model based on the topological connection relationship of the joints specifically includes: first determining an edge connection matrix according to the connection relationship of the body parts, and then calculating a node feature for each joint.
[0018] The calculation of the node feature of each joint specifically includes: the node feature of the end node is the aggregation of the hidden feature and the corresponding joint position feature of the previous frame, and the pivotal node feature is the aggregation of the hidden feature, the corresponding joint position feature of the previous frame, and the motion vector of the node trajectory in the frame, specifically as follows: Where f c (*) represents feature aggregation.
[0019] The step S4 specifically includes: S41: constructing a Bezier decoder, which aims to generate pivotal point trajectory Bezier parameters from a hidden feature space; S42: constructing a skeleton generator, including a skeleton generator based on graph node feature transformation, a pivotal point motion intention fusion module, and a motion sequence iterative generation module.
[0020] The constructed Bezier decoder is composed of convolutional layers and fully connected layers, and the high-order Bezier parameters of the pivotal point motion trajectory are decoded from the hidden feature space by using the Bezier decoder, so as to reconstruct the pivotal point motion trajectory.
[0021] The constructed skeleton generator iteratively synthesizes the motion skeleton of each frame based on the skeleton of the previous frame, the hidden feature and the pivotal point time sequence motion trajectory obtained by the Bezier decoder.
[0022] Another object of the present application is to provide a motion level decoupling representation and analysis method based on a visual language large model for application in virtual character motion synthesis and improvement of motion generation accuracy.
[0023] The present application has the following advantages:
[0024] 1. The present application proposes a motion representation method based on hierarchical decomposition to solve the distortion and unsmoothness problems in motion generation due to the complexity of the skeleton motion sequence in the task of motion synthesis based on text description.
[0025] 2. The present application proposes a method of parameterizing the motion trajectory of the joint based on high-order Bezier curves, and then generating the joint trajectory from the hidden feature by predicting the Bezier control points and reconstructing the Bezier curve based on the control points to solve the problems of abnormal trajectory and inconsistency with kinematic characteristics in the existing coordinate sequence regression based method.
[0026] 3. The present application proposes a skeleton synthesis method based on hierarchical motion intention guidance to solve the problem of cumulative error in frame-by-frame generation in the process of skeleton motion sequence synthesis. BRIEF DESCRIPTION OF DRAWINGS
[0027] Figure 1 The present application is a whole flowchart.
[0028] Figure 2 The present application is a pivotal joint trajectory reconstruction diagram based on high-order Bezier curves.
[0029] Figure 3 A text description-based motion sequence synthesis model structure and a training schematic diagram of the present application.
[0030] Figure 4 An implementation example schematic diagram of the present application in a text description-based motion sequence generation task. DETAILED DESCRIPTION
[0031] The present application will be described in detail below with reference to the accompanying drawings and implementation examples.
[0032] Example 1 Taking a text description-based motion sequence generation task as an example, the task aims to generate the corresponding motion sequence based on a text description of a motion. As shown in Figure 1 A graph structure constraint-based cross-domain pose estimation method provided by the present application includes the following steps:
[0033] Step S1: As shown in Figure 3 , a text encoder based on a large language model, a motion sequence feature extraction model are constructed, and visual-linguistic multi-modal feature learning and feature space alignment learning are performed;
[0034] Step S2: Hierarchical representation of the motion sequence is constructed, and the time sequence motion trajectory of the body pivot joint node is represented by Bezier parameterization;
[0035] Step S3: The skeleton is graphically modeled, which models the generation process of the skeleton as a graph node feature transformation process under the condition of edge connection determination, and then constructs a ternary sample pair of text description-skeleton graph structure sequence-pivot joint node trajectory Bezier parameter;
[0036] Step S4: A motion sequence decoding model is constructed, including a trajectory generation model based on high-order Bezier curve representation and a skeleton generator based on graph structure, and the model is trained.
[0037] Further, the step S1 specifically includes:
[0038] Step S11: A text description-motion sequence training sample pair is constructed;
[0039] Step S12: A text encoder based on a large language model and a visual-linguistic modal feature extraction alignment model are constructed;
[0040] Step S13: The cross-modal feature alignment model is trained based on the above training sample pair.
[0041] Preferably, the cross-modal feature extraction alignment model includes a text encoder, a motion sequence encoder, and a hidden feature quantization module.
[0042] Preferably, the latent feature quantization module obtains the corresponding quantized latent feature z by finding the nearest element in the learnable codebook C for each feature vector z[i]. e [i]:
[0043] Preferably, the objective function for training the cross-modal feature alignment model based on the above training samples is: Where α is the weight, and sg[*] indicates gradient stopping.
[0044] Furthermore, step S2 specifically includes:
[0045] Step S21: Divide each joint point into different regions according to body parts, including the torso and the four limbs;
[0046] Step S22: Based on the motion characteristics of each joint, divide the joints of each region into a pivot joint and several end joints, where the pivot joint represents the connection point between the region and the body trunk.
[0047] Step S23: As Figure 2 As shown, the motion trajectory of the pivot joints of various parts of the body is parametrically modeled based on high-order Bézier curves, and represented as a set of Bézier control points.
[0048] Preferably, the motion trajectory t={t1,t2,…t} of the pivot joints of various parts of the body is based on a higher-order Bézier curve. l Parametric modeling specifically includes: first, initializing the Bézier control points; then, iteratively calculating the gradient until convergence; and finally updating the Bézier control points.
[0049] Preferably, the gradient calculation method is as follows:
[0050] Preferably, B(u) represents the set of n-order Bezier control points. Reconstructed joint trajectory curves:
[0051] Furthermore, step S3 specifically includes:
[0052] Step S31: Construct a skeleton graph model based on the topological connection relationships of key points;
[0053] Step S32: constructing a ternary training sample according to the text-motion sequence dataset, including a text description, a skeleton graph structure sequence, and a high-order Bezier parameter of a pivotal joint time sequence motion trajectory;
[0054] Step S33: loading a pre-trained text-motion feature extraction alignment model, and decoding a training motion sequence model according to the constructed ternary sample.
[0055] Preferably, constructing the skeleton graph model based on the topological connection relationship of the joints specifically includes: first determining an edge connection matrix according to the connection relationship of the body parts, and then calculating a node feature for each joint.
[0056] Preferably, the calculation of the node feature of each joint specifically includes: the node feature of an end node is an aggregation of a hidden feature and a corresponding joint position feature of a previous frame, and the node feature of a pivotal node is an aggregation of a hidden feature, a corresponding joint position feature of a previous frame, and a motion vector of the node trajectory in the frame, specifically as follows: wherein f c (*) represents feature aggregation.
[0057] Further, the step S4 specifically includes:
[0058] Step S41: constructing a Bezier decoder, which aims to generate a pivotal point trajectory Bezier parameter from a hidden feature space;
[0059] Step S42: constructing a skeleton generator, including a skeleton generator based on graph node feature transformation, a pivotal point motion intention fusion module, and a motion sequence iterative generation module.
[0060] Preferably, the constructed Bezier decoder is composed of a convolutional layer and a fully connected layer, and the high-order Bezier parameter of the pivotal point motion trajectory is decoded from the hidden feature space by using the Bezier decoder, so as to reconstruct the pivotal point motion trajectory.
[0061] Preferably, as shown in Figure 3 the constructed skeleton generator iteratively synthesizes a motion skeleton of each frame based on a skeleton of a previous frame, a hidden feature, and a pivotal point time sequence motion trajectory obtained by the Bezier decoder.
[0062] Figure 4An implementation example schematic diagram of the method on a text description based motion sequence generation task is shown, a data set HumanML3D is adopted, the data set contains 14616 human motion key point sequences, and 44970 text descriptions, covering multiple motion types including "jumping", "running" and the like. The method is verified on the data set, and is compared with current mainstream text based motion generation models T2M-GPT (CVPR2023), MotionDiffusion (TPAMI2024) and MLD (CVPR2023), and the experimental results are as shown in Table 1. From the experimental results, it can be seen that the method has obvious advantages in R-Precision and FID and the like indicators.
[0063] Table 1 comparative experimental results on the HumanML3D data set
Claims
1. A motion level decoupling representation and analysis method based on a visual language large model, characterized in that, The method comprises the following steps: S1: constructing a text encoder based on a large language model, a motion sequence feature extraction model, and performing visual-linguistic multi-modal feature learning and feature space alignment learning; S2: constructing a hierarchical representation of the motion sequence, and performing Bezier parameterization representation on the time sequence motion trajectory of the body pivot joint; S3: modeling the skeleton as a graph structure, modeling the generation process of the skeleton as a graph node feature transformation process under the condition of edge connection determination, and then constructing a ternary sample pair of the text description-skeleton graph structure sequence-pivot joint trajectory Bezier parameter; S4: constructing a motion sequence decoding model, including a trajectory generation model based on a high-order Bezier curve representation and a skeleton generator based on a graph structure, and training the model.
2. The motion level decoupling representation and analysis method based on a visual language large model according to claim 1, characterized in that, The step S1 specifically comprises: S11: constructing a text description-motion sequence training sample pair; S12: constructing a text encoder based on a large language model, and a visual-linguistic multi-modal feature extraction alignment model; the visual-linguistic modal feature extraction alignment model comprises a text encoder, a motion sequence encoder, and a hidden feature quantization module; S13: training the cross-modal feature alignment model based on the above training sample pair, and the objective function of the training is: Wherein α is a weight, and sg[*] represents gradient stop.
3. The method of claim 2, wherein the method further comprises: The hidden feature quantization module in S12 obtains the corresponding quantized hidden feature z by searching for the closest element from the learnable codebook C for each feature vector z[i] e [i]:
4. The motion level decoupling representation and analysis method based on a visual language large model according to claim 1, characterized in that, The step S2 specifically comprises: S21: dividing the joints into different regions according to the body parts, including the trunk part and the four limb parts; S22: dividing the joints in each region into a pivot joint and a plurality of end joints according to the motion characteristics of the joints. The pivot joint represents the connection point of the region and the body trunk part; S23: parameterize modeling the motion trajectory of the pivotal joint nodes of each part of the body based on a high-order Bezier curve, and represent it as a set of Bezier control points; the parameterize modeling the motion trajectory of the pivotal joint nodes of each part of the body based on a high-order Bezier curve The parameterize modeling specifically includes: first initializing the Bezier control points, and then iteratively calculating the gradient until convergence to update the Bezier control points 5. The motion level decoupling representation and analysis method based on the visual language large model according to claim 4, characterized in that, The iterative calculation of the gradient is calculated in the following manner: where B(u) represents a set of n-th order Bezier control points based on Reconstructed joint trajectory curve:
6. The motion level decoupling representation and analysis method based on a visual language large model according to claim 1, characterized in that, The step S3 specifically comprises: S31: constructing a skeleton graph model based on the topological connection relationship of the key points; first, the edge connection matrix is determined according to the connection relationship of the body parts, and then the node features of each joint are calculated; S32: constructing a ternary training sample according to the text-motion sequence data set, including the text description, the skeleton graph structure sequence, and the high-order Bezier parameter of the time sequence motion trajectory of the pivot joint; S33: loading the pre-trained text-motion feature extraction alignment model, and training the motion sequence decoding model according to the constructed ternary sample pair.
7. The motion level decoupling representation and analysis method based on a visual language large model according to claim 6, characterized in that, The calculation of the node feature of each joint specifically comprises: the node feature of the end node is the aggregation of the hidden feature and the corresponding joint position feature of the previous frame, and the node feature of the pivot node is the aggregation of the hidden feature, the corresponding joint position feature of the previous frame, and the motion vector of the node trajectory in the frame, which is specifically as follows: wherein f c (*) indicates a characteristic aggregation.
8. The motion level decoupling representation and analysis method based on a visual language large model according to claim 1, characterized in that, The step S4 specifically comprises: S41: constructing a Bezier decoder, which is used to generate the Bezier parameter of the pivot point trajectory from the hidden feature space; the constructed Bezier decoder is composed of convolutional layers and fully connected layers, and the high-order Bezier parameter of the pivot point motion trajectory is obtained from the hidden feature space by using the Bezier decoder, so as to reconstruct the pivot point motion trajectory; S42: constructing a skeleton generator, including a skeleton generator based on graph node feature transformation, a pivot point motion intention fusion module, and a motion sequence iterative generation module.
9. The motion level decoupling representation and analysis method based on a visual language large model according to claim 8, characterized in that, The skeleton generator constructed in S42 iteratively synthesizes the motion skeleton of each frame based on the skeleton of the previous frame, the hidden features, and the timing motion trajectory of the pivot point obtained by the Bezier decoder.
10. The motion level decoupling representation and analysis method based on a visual language large model according to claim 1 is applied in virtual character synthesis and improving motion generation accuracy.