A human motion language unified modeling method and system based on continuous latent representation
Patent Information
- Application Number
- CN202610730463.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-26
- Publication Date
- 2026-10-09
AI Technical Summary
[0005]本发明的目的是针对现有技术中离散动作标记带来的信息瓶颈、单向任务建模造成的能力割裂以及动作与文本对齐不足的技术问题，提出一种基于连续潜表示的人体动作语言统一建模方法及系统
[0045]1、本发明通过采用动作变分自编码器与扩散生成头在连续潜空间内建模人体动作，解决了现有技术中将连续动作强制离散化而造成的量化误差、细粒度动态丢失和长序列漂移问题，达到了提升动作生成平滑性、细节保真度和长时序稳定性的效果。
Smart Images
Figure CN122886672A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a unified modeling method and system for human action language based on continuous latent representation, belonging to the field of multimodal intelligent analysis and generation technology, and is applied to scenarios such as digital human driving, action generation, action description, action question answering, human-computer interaction, and embodied intelligent control. Background Technology
[0002] With the development of digital humans, virtual reality, and embodied intelligence technologies, joint modeling of human movements and natural language has become an important direction in multimodal research. Existing technologies typically aim to support text-to-action generation and action-to-text understanding simultaneously within the same framework, so that the model can both generate semantically constrained action sequences based on language instructions and describe, explain, and answer questions about input actions, thereby serving applications such as action generation, human-computer interaction, rehabilitation training, and intelligent agent control.
[0003] Existing action language modeling methods suffer from the following main drawbacks: First, many methods use vector quantization or discrete tokenization to map continuous actions to discrete tokens in a finite codebook, which easily introduces quantization errors, loss of fine-grained semantics, and long-term temporal drift, resulting in jittery generated actions or inaccurate descriptions. Second, while some continuous action generation methods can produce high-quality actions, they are usually only geared towards text-to-action unidirectional tasks and lack a unified framework for sharing parameters with action understanding. Third, some continuous representation methods mainly focus on action understanding and fail to achieve high-quality action generation within the same system. At the same time, the modal differences between actions and text are still significant, affecting conditional control capabilities and cross-task transfer capabilities.
[0004] Therefore, how to construct a unified modeling method that does not require discrete quantization, can simultaneously support action generation and action understanding in a shared backbone network, and reduces the difference between action and text modalities through a hierarchical alignment mechanism has become a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0005] The purpose of this invention is to address the technical problems in existing technologies, such as information bottlenecks caused by discrete action labeling, capability fragmentation resulting from unidirectional task modeling, and insufficient alignment between actions and text. It proposes a unified modeling method and system for human action language based on continuous latent representation. This improves the fine-grained fidelity of action generation, the semantic consistency of text description, and the unified modeling capability for bidirectional tasks.
[0006] The objective of this invention is achieved through the following technical solution:
[0007] This invention discloses a unified modeling method for human action language based on continuous latent representation, comprising the following steps:
[0008] Step 1: Extract frame-level and sequence-level samples from the human action sequences in the paired dataset consisting of human action sequences and corresponding text descriptions, and perform word segmentation and tokenization on the text descriptions to obtain the text input sequence.
[0009] Step 1.1: Construct a paired dataset consisting of chronologically ordered sequences of human actions and corresponding text descriptions;
[0010] Step 1.1.1: Arrange the three-dimensional joint positions, rotation information, and velocity information of consecutive frames in chronological order to form a sequence of human motion.
[0011] Step 1.1.2: Construct a paired dataset by combining human action sequences with corresponding text descriptions;
[0012] Step 1.2: Extract the human action sequence as single-frame pose and action sequence respectively, and perform word segmentation and tokenization on the text description to obtain frame-level samples, sequence-level samples and text input sequence in sequence;
[0013] Step 1.2.1: Extract single-frame poses from the human action sequence and construct frame-level samples with the corresponding text;
[0014] Step 1.2.2: Extract the human action sequence and construct sequence-level samples from the corresponding text;
[0015] Step 1.2.3: Segment and tokenize the text description to form a text input sequence;
[0016] Step 2: Perform local alignment of gesture text and global alignment of action language using a two-stage approach;
[0017] Step 2.1: Employ a Transformer-based attitude encoder With text encoder Frame-level samples and text input sequences are encoded into the same semantic space, and the InfoNCE contrastive loss is used to adjust the pose encoder. With text encoder Optimize and complete the first stage of local alignment of the pose text;
[0018] Step 2.2: Connect the first-stage attitude encoder Pose weights are frozen and the learnable query vectors are pre-concatenated to the pose features extracted frame by frame, and then a temporal encoder is applied sequentially. Compared with InfoNCE, loss optimization was performed to complete the second stage of global action language alignment;
[0019] Step 2.2.1: Connect the first-stage attitude encoder Pose weights are frozen, and pose features of human motion sequences in sequence-level samples are extracted frame by frame.
[0020] Step 2.2.2: Use learnable query vectors to pre-assemble the pose features extracted frame by frame, and then pass them through a temporal encoder. After capturing the action, an action-level semantic representation corresponding to the learnable query vector is obtained;
[0021] Step 2.2.3: Use InfoNCE contrastive loss to optimize the action-level semantic representation and text representation, thereby completing the second stage of global alignment of action language;
[0022] Step 3: Construct an action variational autoencoder to encode the continuous latent representation of the action sequence and decode the action sequence reconstruction, and map it with a fixed-length continuous prefix;
[0023] Step 3.1: Use the encoder in the action variational autoencoder to compress the action sequence in the sequence-level samples to obtain a continuous latent representation, and use its own decoder to reconstruct the action sequence;
[0024] Step 3.2: Map the continuous latent representations using a fixed-length continuous prefix; this will serve as the conditional input for the action-to-text task;
[0025] Step 4: Construct a decoder-style shared backbone network with shared parameters and dual task heads for text-to-action tasks and action-to-text tasks;
[0026] Step 4.1: Construct a decoder-based shared backbone network so that text-to-action tasks and action-to-text tasks can complete conditional modeling within the same hidden state space;
[0027] Step 4.2: In the action-to-text task, the continuous prefix obtained in step 3.2 is concatenated with the text start token and input into the shared backbone network, and the action description text is output through the autoregressive text generation head;
[0028] Step 4.3: In the text-to-action task, input the text token into the shared backbone network and reserve an interface area in the input sequence to obtain the hidden state corresponding to the interface area;
[0029] Step 4.4: Pool the hidden state to obtain a condition vector, and input the condition vector into the diffusion generator head to perform noise prediction and reverse denoising sampling on the action continuous latent representation to obtain the target action latent representation;
[0030] Step 4.5: Use the action decoder in step 3.1 to decode the latent representation of the target action to obtain the human action sequence corresponding to the semantics of the input text;
[0031] Step 5: Train the unified model using a phased training strategy;
[0032] Step 5.1: Pre-train the two-stage process from Step 2, and then pre-train the attitude encoder separately. With text encoder The weights are frozen, the decoder-style shared backbone network and the diffusion generation head are trained, and the text-to-action path is warmed up during training.
[0033] Step 5.2: After completing the preheating training, the decoder-type shared backbone network, the autoregressive text generation head, and the diffusion generation head are jointly trained, and a unified optimization is completed using a weighted objective function of text generation loss and diffusion loss.
[0034] Step 6: Perform bidirectional inference output;
[0035] Step 6.1: When the input is a text description or instruction, the target human action sequence is generated through a shared backbone network, a diffusion generation head, and an action decoder;
[0036] Step 6.2: When the input is a sequence of human actions, the action encoder, prefix mapping module, shared backbone network and autoregressive text generation head output action description text or action-related question and answer results;
[0037] This invention discloses a unified modeling system for human action language based on continuous latent representation, used to implement the aforementioned method. The unified modeling system for human action language based on continuous latent representation disclosed in this invention includes a data construction module, a two-stage alignment module, an action latent representation module, a unified backbone modeling module, a diffusion generation module, and a text generation module.
[0038] The data construction module is used to acquire human action sequences and text description pairing data, and generate frame-level pose samples, sequence-level action samples and text token sequences; these will serve as input to the two-stage alignment module.
[0039] The two-stage alignment module is used to first perform spatial pose-text alignment, and then perform temporal action-text alignment to construct a shared semantic space for actions and text; it will serve as the input to the action latent representation module.
[0040] The action latent representation module is used to encode human action sequences into continuous latent representations through an action variational autoencoder, and to reconstruct, decode, or prefix map the continuous latent representations as needed; it will serve as the input to the diffusion generation module.
[0041] The unified backbone modeling module is used to uniformly carry text-to-action tasks and action-to-text tasks in a decoder-type backbone network with shared parameters; it will serve as the input to the diffusion generation module.
[0042] The diffusion generation module is used to perform noise prediction and reverse denoising sampling of the action latent representation in the continuous latent space based on the condition vector output by the unified backbone modeling module, and to generate the target action latent representation.
[0043] The text generation module is used to output action description text, action explanation text, or action question and answer results in an autoregressive manner based on the continuous prefixes obtained by the action latent representation mapping.
[0044] Compared with existing technologies, it has the following beneficial effects:
[0045] 1. This invention solves the problems of quantization error, loss of fine-grained dynamics and long sequence drift caused by the forced discretization of continuous actions in the prior art by using a motion variational autoencoder and a diffusion generator to model human motion in a continuous latent space. This achieves the effect of improving the smoothness of motion generation, the fidelity of details and the stability of long time series.
[0046] 2. This invention solves the problems of separate modeling for generation and understanding, inability to share parameters, and insufficient cross-task transfer in the prior art by uniformly realizing text-to-action generation and action-to-text understanding within the same decoder-based shared backbone network. This achieves the effects of improving model reusability, enhancing bidirectional task collaboration capabilities, and reducing system deployment complexity.
[0047] 3. This invention solves the problems of large differences between action and text modalities, unstable condition control, and insufficient semantic consistency in the prior art by setting a two-stage action-text alignment mechanism that first establishes a local semantic correspondence between posture and text, and then establishes a global semantic correspondence between action dynamics and text. This achieves the effect of improving the controllability of action conditions, the accuracy of action description, and the semantic consistency of action question answering.
[0048] 4. This invention solves the problem that a unified model is difficult to balance generation stability and understanding ability in the early stage of training by adopting a phased training strategy of alignment pre-training, text-to-action path warm-up and dual-task joint training. This achieves the effects of improving training stability, accelerating model convergence and enhancing the usability of practical applications. Attached Figure Description
[0049] Figure 1 is a schematic diagram of the process of the present invention; Detailed Implementation
[0050] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.
[0051] Example
[0052] like Figure 1 As shown in the figure, the specific implementation steps of the unified modeling method for human action language based on continuous latent representation in this embodiment are as follows:
[0053] Step 1: Extract frame-level and sequence-level samples from the human action sequences in the paired dataset consisting of human action sequences and corresponding text descriptions, and perform word segmentation and tokenization on the text descriptions to obtain the text input sequence.
[0054] Step 1.1: Construct a paired dataset consisting of chronologically ordered sequences of human actions and corresponding text descriptions;
[0055] Step 1.1.1: Arrange the three-dimensional joint positions, rotation information, and velocity information of consecutive frames in chronological order to form a sequence of human motion.
[0056] Step 1.1.2: Construct a paired dataset by combining human action sequences with corresponding text descriptions;
[0057] Step 1.2: Extract the human action sequence as single-frame pose and action sequence respectively, and perform word segmentation and tokenization on the text description to obtain frame-level samples, sequence-level samples and text input sequence in sequence;
[0058] Step 1.2.1: Extract single-frame poses from the human action sequence and construct frame-level samples with the corresponding text;
[0059] Step 1.2.2: Extract the human action sequence and construct sequence-level samples from the corresponding text;
[0060] Step 1.2.3: Segment and tokenize the text description to form a text input sequence;
[0061] In this embodiment, human motion-text pairing samples and multi-granularity input representations are first constructed. Preferably, the raw data can consist of a human motion sequence X and its corresponding text description y, wherein the motion sequence can be composed of the three-dimensional joint positions, rotation information, velocity information, or combinations thereof of consecutive frames. To serve local alignment and global alignment respectively, single-frame poses or short segments can be extracted from the motion sequence to form frame-level samples, while the complete motion sequence is retained to form sequence-level samples; the text description is input into the text encoder or shared backbone network after word segmentation and tokenization.
[0062] Step 2: Perform local alignment of gesture text and global alignment of action language using a two-stage approach;
[0063] Step 2.1: Employ a Transformer-based attitude encoder With text encoder Frame-level samples and text input sequences are encoded into the same semantic space, and the InfoNCE contrastive loss is used to adjust the pose encoder. With text encoder Optimize and complete the first stage of local alignment of the pose text;
[0064] Step 2.2: Connect the first-stage attitude encoder Pose weights are frozen and the learnable query vectors are pre-concatenated to the pose features extracted frame by frame, and then a temporal encoder is applied sequentially. Compared with InfoNCE, loss optimization was performed to complete the second stage of global action language alignment;
[0065] Step 2.2.1: Connect the first-stage attitude encoder Pose weights are frozen, and pose features of human motion sequences in sequence-level samples are extracted frame by frame.
[0066] Step 2.2.2: Use learnable query vectors to pre-assemble the pose features extracted frame by frame, and then pass them through a temporal encoder. After capturing the action, an action-level semantic representation corresponding to the learnable query vector is obtained;
[0067] Step 2.2.3: Use InfoNCE contrastive loss to optimize the action-level semantic representation and text representation, thereby completing the second stage of global alignment of action language;
[0068] In this embodiment, a two-stage action-text alignment module is constructed and pre-trained. In the first stage, a Transformer-based pose encoder is constructed. With text encoder The first stage encodes single-frame pose samples and corresponding text descriptions into the same semantic space, and uses InfoNCE contrastive loss for optimization to achieve pose-level semantic alignment. In the second stage, the pose encoder obtained in the first stage is frozen, and pose features H={ are extracted frame-by-frame from the complete action sequence. A learnable query vector q is concatenated before the feature sequence, and then processed by a temporal encoder. After modeling the action dynamics, the output corresponding to the query vector is used as the action-level semantic representation m. The action-level semantic representation and the text representation are then compared and learned again using InfoNCE to establish a global alignment relationship between the action dynamics and the language description.
[0069] Step 3: Construct an action variational autoencoder to encode the continuous latent representation of the action sequence and decode the action sequence reconstruction, and map it with a fixed-length continuous prefix;
[0070] Step 3.1: Use the encoder in the action variational autoencoder to compress the action sequence in the sequence-level samples to obtain a continuous latent representation, and use its own decoder to reconstruct the action sequence;
[0071] Step 3.2: Map the continuous latent representations using a fixed-length continuous prefix; this will serve as the conditional input for the action-to-text task;
[0072] In this embodiment, a human motion encoding and decoding module for a continuous latent representation is constructed. Preferably, a motion variational autoencoder is used to encode the motion sequence, compressing the input motion into a continuous latent representation z, and a motion decoder is used to recover the human motion sequence from the continuous latent representation. Unlike discrete codebooks, this continuous latent representation does not need to map motions to a finite number of discrete tokens, thus it is more conducive to preserving speed variations, limb details, and long-term dependency information. Further, a prefix mapping module is used... The continuous latent representation z is mapped to a fixed-length continuous prefix P to serve as a conditional prefix input in the action-to-text task.
[0073] Step 4: Construct a decoder-style shared backbone network with shared parameters and dual task heads for text-to-action tasks and action-to-text tasks;
[0074] Step 4.1: Construct a decoder-based shared backbone network so that text-to-action tasks and action-to-text tasks can complete conditional modeling within the same hidden state space;
[0075] Step 4.2: In the action-to-text task, the continuous prefix obtained in step 3.2 is concatenated with the text start token and input into the shared backbone network, and the action description text is output through the autoregressive text generation head;
[0076] Step 4.3: In the text-to-action task, input the text token into the shared backbone network and reserve an interface area in the input sequence to obtain the hidden state corresponding to the interface area;
[0077] Step 4.4: Pool the hidden state to obtain a condition vector, and input the condition vector into the diffusion generator head to perform noise prediction and reverse denoising sampling on the action continuous latent representation to obtain the target action latent representation;
[0078] Step 4.5: Use the action decoder in step 3.1 to decode the latent representation of the target action to obtain the human action sequence corresponding to the semantics of the input text;
[0079] In this embodiment, a unified backbone network with shared parameters and a dual task head are constructed. For action-to-text tasks, the continuous prefix P obtained in step 3 is concatenated with the text start sequence and then input into the shared backbone network. The text-to-action task outputs action description text word by word through an autoregressive text generation head. For the text-to-action task, a text token is input into the shared backbone network, and a reserved interface region of length K is set in the input sequence. The hidden state H_if corresponding to this interface region is taken, and a condition vector c is obtained through average pooling, attention pooling, or other pooling methods. This condition vector, along with the diffusion time step t, is input into the diffusion generation head. It is used to perform noise prediction on noisy action latent representations within a continuous latent space.
[0080] Preferably, the forward noise addition process of the diffusion generator head can be expressed as: gradually adding Gaussian noise to the action latent representation z to obtain... During training, the noise introduced into the diffusion-generated head prediction is introduced; during inference, inverse denoising is performed starting from Gaussian noise to obtain the latent representation of the target action. The human action sequence is then decoded by the action decoder. This approach allows the shared backbone network to handle only semantic condition modeling, while the generation of continuous actions is handled by a diffusion mechanism that is more suitable for modeling continuous signals.
[0081] Step 5: Train the unified model using a phased training strategy;
[0082] Step 5.1: Pre-train the two-stage process from Step 2, and then pre-train the attitude encoder separately. With text encoder The weights are frozen, the decoder-style shared backbone network and the diffusion generation head are trained, and the text-to-action path is warmed up during training.
[0083] Step 5.2: After completing the preheating training, the decoder-type shared backbone network, the autoregressive text generation head, and the diffusion generation head are jointly trained, and a unified optimization is completed using a weighted objective function of text generation loss and diffusion loss.
[0084] In this embodiment, a phased training strategy is employed to train the unified model. Preferably, the two-stage alignment module is first pre-trained on large-scale action-text pairing data; then, the two-stage alignment module is frozen, and only the shared backbone network and the diffusion generation head are trained to warm up the text-to-action path; finally, the shared backbone network, the autoregressive text generation head, and the diffusion generation head are jointly trained, and a weighted sum of the text generation loss and the diffusion loss is used for unified optimization. This training process can effectively stabilize continuous action generation and avoid performance fluctuations caused by target conflicts in the early stages of joint training.
[0085] In one alternative embodiment, the two-stage alignment module can be pre-trained on large-scale pose-text pairing data such as MotionX++, and the unified model can be trained and evaluated on standard action-text datasets such as HumanML3D; the action variational autoencoder can adopt a Transformer-based encoder-decoder structure, and the latent representation dimension can be set to 128, 256, 512 or other adapted values; the shared backbone network can adopt a decoder-only structure and can be trained using efficient parameter fine-tuning methods.
[0086] Step 6: Perform bidirectional inference output;
[0087] Step 6.1: When the input is a text description or instruction, the target human action sequence is generated through a shared backbone network, a diffusion generation head, and an action decoder;
[0088] Step 6.2: When the input is a sequence of human actions, the action encoder, prefix mapping module, shared backbone network and autoregressive text generation head output action description text or action-related question and answer results;
[0089] In this embodiment, bidirectional inference output is performed. When the input is a text description, control command, or open-ended action request, the system outputs a sequence of human actions consistent with the text semantics; when the input is a sequence of human actions, the system outputs action description text, action explanation text, or natural language response to the action. Since text-to-action and action-to-text share the same backbone network and the same action latent representation system, this invention can support multiple action language tasks such as generation, understanding, and question answering within a unified model.
[0090] This invention discloses a unified modeling system for human action language based on continuous latent representation, used to implement the aforementioned method. The unified modeling system for human action language based on continuous latent representation disclosed in this invention includes a data construction module, a two-stage alignment module, an action latent representation module, a unified backbone modeling module, a diffusion generation module, and a text generation module.
[0091] The data construction module is used to acquire human action sequences and text description pairing data, and generate frame-level pose samples, sequence-level action samples and text token sequences; these will serve as input to the two-stage alignment module.
[0092] The two-stage alignment module is used to first perform spatial pose-text alignment, and then perform temporal action-text alignment to construct a shared semantic space for actions and text; it will serve as the input to the action latent representation module.
[0093] The action latent representation module is used to encode human action sequences into continuous latent representations through an action variational autoencoder, and to reconstruct, decode, or prefix map the continuous latent representations as needed; it will serve as the input to the diffusion generation module.
[0094] The unified backbone modeling module is used to uniformly carry text-to-action tasks and action-to-text tasks in a decoder-type backbone network with shared parameters; it will serve as the input to the diffusion generation module.
[0095] The diffusion generation module is used to perform noise prediction and reverse denoising sampling of the action latent representation in the continuous latent space based on the condition vector output by the unified backbone modeling module, and to generate the target action latent representation.
[0096] The text generation module is used to output action description text, action explanation text, or action question and answer results in an autoregressive manner based on the continuous prefixes obtained by the action latent representation mapping.
[0097] In this embodiment, the data construction module, the two-stage alignment module, the action latent representation module, the unified backbone modeling module, the diffusion generation module, and the text generation module are interconnected according to... Figure 1 The processes shown work together to complete the two-way conversion between human movements and natural language.
[0098] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention; any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A unified modeling method for human action language based on continuous latent representation, characterized in that: Includes the following steps, Step 1: Extract frame-level and sequence-level samples from the human action sequences in the paired dataset consisting of human action sequences and corresponding text descriptions, and perform word segmentation and tokenization on the text descriptions to obtain the text input sequence. Step 2: Perform local alignment of gesture text and global alignment of action language using a two-stage approach; Step 2.1: Employ a Transformer-based attitude encoder With text encoder Frame-level samples and text input sequences are encoded into the same semantic space, and the InfoNCE contrastive loss is used to adjust the pose encoder. With text encoder Optimize and complete the first stage of local alignment of the pose text; Step 2.2: Connect the first-stage attitude encoder Pose weights are frozen and the learnable query vectors are pre-concatenated to the pose features extracted frame by frame, and then a temporal encoder is applied sequentially. Compared with InfoNCE, loss optimization was performed to complete the second stage of global action language alignment; Step 3: Construct an action variational autoencoder to encode the continuous latent representation of the action sequence and decode the action sequence reconstruction, and map it with a fixed-length continuous prefix; Step 4: Construct a decoder-style shared backbone network with shared parameters and dual task heads for text-to-action tasks and action-to-text tasks; Step 4.1: Construct a decoder-based shared backbone network so that text-to-action tasks and action-to-text tasks can complete conditional modeling within the same hidden state space; Step 4.2: In the action-to-text task, the continuous prefix obtained in step 3.2 is concatenated with the text start token and input into the shared backbone network, and the action description text is output through the autoregressive text generation head; Step 4.3: In the text-to-action task, input the text token into the shared backbone network and reserve an interface area in the input sequence to obtain the hidden state corresponding to the interface area; Step 4.4: Pool the hidden state to obtain a condition vector, and input the condition vector into the diffusion generator head to perform noise prediction and reverse denoising sampling on the action continuous latent representation to obtain the target action latent representation; Step 4.5: Use the action decoder in step 3.1 to decode the latent representation of the target action to obtain the human action sequence corresponding to the semantics of the input text; Step 5: Train the unified model using a phased training strategy; Step 5.1: Pre-train the two-stage process from Step 2, and then pre-train the attitude encoder separately. With text encoder The weights are frozen, the decoder-style shared backbone network and the diffusion generation head are trained, and the text-to-action path is warmed up during training. Step 5.2: After completing the preheating training, the decoder-type shared backbone network, the autoregressive text generation head, and the diffusion generation head are jointly trained, and a unified optimization is completed using a weighted objective function of text generation loss and diffusion loss. Step 6: Perform bidirectional inference output; Step 6.1: When the input is a text description or instruction, the target human action sequence is generated through a shared backbone network, a diffusion generation head, and an action decoder; Step 6.2: When the input is a sequence of human actions, the action encoder, prefix mapping module, shared backbone network and autoregressive text generation head output action description text or action-related question and answer results.
2. The unified modeling method for human action language based on continuous latent representation as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Construct a paired dataset consisting of chronologically ordered sequences of human actions and corresponding text descriptions; Step 1.2: Extract the human action sequence as single-frame pose and action sequence respectively, and perform word segmentation and tokenization on the text description to obtain frame-level samples, sequence-level samples and text input sequence in sequence.
3. The unified modeling method for human action language based on continuous latent representation as described in claim 2, characterized in that: Step 1.1 is implemented as follows: Step 1.1.1: Arrange the three-dimensional joint positions, rotation information, and velocity information of consecutive frames in chronological order to form a sequence of human motion. Step 1.1.2: Construct a paired dataset by combining human action sequences with corresponding text descriptions.
4. The unified modeling method for human action language based on continuous latent representation as described in claim 2, characterized in that: Step 1.2 is implemented as follows: Step 1.2.1: Extract single-frame poses from the human action sequence and construct frame-level samples with the corresponding text; Step 1.2.2: Extract the human action sequence and construct sequence-level samples from the corresponding text; Step 1.2.3: Segment and tokenize the text description to form a text input sequence.
5. The unified modeling method for human action language based on continuous latent representation as described in claim 1, characterized in that: Step 2.2 is implemented as follows: Step 2.2.1: Connect the first-stage attitude encoder Pose weights are frozen, and pose features of human motion sequences in sequence-level samples are extracted frame by frame. Step 2.2.2: Use learnable query vectors to pre-assemble the pose features extracted frame by frame, and then pass them through a temporal encoder. After capturing the action, an action-level semantic representation corresponding to the learnable query vector is obtained; Step 2.2.3: Use InfoNCE contrastive loss to optimize the action-level semantic representation and text representation, thereby completing the second stage of global alignment of action language.
6. The unified modeling method for human action language based on continuous latent representation as described in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Use the encoder in the action variational autoencoder to compress the action sequence in the sequence-level samples to obtain a continuous latent representation, and use its own decoder to reconstruct the action sequence; Step 3.2: Map the continuous latent representations using a fixed-length continuous prefix; this will serve as the conditional input for the action-to-text task.
7. A unified modeling system for human action language based on continuous latent representation, as described in claim 1, characterized in that: It includes a data construction module, a two-stage alignment module, an action latent representation module, a unified backbone modeling module, a diffusion generation module, and a text generation module; The data construction module is used to acquire human action sequences and text description pairing data, and generate frame-level pose samples, sequence-level action samples and text token sequences; these will serve as input to the two-stage alignment module. The two-stage alignment module is used to first perform spatial pose-text alignment, and then perform temporal action-text alignment to construct a shared semantic space for actions and text; it will serve as the input to the action latent representation module. The action latent representation module is used to encode human action sequences into continuous latent representations through an action variational autoencoder, and to reconstruct, decode, or prefix map the continuous latent representations as needed. It will be used as input to the diffusion generation module; The unified backbone modeling module is used to uniformly carry text-to-action tasks and action-to-text tasks in a decoder-type backbone network with shared parameters; it will serve as the input to the diffusion generation module. The diffusion generation module is used to perform noise prediction and reverse denoising sampling of the action latent representation in the continuous latent space based on the condition vector output by the unified backbone modeling module, and to generate the target action latent representation. The text generation module is used to output action description text, action explanation text, or action question and answer results in an autoregressive manner based on the continuous prefixes obtained by the action latent representation mapping.