SMPL-X action-to-text generation method based on global and local feature fusion

Abstract

The invention provides a global and local feature fusion SMPL-X action-to-text generation method, and belongs to the field of artificial intelligence. The method comprises the following steps: preprocessing and coding an input SMPL-X action sequence into a double-flow action feature; through a cross-modal mapping module, the double-flow action features are mapped to a pre-trained large language model through independent projection branches, and global conditions and local action prefix embedding are obtained; through a text generation module, a decoder of a pre-trained large language model is used as a trunk network, text cue word embedding is extracted based on a text instruction given by a user, local action prefix embedding and text cue word embedding are spliced and then input into the decoder, and global conditions are injected into each layer of the decoder through a cross attention mechanism. And generating a description text in an autoregression mode. According to the method, the description text which is consistent with action semantics and has sufficient details can be stably and accurately generated, and the generation stability and the cross-scene applicability are improved when disturbance exists in the action sequence.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to a global and local feature fusion SMPL-X action to text generation method. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence technology, virtual digital people as a new form of human-computer interaction are gradually developing into multiple fields such as entertainment, education, customer service, etc., showing great market potential and application value. Virtual digital people not only can simulate the appearance and behavior of real people, but also can realize efficient communication with users through natural language processing technology. Due to the rapid development of virtual digital people, intelligent interaction and content production, the demand for automatically generating natural language descriptions based on human actions has also increased significantly. The existing technology usually takes human action sequences as input, extracts action features through an action encoder, and then outputs description text corresponding to the action semantics through a text generation model.

[0003] Since the SMPL-X model parameters contain face and hand information, the SMPL-X model can be used to generate human models, and therefore has gradually attracted more attention. Human models can be used in virtual anchors, digital avatars, film and game production, etc. fields. The existing research on SMPL-X and other parameterized human action to text generally adopts a two-stage paradigm combining action representation learning and conditional text generation: In the action representation learning stage, high-dimensional action sequences are often compressed into continuous latent representations or discretized into token sequences, and bidirectional generation modeling or cross-modal alignment of actions and texts is realized on the intermediate representation.

[0004] For example, the three-dimensional human action and text random tokenization mutual generation model TM2T realizes the bidirectional generation of three-dimensional human action and text through random sampling and discretization strategy; the text-to-three-dimensional action generative pre-training model T2M-GPT converts action sequences into discrete units that can be used for GPT-style generation modeling with discrete representation as the core; the human action generation pre-training large model MotionGPT quantizes the control signal into a discrete code and drives the language model to generate sequences through unified instruction prompts; and the language-conditioned action prior model LaMP constructs a unified language-action framework from the pre-training perspective to support multiple action-language tasks.

[0005] In the conditional text generation stage, the above action representation is usually mapped into a condition form that can be received by the language model, such as conditional tokens or prefix embeddings, to drive text generation, and the word-level target is usually used as the main supervision during training; therefore, this paradigm generally relies on the compression or discrete expression of action information and the input end condition injection in the implementation path, which provides a background for the subsequent deficiencies in detail fidelity, geometric constraint explicit modeling, and condition constraint persistence, etc.

[0006] However, there are some shortcomings in the above paradigm: since the action information is often compressed or discretely expressed and the action semantics and detail constraints are difficult to continuously pass and strengthen in the decoding process, it is often difficult to generate stable and accurate description text consistent with the action semantics and sufficient in details after inputting the SMPL-X action sequence. SUMMARY

[0007] In view of the shortcomings of the existing action-to-text research paradigm, the present application provides a SMPL-X action-to-text generation method with global and local feature fusion, which generates continuous double-flow action representation based on SMPL-X action encoding, maps the decoded continuous double-flow action representation to a cross-modal mapping module containing a global branch and a local branch to obtain global conditions and local action prefix embedding, and then decodes to obtain the description text. During the training process, a loss function is established to enhance the perception of action encoding features to three-dimensional geometric structure and kinematic relationship, and to improve the continuous constraint ability of action conditions to deep decoding process, so as to stably and accurately generate description text consistent with the action semantics and sufficient in details.

[0008] To achieve the above-mentioned purposes, the present application provides a SMPL-X action-to-text generation method with global and local feature fusion, comprising the following steps: After preprocessing the input SMPL-X action sequence, encode it into double-flow action features; Through the cross-modal mapping module, the double-flow action features are respectively mapped to the hidden space of the pre-trained large language model through independent projection branches to obtain global conditions and local action prefix embedding; Through the text generation module, the decoder of the pre-trained large language model is used as the backbone network, the text prompt word embedding is extracted based on the text instruction given by the user, the local action prefix embedding and the text prompt word embedding are spliced and input into the decoder, and the global conditions are injected into each layer of the decoder through the cross-attention mechanism, so as to generate description text in a self-recurrent manner.

[0009] Preferably, the preprocessing includes zero-mean unit-variance normalization processing of the input SMPL-X action sequence to obtain the preprocessed SMPL-X action sequence.

[0010] Preferably, the encoding into double-flow action features comprises: The preprocessed SMPL-X action sequence is mapped to a frame-level embedding sequence through the bone embedding layer of the action encoder; A learnable global marker is spliced at the head of the frame-level embedding sequence, and position encoding is added to obtain an input sequence; inputting the input sequence into an action encoder to obtain a top-level hidden state sequence; extracting a feature at a first position in the top-level hidden state sequence as a sequence-level global action feature and extracting features at subsequent positions as frame-level local fine-grained action features, the sequence-level global action feature and the frame-level local fine-grained action features being used as double-stream action features.

[0011] Further preferably, the action encoder adopts a VAE encoder to encode the input sequence to obtain the top-level hidden state sequence, and the encoding process is represented as: wherein, is the top-level hidden state sequence, is the first hidden state, is the frame top-level hidden state, is a total number of top-level hidden states, is the VAE encoder, is the input sequence, is a feature space dimension.

[0012] Preferably, each of the independent projection branches adopts a plurality of multilayer perceptron structures constructed based on a gated linear unit for nonlinear projection, and a gating mechanism followed by the projection branch in the nonlinear transformation process is represented as for realizing semantic mapping and dimension adaptation from the action feature dimension to the hidden space of the pre-trained large language model; wherein, and are two parts obtained by uniformly dividing the intermediate representation vector output by the same linear layer according to the feature space dimension.

[0013] Preferably, the independent projection branch includes a global branch and a local branch. The global branch is configured to project the sequence-level global action feature in the double-stream action feature to obtain a global condition. The local branch is configured to project the frame-level local fine-grained action feature in the double-stream action feature at each time point to obtain a local action prefix embedding.

[0014] Optionally, the pre-trained large language model adopts the language model of Qwen2.5-VL-7B.

[0015] Preferably, the text prompt word embedding based on the user-given text instruction includes: based on the user-given text instruction obtaining the text prompt word embedding through a text embedding layer of the pre-trained large language model: wherein, is the text prompt word embedding, is an embedding operation of the pre-trained large language model, The length of the text instruction sequence. This represents the latent space feature dimension of a pre-trained large language model.

[0016] Preferably, in the cross-attention mechanism, the query vector originates from the hidden state of the decoder, and both the key vector and the value vector originate from global conditions.

[0017] Preferably, the method of generating descriptive text through autoregression includes: The features obtained by concatenating the local action prefix embedding and the text prompt word embedding are input into the decoder. Each layer of the decoder performs self-attention calculation, cross-attention calculation and feedforward network calculation in sequence to generate descriptive text. The self-attention calculation is performed based on the hidden state of the previous layer's output to obtain the first output; the cross-attention calculation uses the first output of the current layer as the query vector and the global conditions as the key vector and value vector to obtain the second output; the feedforward network performs a nonlinear transformation on the second output to gradually generate descriptive text.

[0018] Preferably, the process of generating descriptive text employs a three-stage training strategy, including: The first phase focuses on rebuilding the losses. With KL divergence loss Based on this, joint rotation reconstruction loss is introduced. Loss of consistency with forward kinematic position Construct the first loss function Used to optimize the action encoder to learn stable two-stream action features; the first loss function is defined as follows: Among them, reconstruction losses , and The first Frame target motion features and reconstructed motion features, KL divergence loss , Let KL divergence be the KL divergence. For coding distribution, The distribution is normal; joint rotation reconstruction loss Forward kinematic position consistency loss , and The first Frame target rotation features and reconstructed rotation features The total number of frames in the action sequence. It is an L1 norm. and The first Frame number The target 3D coordinates and reconstructed 3D coordinates of each joint The total number of joints. It is an L2 norm; In the second stage, the action encoder trained in the first stage is frozen, and a second loss function related to text generation is constructed. This is used to optimize the cross-modal mapping module and the text generation module, making the global conditional and local action prefix embeddings compatible with the latent space of the pre-trained large language model in terms of distribution and semantics, thereby achieving stable reception and decoding of the pre-trained large language model; the second loss function is defined as Among them, vocabulary-level label smooth cross-entropy loss , The target text length, For the vocabulary index within the target text length, For a vocabulary set, For a subset of the word list, The target distribution after smoothing the vocabulary-level labels. For the target text A real vocabulary list, For conditional inputs that include global conditions and local action prefix embeddings, Optimization parameters for large language models; sentence-level loss , For batch size, and The first The reference sentence vector and the generated sentence vector for each sample. It is a function with maximum value. For the first The reference sentence vector of a difficult negative sample, For cosine distance calculation, For interval hyperparameters; length regression loss , For the first The length of the description text for each sample. For the first The objective is to supervise the length of the true descriptive text of each sample. In the third stage, the feature injection into the decoder is performed by fixing the global conditions and the concatenation of the local action prefix embedding sequence and the text prompt word embedding. The second loss function is also used to supervise and fine-tune the ability to generate descriptive text, so as to stably generate descriptive text.

[0019] This invention also provides an SMPL-X action-to-text generation system that fuses global and local features, used to implement the aforementioned SMPL-X action-to-text generation method that fuses global and local features, comprising: The action generation module is used to preprocess the input SMPL-X action sequence and encode it into dual-stream action features; The cross-modal mapping module is used to map the two-stream action features to the latent space of the pre-trained large language model through independent projection branches, so as to obtain global conditional and local action prefix embeddings. The text generation module uses a pre-trained large language model decoder as the backbone network to extract text prompt word embeddings based on user-given text instructions. It concatenates the local action prefix embeddings with the text prompt word embeddings and inputs them into the decoder. It also injects global conditions into each layer of the decoder through a cross-attention mechanism and generates descriptive text through an autoregressive approach.

[0020] Compared with the prior art, the beneficial effects of the present invention include at least the following: The SMPL-X action-to-text generation method provided by this invention, which integrates global and local features, preprocesses the input SMPL-X action sequence to encode it into dual-stream action features. Through a cross-modal mapping module, the dual-stream action features are mapped to the latent space of a pre-trained large language model via independent projection branches, yielding global conditions and local action prefix embeddings. The text generation module uses the decoder of the pre-trained large language model as the backbone network, extracting text prompt word embeddings based on user-given text instructions. The local action prefix embeddings and text prompt word embeddings are concatenated and input into the decoder. A cross-attention mechanism injects global conditions into each layer of the decoder, generating descriptive text in an autoregressive manner. During training, a loss function is established to enhance the action encoding features' perception of 3D geometric structures and kinematic relationships, improving the continuous constraint of action conditions on the deep decoding process. This results in stable and accurate generation of descriptive text that is semantically consistent with the actions and rich in detail, while also improving generation stability and cross-scene applicability when the action sequence is perturbed. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0022] Figure 1 This is a flowchart illustrating the SMPL-X action-to-text generation method that fuses global and local features, as provided in this invention.

[0023] Figure 2 This is a flowchart illustrating the encoded dual-stream action features provided in the embodiment.

[0024] Figure 3 This is a schematic diagram illustrating the process of generating descriptive text using the text generation module provided in this embodiment.

[0025] Figure 4This is a schematic diagram of the structure of the SMPL-X action-to-text generation system that fuses global and local features, as provided in an embodiment of the present invention.

[0026] Figure 5 This diagram illustrates a comparison between the action description generation effect provided in this embodiment of the invention and the effect of existing technologies. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and given in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0028] The inventive concept of this invention is as follows: Addressing the problems in SMPL-X action-to-text generation tasks, such as the easy loss of fine-grained action information, insufficient geometric perception capability of action encoding features, constraint attenuation of action conditions during deep decoding, and the lack of sentence-level semantic and textual structure constraints in the generated text, this invention provides an SMPL-X action-to-text generation method that fuses global and local features. This method employs continuous dual-stream action representation at the action encoding end to simultaneously preserve global semantics and local details, and enhances the geometric perception capability of the action encoder in extracting features through geometric and kinematic consistency constraints. At the text decoding end, a dual-path conditional injection mechanism is employed, utilizing a local feature mapping module to provide frame-level detailed conditions, and a global feature mapping module combined with cross-attention to continuously inject global semantic constraints during the decoding process. Simultaneously, multi-granularity text supervision is combined to improve the semantic consistency, content coverage, and textual controllability of the generated text, thereby achieving stable action description generation.

[0029] like Figure 1 As shown in the embodiment, an SMPL-X action-to-text generation method based on the fusion of global and local features includes the following steps: S1. After preprocessing the input SMPL-X motion sequence, it is encoded into dual-stream motion features. The dual-stream motion features simultaneously include sequence-level global motion features and frame-level local fine-grained motion features. Sequence-level global motion features are used to represent the overall semantic information of the entire motion, while frame-level local fine-grained motion features are used to preserve the dynamic details of posture and hand movements that change over time.

[0030] In the embodiment, the input SMPL-X action sequence After performing zero-mean unit variance normalization, the preprocessed SMPL-X action sequence is obtained, where For the number of frames in the action sequence, For the first Frame SMPL-X action sequence vector.

[0031] Next, the preprocessed SMPL-X action sequence is encoded into dual-stream action features, such as... Figure 2 As shown, the variational autoencoder (VAE) from the motion diffusion model (MLD) is used as the backbone network. The skeleton embedding layer maps the preprocessed SMPL-X action sequence to a 512-dimensional feature space to obtain the frame-level embedding sequence. : , in, For bone embedding layer; A learnable global tag is appended to the beginning of the frame-level embedding sequence. Construct encoder input The actual encoder input is obtained by superimposing position encoding. ,Will Input the action encoder to obtain the top-level continuous hidden state sequence: , in, The top-level hidden state sequence, The first hidden state in the top-level hidden state sequence The top layer of the frame is hidden. This represents the total number of hidden states at the top level. For VAE encoders using the MLD framework, Given the input sequence, is the dimension of the feature space.

[0032] This embodiment does not use the latent motion vectors obtained by variational modeling as action representations, but instead uses the top-level hidden state as a continuous action representation, and obtains dual-stream action features based on the input splicing structure decoupling: , , in, For sequence-level global action features, It represents frame-level local fine-grained motion features.

[0033] S2. Through the cross-modal mapping module, the dual-stream action features are mapped to the latent space of the pre-trained large language model through independent projection branches, obtaining global conditional and local action prefix embeddings. This step is used to map the continuous dual-stream action features output from S1 to the latent space of the text decoder, making them usable as conditional inputs to the pre-trained large language model. The cross-modal mapping module adopts a two-branch structure: the global branch projects the sequence-level global action features to generate a global conditional representation, which is continuously injected as conditional information for cross-attention during text decoding; the local branch projects the frame-level local fine-grained action feature sequence time-by-time to generate an action prefix embedding sequence, which is used together with the text prompt word embeddings after being concatenated in the sequence dimension as decoder input, thereby achieving the adaptation from the action representation space to the language model latent space.

[0034] In the embodiments, such as Figure 3 As shown, the sequence-level global action features are based on the output of S1. With frame-level local fine-grained motion features ,in The cross-modal mapping module is an action semantic mapping module based on gated linear units, containing global and local branches, and defining global and local projection operators respectively to project action features onto the language model's hidden space to obtain global conditions. and local action prefix embedding : , , in, and The network structures are consistent but the parameters are independent of each other. They all adopt the action semantic mapping structure based on gated linear units: the input action features are first processed by root mean square normalization, and then projected by a multilayer perceptron formed by stacking several gated linear unit blocks. Both independent projection branches (global branch and local branch) employ a multilayer perceptron structure built with several gated linear units for nonlinear projection, and the gating mechanism followed by the projection branches during the nonlinear transformation is expressed as follows: This is used to achieve semantic mapping and dimensional adaptation from the action feature dimension to the latent space of a pre-trained large language model; among which, and The intermediate representation vector output by the same linear layer is divided into two parts by uniformly dividing the feature space dimension.

[0035] S3. Through the text generation module, the decoder of the pre-trained large language model is used as the backbone network. Based on the text instructions given by the user, the text prompt word embedding is extracted. The local action prefix embedding and the text prompt word embedding are concatenated and input into the decoder. The global conditions are injected into each layer of the decoder through the cross attention mechanism to generate descriptive text in an autoregressive manner.

[0036] In this embodiment, the text generation module uses the decoder of the Qwen2.5-VL-7B language model as the backbone network, which consists of 28 Transformer layers. Local action prefix embedding sequences are concatenated with text cue word embeddings along the sequence dimension to participate in self-attention modeling. Global conditions, as cross-attention information, are continuously injected into each layer of the decoder, thus preserving both local temporal details and global semantic constraints during the decoding process.

[0037] Specifically, based on the global conditions of S2 output. Embedded with local action prefixes Based on user-provided text instructions By using the text embedding layer of a pre-trained large language model, the text prompt word embeddings are obtained: ,in, Embed text prompts, Embedding operations for pre-trained large language models The length of the text instruction sequence. This represents the latent space feature dimension of a pre-trained large language model.

[0038] The local action prefix and the cue word are embedded in the sequence dimension and concatenated to obtain the decoder input. .

[0039] In the decoder's Within the layer decoding block ( The hidden states are updated in the order of "self-attention - cross-attention - feedforward network". Let the th... The decoding input of the layer is Then we have: , , , in, This indicates a normalization operation. This indicates multi-head self-attention calculation. This represents multi-head cross-attention computation, and FFN(⋅) represents a feedforward network. The first output is obtained from multi-head self-attention computation. The second output is calculated using multi-head cross-attention. This is the generated description text.

[0040] Cross-attention satisfies the following conditions: the query vector originates from the current hidden state, and both the key and value vectors originate from global conditions. Defined as: , in, For the first The learnable parameters of each layer are decoded through 28 layers to obtain the top-level hidden state. The descriptive text is generated step by step through the output header, and the generation location is... have ,in express In generating the latent vector at position a, This is the output header for the vocabulary projection.

[0041] In this embodiment, the method employs a three-stage training strategy, including: The first phase focuses on rebuilding the losses. With KL divergence loss Based on this, joint rotation reconstruction loss is introduced. Loss of consistency with forward kinematic position Construct the first loss function Used to optimize the action encoder to learn stable two-stream action features; the first loss function is defined as follows: Among them, reconstruction losses , and The first Frame target motion features and reconstructed motion features, KL divergence loss , Let KL divergence be the KL divergence. For coding distribution, The distribution is normal; joint rotation reconstruction loss Forward kinematic position consistency loss , and The first Frame target rotation features and reconstructed rotation features The total number of frames in the action sequence. It is an L1 norm. and The first Frame number The target 3D coordinates and reconstructed 3D coordinates of each joint The total number of joints. It is an L2 norm; In the second stage, the action encoder trained in the first stage is frozen, and a second loss function related to text generation is constructed. This is used to optimize the cross-modal mapping module and the text generation module, making the global conditional and local action prefix embeddings compatible with the latent space of the pre-trained large language model in terms of distribution and semantics, thereby achieving stable reception and decoding of the pre-trained large language model; the second loss function is defined as Among them, vocabulary-level label smooth cross-entropy loss , The target text length, For the vocabulary index within the target text length, For a vocabulary set, For a subset of the word list, The target distribution after smoothing the vocabulary-level labels. For the target text A real vocabulary list, For conditional inputs that include global conditions and local action prefix embeddings, Optimization parameters for large language models; sentence-level loss , For batch size, and The first The reference sentence vector and the generated sentence vector for each sample. It is a function with maximum value. For the first The reference sentence vector of a difficult negative sample, For cosine distance calculation, For interval hyperparameters; length regression loss , For the first The length of the description text for each sample. For the first The objective is to supervise the length of the true descriptive text of each sample. In the third stage, the feature injection into the decoder is performed by fixing the global conditions and the concatenation of the local action prefix embedding sequence and the text prompt word embedding. The second loss function is also used to supervise and fine-tune the ability to generate descriptive text, so as to stably generate descriptive text.

[0042] Based on the same inventive concept, the embodiment also provides an SMPL-X action-to-text generation system that fuses global and local features, used to implement the aforementioned SMPL-X action-to-text generation method that fuses global and local features, such as... Figure 4 As shown, it includes: The action generation module is used to preprocess the input SMPL-X action sequence and encode it into dual-stream action features; The cross-modal mapping module is used to map the two-stream action features to the latent space of the pre-trained large language model through independent projection branches, so as to obtain global conditional and local action prefix embeddings. The text generation module uses a pre-trained large language model decoder as the backbone network to extract text prompt word embeddings based on user-given text instructions. It concatenates the local action prefix embeddings with the text prompt word embeddings and inputs them into the decoder. It also injects global conditions into each layer of the decoder through a cross-attention mechanism and generates descriptive text through an autoregressive approach.

[0043] To further verify the feasibility and effectiveness of the SMPL-X action-to-text generation method proposed in this invention, a comparative experiment was conducted on a publicly available action description dataset, and representative complex action samples were randomly selected for visualization analysis.

[0044] like Figure 5 The keyframe visualization rendering of the input SMPL-X motion sequence is shown, which contains limb details with specific semantics (e.g., a human body jumping to the right while keeping its orientation unchanged).

[0045] The descriptive text generated by the existing Lamp model and the descriptive text generated by the method of this invention are compared. The comparative analysis shows that the existing Lamp model, due to its lack of independent perception and fusion of local fine-grained action features, often only captures the coarse-grained global semantics of "side jump" and outputs the descriptive text "a person completed a side jump to the right," failing to accurately describe the key fine-grained posture constraint of "maintaining the torso facing forward" during the action.

[0046] In contrast, the method of this invention benefits from the introduction of continuous dual-stream motion representation at the encoding end and the use of a local motion prefix embedding mechanism at the decoding end. It can effectively capture and retain fine-grained information such as hand micro-movements and body orientation constraints. The generated descriptive text accurately represents the complex semantics of "jumping to the right without turning the body," which is highly consistent with the real labeled text "a person completes a jump displacement to the right while keeping the body facing forward, without turning the body to face the target direction of the jump."

[0047] The above experimental results strongly demonstrate that the technical solution proposed in this invention, which integrates sequence-level global features and frame-level local fine-grained features, can significantly solve the problem of easy loss of fine-grained action information in the prior art. While ensuring the fluency of the generated text sentences, it greatly improves the accuracy and richness of action description, thus verifying the authenticity and effectiveness of the technical solution of this invention.

[0048] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of components and steps described in these embodiments do not limit the scope of the invention. Of course, the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of the invention. All equivalent changes or modifications made to the structures, features, and principles described in the claims of this invention should be included within the scope of the claims of this invention.

[0049] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating action-to-text using SMPL-X by fusing global and local features, characterized in that, Includes the following steps: After preprocessing the input SMPL-X action sequence, it is encoded into dual-stream action features; Through the cross-modal mapping module, the dual-stream action features are mapped to the latent space of the pre-trained large language model through independent projection branches to obtain global conditional and local action prefix embeddings; The text generation module uses a pre-trained large language model decoder as the backbone network. It extracts text prompt word embeddings based on user-given text instructions, concatenates local action prefix embeddings with text prompt word embeddings, and inputs them into the decoder. A cross-attention mechanism is used to inject global conditions into each layer of the decoder to generate descriptive text in an autoregressive manner.

2. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 1, characterized in that, The encoding is a dual-stream action feature, including: The preprocessed SMPL-X motion sequence is mapped to a frame-level embedding sequence through the skeleton embedding layer of the motion encoder; A learnable global tag is appended to the beginning of the frame-level embedded sequence, and positional encoding is added to obtain the input sequence; The input sequence is fed into the action encoder for encoding to obtain the top-level hidden state sequence; The features at the beginning of the top-level hidden state sequence are extracted as sequence-level global action features, and the features at subsequent positions are extracted as frame-level local fine-grained action features. The sequence-level global action features and the frame-level local fine-grained action features are used as dual-stream action features.

3. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 2, characterized in that, The action encoder uses a VAE encoder to encode the input sequence to obtain the top-level hidden state sequence. The encoding process is represented as follows: ,in, The top-level hidden state sequence, For the first The top layer of the frame is hidden. This represents the total number of hidden states at the top level. For VAE encoders, Given the input sequence, is the dimension of the feature space.

4. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 1, characterized in that, The independent projection branches all employ a multilayer perceptron structure built upon gated linear units for nonlinear projection, and the gating mechanism followed by the projection branches during the nonlinear transformation is expressed as follows: This is used to achieve semantic mapping and dimensional adaptation from the action feature dimension to the latent space of a pre-trained large language model; where u and v are two parts of the intermediate representation vector output by the same linear layer, which are uniformly divided according to the feature space dimension.

5. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 4, characterized in that, The independent projection branches include global branches and local branches; The global branch is used to project the sequence-level global action features in the dual-stream action features to obtain global conditions; The local branch is used to project the frame-level local fine-grained motion features in the dual-stream motion features on a time-by-time basis to obtain the local motion prefix embedding.

6. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 1, characterized in that, The method of extracting text prompt word embeddings based on user-given text instructions includes: Based on user-provided text instructions By using the text embedding layer of a pre-trained large language model, the text prompt word embeddings are obtained: ,in, Embed text prompts, Embedding operations for pre-trained large language models The length of the text instruction sequence. This represents the latent space feature dimension of a pre-trained large language model.

7. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 1, characterized in that, The method of generating descriptive text through autoregression includes: The features obtained by concatenating the local action prefix embedding and the text prompt word embedding are input into the decoder. Each layer of the decoder performs self-attention calculation, cross-attention calculation and feedforward network calculation in sequence to generate descriptive text. The self-attention calculation is performed based on the hidden state of the previous layer's output to obtain the first output; the cross-attention calculation uses the first output of the current layer as the query vector and the global conditions as the key vector and value vector to obtain the second output; the feedforward network performs a nonlinear transformation on the second output to gradually generate descriptive text.

8. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 1, characterized in that, The process of generating descriptive text employs a three-stage training strategy, including: The first phase focuses on rebuilding the losses. With KL divergence loss Based on this, joint rotation reconstruction loss is introduced. Loss of consistency with forward kinematic position Construct the first loss function Used to optimize the action encoder to learn stable dual-stream action features; In the second stage, the action encoder trained in the first stage is frozen, and a second loss function related to text generation is constructed. It is used to optimize the cross-modal mapping module and the text generation module, so that the global condition and local action prefix embedding are compatible with the latent space of the pre-trained large language model in terms of distribution and semantics, and realize the stable reception and decoding of the pre-trained large language model. In the third stage, the feature injection into the decoder is performed by fixing the global conditions and the concatenation of the local action prefix embedding sequence and the text prompt word embedding. The second loss function is also used to supervise and fine-tune the ability to generate descriptive text, so as to stably generate descriptive text.

9. The SMPL-X action-to-text generation method based on global and local feature fusion according to claim 8, characterized in that, The first loss function is defined as follows: Among them, reconstruction losses , and The first Frame target motion features and reconstructed motion features, KL divergence loss , Let KL divergence be the KL divergence. For coding distribution, The distribution is normal; joint rotation reconstruction loss Forward kinematic position consistency loss , and The first Frame target rotation features and reconstructed rotation features The total number of frames. It is an L1 norm. and The first Frame number The target 3D coordinates and reconstructed 3D coordinates of each joint The total number of joints. It is an L2 norm; The second loss function is defined as follows: Among them, vocabulary-level label smooth cross-entropy loss , The target text length, For the vocabulary index within the target text length, For a vocabulary set, For a subset of the word list, The target distribution after smoothing the vocabulary-level labels. For the target text's first A real vocabulary list, For conditional inputs that include global conditions and local action prefix embeddings, Optimization parameters for large language models; sentence-level loss , For batch size, and The first The reference sentence vector and the generated sentence vector for each sample. It is a function with maximum value. For the first The reference sentence vector of a difficult negative sample, For cosine distance calculation, For interval hyperparameters; length regression loss , For the first The length of the description text for each sample. For the first The objective is to supervise the length of the true description text of each sample.

10. An SMPL-X action-to-text generation system that fuses global and local features, characterized in that, The SMPL-X action-to-text generation method for implementing the global and local feature fusion of any one of claims 1 to 9 includes: The action generation module is used to preprocess the input SMPL-X action sequence and encode it into dual-stream action features; The cross-modal mapping module is used to map the two-stream action features to the latent space of the pre-trained large language model through independent projection branches, so as to obtain global conditional and local action prefix embeddings. The text generation module uses a pre-trained large language model decoder as the backbone network to extract text prompt word embeddings based on user-given text instructions. It concatenates the local action prefix embeddings with the text prompt word embeddings and inputs them into the decoder. It also injects global conditions into each layer of the decoder through a cross-attention mechanism and generates descriptive text through an autoregressive approach.

Images