Smpl-x action-to-text generation method with global and local feature fusion

Abstract

The application provides a SMPL-X action-to-text generation method based on global and local feature fusion, and belongs to the field of artificial intelligence. The input SMPL-X action sequence is preprocessed and encoded into double-flow action features; the double-flow action features are respectively mapped to a pre-trained large language model through independent projection branches of a cross-modal mapping module to obtain global conditions and local action prefix embeddings; a text generation module takes the decoder of the pre-trained large language model as a backbone network, extracts text prompt word embeddings based on a user-given text instruction, inputs the local action prefix embeddings and the text prompt word embeddings into the decoder after splicing, and injects the global conditions into each layer of the decoder through a cross-attention mechanism to generate description text in a self-recurrent manner. The application can stably and accurately generate description text that is consistent with the semantics of the action and sufficient in details, and improves the generation stability and cross-scene applicability when the action sequence is disturbed.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to a SMPL-X action-to-text generation method based on global and local feature fusion. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence technology, virtual digital humans, as a new form of human-computer interaction, are gradually developing into multiple fields such as entertainment, education, customer service, etc., and showing great market potential and application value. Virtual digital humans not only can simulate the appearance and behavior of real people, but also can realize efficient communication with users through natural language processing technology. Due to the rapid development of virtual digital humans, intelligent interaction and content production, the demand for automatically generating natural language descriptions based on human actions has also increased significantly. Existing technologies usually take human action sequences as input, extract action features through action encoders, and then output description texts corresponding to action semantics through text generation models.

[0003] Since the SMPL-X model parameters contain face and hand information, the SMPL-X model can be used to generate human models, and therefore has gradually attracted more attention. Human models can be used in virtual anchors, digital avatars, film and game production, etc. fields. Existing researches on SMPL-X and other parameterized human action-to-text generally adopt a two-stage paradigm combining action representation learning and conditional text generation:

[0004] In the action representation learning stage, high-dimensional action sequences are often compressed into continuous latent representations or discretized into token sequences, and bidirectional generation modeling or cross-modal alignment of actions and texts is realized on the intermediate representation.

[0005] For example, the three-dimensional human action and text random tokenization mutual generation model TM2T realizes the bidirectional generation of three-dimensional human action and text through random sampling and discretization strategy; the text-to-three-dimensional action generative pre-training model T2M-GPT converts action sequences into discrete units that can be used for GPT-style generation modeling with discrete representation as the core; the human action generation pre-training large model MotionGPT quantizes control signals into discrete codes and drives the language model to generate sequences through unified instruction prompts; and the language-conditioned action prior model LaMP constructs a unified language-action framework from the perspective of pre-training to support multiple action-language tasks.

[0006] In the conditional text generation stage, the above action representation is usually mapped into a conditional form receivable by the language model, such as conditional tokens or prefix embeddings, to drive text generation, and the word-level target is usually used as the main supervision during training; therefore, this paradigm generally relies on the compression or discrete expression of action information and the conditional injection at the input end in the implementation path, which provides a background for the subsequent deficiencies in detail fidelity, geometric constraint explicit modeling, and conditional constraint persistence.

[0007] However, there are some deficiencies in the above paradigm: since the action information is usually compressed or discretely expressed and the conditional injection is usually performed at the input end of the text generation model, the action semantics and detail constraints are difficult to be continuously transmitted and strengthened during the decoding process, so it is difficult to stably and accurately generate a description text consistent with the action semantics and sufficient in details after inputting the SMPL-X action sequence. SUMMARY

[0008] In view of the deficiencies of the existing action-to-text research paradigm, the present application provides a SMPL-X action-to-text generation method with global and local feature fusion, which generates continuous double-stream action representation based on SMPL-X action coding, decodes the continuous double-stream action representation, and maps it to a cross-modal mapping module containing a global branch and a local branch to obtain global conditions and local action prefix embeddings, and then decodes to obtain a description text, and establishes a loss function during the training process to enhance the perception ability of the action coding features to the three-dimensional geometric structure and kinematic relationship, and to improve the continuous constraint ability of the action conditions to the deep decoding process, so as to stably and accurately generate a description text consistent with the action semantics and sufficient in details.

[0009] To achieve the above-mentioned purposes, the present application provides a SMPL-X action-to-text generation method with global and local feature fusion, comprising the following steps:

[0010] After preprocessing the input SMPL-X action sequence, encode it into double-stream action features;

[0011] Through the cross-modal mapping module, the double-stream action features are respectively mapped to the hidden space of the pre-trained large language model through independent projection branches to obtain global conditions and local action prefix embeddings;

[0012] Through the text generation module, the decoder of the pre-trained large language model is used as the backbone network, the text prompt word embedding is extracted based on the text instruction given by the user, the local action prefix embedding is spliced with the text prompt word embedding and input into the decoder, and the global condition is injected into each layer of the decoder through the cross-attention mechanism, and the description text is generated in a self-recurrent manner.

[0013] Preferably, the preprocessing comprises: performing zero-mean unit-variance normalization on the input SMPL-X action sequence to obtain a preprocessed SMPL-X action sequence.

[0014] Preferably, the encoding is a double-flow action feature, comprising:

[0015] mapping the preprocessed SMPL-X action sequence into a frame-level embedding sequence by a skeleton embedding layer of the action encoder;

[0016] concatenating a learnable global token at the head of the frame-level embedding sequence and adding position encoding to obtain an input sequence;

[0017] inputting the input sequence into the action encoder for encoding to obtain a top-level hidden state sequence;

[0018] extracting a feature at the head of the top-level hidden state sequence as a sequence-level global action feature and extracting features at subsequent positions as frame-level local fine-grained action features, the sequence-level global action feature and the frame-level local fine-grained action features being the double-flow action feature.

[0019] Further preferably, the action encoder adopts a VAE encoder to encode the input sequence to obtain the top-level hidden state sequence, and the encoding process is represented as: wherein, is the top-level hidden state sequence, is the top-level hidden state at the i-th frame, is the total number of top-level hidden states, is the VAE encoder, is the input sequence, is a feature space dimension.

[0020] Preferably, each of the independent projection branches adopts a plurality of multilayer perceptron structures constructed based on a gated linear unit for nonlinear projection, and the gating mechanism followed by the projection branch in the nonlinear transformation process is represented as for realizing semantic mapping and dimension adaptation from the action feature dimension to the hidden space of the pre-trained large language model; wherein, and are two parts uniformly divided according to the feature space dimension from an intermediate representation vector output by the same linear layer.

[0021] Preferably, the independent projection branch comprises a global branch and a local branch.

[0022] The global branch is configured to project the sequence-level global action feature in the double-flow action feature to obtain a global condition.

[0023] ​The local branch is configured to project frame-level local fine-grained action features in the dual-flow action features to obtain a local action prefix embedding.

[0024] Optionally, the pre-trained large language model is a language model of Qwen2.5-VL-7B.

[0025] Preferably, the text prompt embedding is extracted based on the user-given text instruction, including:

[0026] Based on the user-given text instruction , the text prompt embedding is obtained through a text embedding layer of the pre-trained large language model: , wherein, is the text prompt embedding, is an embedding operation of the pre-trained large language model, is a sequence length of the text instruction, is a hidden space feature dimension of the pre-trained large language model.

[0027] Preferably, in the cross-attention mechanism, the query vector is derived from the hidden state of the decoder, and the key vector and the value vector are both derived from the global condition.

[0028] Preferably, the description text is generated through a self-recurrent manner, including:

[0029] The feature obtained by splicing the local action prefix embedding and the text prompt embedding is input into the decoder, and each layer of the decoder sequentially performs self-attention calculation, cross-attention calculation and feedforward network calculation to generate the description text.

[0030] The input of the self-attention calculation is based on the hidden state output by the previous layer to obtain a first output through self-attention calculation; the cross-attention calculation takes the first output of the current layer as a query vector and takes the global condition as a key vector and a value vector to obtain a second output through cross-attention calculation; and the feedforward network performs nonlinear transformation on the second output to gradually generate the description text.

[0031] Preferably, the process of generating the description text adopts a three-stage training strategy, including:

[0032] In the first stage, a reconstruction loss and a KL divergence loss are introduced as the basis to introduce a joint rotation reconstruction loss and a forward kinematics position consistency loss to construct a first loss function for optimizing the action encoder to learn stable dual-flow action features; the first loss function is defined as ; wherein the reconstruction loss , and The first Frame target motion features and reconstructed motion features, KL divergence loss , Let KL divergence be the KL divergence. For coding distribution, The distribution is normal; joint rotation reconstruction loss Forward kinematic position consistency loss , and The first Frame target rotation features and reconstructed rotation features The total number of frames in the action sequence. It is an L1 norm. and The first Frame number The target 3D coordinates and reconstructed 3D coordinates of each joint The total number of joints. It is an L2 norm;

[0033] In the second stage, the action encoder trained in the first stage is frozen, and a second loss function related to text generation is constructed. This is used to optimize the cross-modal mapping module and the text generation module, making the global conditional and local action prefix embeddings compatible with the latent space of the pre-trained large language model in terms of distribution and semantics, thereby achieving stable reception and decoding of the pre-trained large language model; the second loss function is defined as Among them, vocabulary-level label smooth cross-entropy loss , The target text length, For the vocabulary index within the target text length, For a vocabulary set, For a subset of the word list, The target distribution after smoothing the vocabulary-level labels. For the target text's first A real vocabulary list, For conditional inputs that include global conditions and local action prefix embeddings, Optimization parameters for large language models; sentence-level loss , For batch size, and The first The reference sentence vector and the generated sentence vector for each sample. It is a function with maximum value. For the first The reference sentence vector of a difficult negative sample, For cosine distance calculation, For interval hyperparameters; length regression loss , For the first The length of the description text for each sample. For the first The objective is to supervise the length of the true descriptive text of each sample.

[0034] In the third stage, the feature injection into the decoder is performed by fixing the global conditions and the concatenation of the local action prefix embedding sequence and the text prompt word embedding. The second loss function is also used to supervise and fine-tune the ability to generate descriptive text, so as to stably generate descriptive text.

[0035] This invention also provides an SMPL-X action-to-text generation system that fuses global and local features, used to implement the aforementioned SMPL-X action-to-text generation method that fuses global and local features, comprising:

[0036] The action generation module is used to preprocess the input SMPL-X action sequence and encode it into dual-stream action features;

[0037] The cross-modal mapping module is used to map the two-stream action features to the latent space of the pre-trained large language model through independent projection branches, so as to obtain global conditional and local action prefix embeddings.

[0038] The text generation module uses a pre-trained large language model decoder as the backbone network to extract text prompt word embeddings based on user-given text instructions. It concatenates the local action prefix embeddings with the text prompt word embeddings and inputs them into the decoder. It also injects global conditions into each layer of the decoder through a cross-attention mechanism and generates descriptive text through an autoregressive approach.

[0039] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0040] The application provides a SMPL-X action-to-text generation method with global and local feature fusion. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows.

[0042] Figure 1 A flowchart of the SMPL-X action-to-text generation method with global and local feature fusion provided by the present application.

[0043] Figure 2 A flowchart of encoding the double-stream action features provided by the embodiment.

[0044] Figure 3 A flowchart of the text generation module generating description text provided by the embodiment.

[0045] Figure 4 A structural diagram of the SMPL-X action-to-text generation system with global and local feature fusion provided by the embodiment of the present application.

[0046] Figure 5 A comparison diagram of the action description generation effect provided by the embodiment of the present application and the effect of the prior art. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and given in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0048] The inventive concept of the present application is that, in order to solve the problems of easy loss of fine-grained action information, insufficient geometric perception ability of action encoding features, attenuation of action conditions in deep decoding, and lack of sentence-level semantics and length structure constraints in generated text in the SMPL-X action-to-text generation task, the present application provides a global and local feature fusion SMPL-X action-to-text generation method. The method uses continuous double-flow action representation at the action encoding end to simultaneously retain global semantics and local details, and enhances the geometric perception ability of the action encoder extracted features through geometric and kinematic consistency constraints; at the text decoding end, a double-path condition injection mechanism is used, the local feature mapping module is used to provide frame-level detail conditions, and the global feature mapping module is used to continuously inject global semantic constraints in the decoding process combined with cross attention; at the same time, multi-granularity text supervision is combined to improve the semantic consistency, content coverage and length controllability of the generated text, so as to realize stable action description generation.

[0049] As shown in Figure 1 , the global and local feature fusion SMPL-X action-to-text generation method provided by the embodiment includes the following steps:

[0050] S1, after preprocessing the input SMPL-X action sequence, encode it into double-flow action features. The double-flow action features contain both sequence-level global action features and frame-level local fine-grained action features. The sequence-level global action features are used to represent the overall semantic information of the whole action, and the frame-level local fine-grained action features are used to retain the dynamic details such as posture and hand over time.

[0051] In the embodiment, the input SMPL-X action sequence is normalized by zero mean and unit variance to obtain the preprocessed SMPL-X action sequence, wherein is the number of action sequence frames, is the SMPL-X action sequence vector of the th frame.

[0052] Next, the pre-processed SMPL-X motion sequence is encoded into dual-stream motion features as Figure 2 As shown, a variational autoencoder (VAE) in the motion diffusion model (MLD) is used as the backbone network, and a skeleton embedding layer is used to map the pre-processed SMPL-X motion sequence to a 512-dimensional feature space to obtain a frame-level embedding sequence

[0053]

[0054] wherein, is the skeleton embedding layer;

[0055] A learnable global token is concatenated at the beginning of the frame-level embedding sequence to construct the encoder input and superimpose the position encoding to obtain the actual encoder input is input into the motion encoder to obtain a top-level continuous hidden state sequence:

[0056]

[0057] wherein, is the top-level hidden state sequence, is the top-level hidden state at the frame of the top-level hidden state sequence, is the total number of top-level hidden states, is the VAE encoder using the MLD framework, is the input sequence, is the feature space dimension.

[0058] Instead of using the latent motion vector obtained by variational modeling as the motion representation, the top-level hidden state is used as the continuous motion representation, and the dual-stream motion features are obtained based on the input splicing structure decoupling:

[0059]

[0060]

[0061] wherein, is the sequence-level global motion feature, is the frame-level local fine-grained motion feature.

[0062] ​​​​​​S2. Through the cross-modal mapping module, the dual-stream action features are mapped to the latent space of the pre-trained large language model through independent projection branches, obtaining global conditional and local action prefix embeddings. This step is used to map the continuous dual-stream action features output from S1 to the latent space of the text decoder, making them usable as conditional inputs to the pre-trained large language model. The cross-modal mapping module adopts a two-branch structure: the global branch projects the sequence-level global action features to generate a global conditional representation, which is continuously injected as conditional information for cross-attention during text decoding; the local branch projects the frame-level local fine-grained action feature sequence time-by-time to generate an action prefix embedding sequence, which is used together with the text prompt word embeddings after being concatenated in the sequence dimension as decoder input, thereby achieving the adaptation from the action representation space to the language model latent space.

[0063] In the embodiments, such as Figure 3 As shown, the sequence-level global action features are based on the output of S1. With frame-level local fine-grained motion features ,in The cross-modal mapping module is an action semantic mapping module based on gated linear units, containing global and local branches, and defining global and local projection operators respectively to project action features onto the language model's hidden space to obtain global conditions. and local action prefix embedding :

[0064] ,

[0065] ,

[0066] in, and The network structures are consistent but the parameters are independent of each other. They all adopt the action semantic mapping structure based on gated linear units: the input action features are first processed by root mean square normalization, and then projected by a multilayer perceptron formed by stacking several gated linear unit blocks.

[0067] Both independent projection branches (global branch and local branch) employ a multilayer perceptron structure built with several gated linear units for nonlinear projection, and the gating mechanism followed by the projection branches during the nonlinear transformation is expressed as follows: This is used to achieve semantic mapping and dimensional adaptation from the action feature dimension to the latent space of a pre-trained large language model; among which, and The intermediate representation vector output by the same linear layer is divided into two parts by uniformly dividing the feature space dimension.

[0068] S3, by the text generation module, taking the decoder of the pre-trained large language model as the backbone network, based on the text instruction given by the user to extract the text prompt word embedding, concatenating the local action prefix embedding with the text prompt word embedding and inputting the decoder, and injecting the global condition into each layer of the decoder through the cross-attention mechanism, to generate the description text in a self-recurrent manner.

[0069] The text generation module in the embodiment takes the decoder of the language model of Qwen2.5-VL-7B as the backbone network, and the decoder is composed of 28 layers of Transformer. Among them, the local action prefix embedding sequence is concatenated with the text prompt word embedding in the sequence dimension to participate in the self-attention modeling, and the global condition is continuously injected as the condition information of cross-attention in each layer of the decoder, so as to retain the local timing details and global semantic constraints at the same time in the decoding process.

[0070] Specifically, based on the global condition output by S2 and the local action prefix embedding , based on the text instruction given by the user , the text prompt word embedding is obtained through the text embedding layer of the pre-trained large language model: , wherein, is the text prompt word embedding, is the embedding operation of the pre-trained large language model, is the length of the text instruction sequence, is the hidden space feature dimension of the pre-trained large language model.

[0071] Concatenate the local action prefix and the prompt word embedding in the sequence to obtain the decoder input .

[0072] In the decoding block of the first layer of the decoder (L1), update the hidden state in the order of “self-attention-cross-attention-feedforward network”. Let the decoding input of the first layer be , then:

[0073] ,

[0074] ,

[0075] ,

[0076] wherein, denotes the normalization operation, denotes the multi-head self-attention calculation, denotes the multi-head cross-attention calculation, and FFN(⋅) denotes the feedforward network, The first output is obtained from multi-head self-attention computation. The second output is calculated using multi-head cross-attention. This is the generated description text.

[0077] Cross-attention satisfies the following conditions: the query vector originates from the current hidden state, and both the key and value vectors originate from global conditions. Defined as:

[0078] ,

[0079] in, For the first The learnable parameters of each layer are decoded through 28 layers to obtain the top-level hidden state. The descriptive text is generated step by step through the output header, and the generation location is... have ,in express In generating the latent vector at position a, This is the output header for the vocabulary projection.

[0080] In this embodiment, the method employs a three-stage training strategy, including:

[0081] The first phase focuses on rebuilding the losses. With KL divergence loss Based on this, joint rotation reconstruction loss is introduced. Loss of consistency with forward kinematic position Construct the first loss function Used to optimize the action encoder to learn stable two-stream action features; the first loss function is defined as follows: Among them, reconstruction losses , and The first Frame target motion features and reconstructed motion features, KL divergence loss , Let KL divergence be the KL divergence. For coding distribution, The distribution is normal; joint rotation reconstruction loss Forward kinematic position consistency loss , and The first Frame target rotation features and reconstructed rotation features The total number of frames in the action sequence. It is an L1 norm. and The first frame the target three-dimensional coordinates of the joint and the reconstructed three-dimensional coordinates, is the total number of joints, is the L2 norm;

[0082] In the second stage, the action encoder trained in the first stage is frozen, and a second loss function related to text generation is constructed to optimize the cross-modal mapping module and the text generation module, so that the global condition and the local action prefix embedding are compatible in distribution and semantics with the hidden space of the pre-trained large language model, and the stable reception and decoding of the pre-trained large language model are realized; the second loss function is defined as wherein the vocabulary-level label smoothed cross-entropy loss , is the target text length, is the vocabulary index in the target text length, is the vocabulary set, is the vocabulary subset, is the target distribution after vocabulary-level label smoothing, is the th real vocabulary of the target text, is the conditional input containing the global condition and the local action prefix embedding, is the optimization parameter of the large language model; the sentence-level loss , is the batch size, and are the reference sentence vector and the generated sentence vector of the th sample, respectively, is the max function, is the reference sentence vector of the th difficult negative sample, is the cosine distance calculation, is the interval hyperparameter; the length regression loss , is the description text length of the th sample, is the real description text length supervision target of the th sample;

[0083] In the third stage, the way of fixing the global condition and the local action prefix embedding sequence and the feature injection decoder after splicing the text prompt word embedding is also used to supervise and fine-tune the generation ability of the description text using the second loss function, for stable generation of the description text.

[0084] Based on the same inventive concept, the embodiment also provides a global and local feature fusion SMPL-X action to text generation system for implementing the global and local feature fusion SMPL-X action to text generation method. Figure 4 As shown, comprising:

[0085] An action generation module for pre-processing the input SMPL-X action sequence and encoding it into double-flow action features;

[0086] A cross-modal mapping module for mapping the double-flow action features to the hidden space of a pre-trained large language model through independent projection branches respectively, to obtain global conditions and local action prefix embeddings;

[0087] A text generation module for taking the decoder of the pre-trained large language model as the backbone network, extracting text prompt word embeddings based on the text instructions given by the user, inputting the local action prefix embeddings and the text prompt word embeddings into the decoder after splicing, and injecting the global conditions into each layer of the decoder through the cross-attention mechanism, and generating description text through the autoregressive method.

[0088] In order to further verify the feasibility and effectiveness of the global and local feature fusion SMPL-X action to text generation method proposed in the present application, comparative experiments were carried out on the public action description dataset, and representative complex action samples were randomly selected for visual analysis.

[0089] As shown, Figure 5 The key frame visualization rendering diagram of the input SMPL-X action sequence is shown, which contains limb details with specific semantics (for example: the human body jumps to the right side while keeping the body orientation unchanged).

[0090] The description text generated by the prior art Lamp model and the description text generated by the method of the present application are compared. Through comparative analysis, it can be seen that the prior art Lamp model lacks independent perception and fusion of local fine-grained action features, and the description text generated by the prior art Lamp model can only capture the coarse-grained global semantics of "side jump", and the output description text is "a person completes a side jump action to the right direction", but fails to accurately describe the key fine-grained posture constraint "maintain the trunk forward" in the action process.

[0091] In contrast, the method of the present application benefits from introducing continuous double-flow action representation at the encoding end and adopting local action prefix embedding mechanism at the decoding end, which can effectively capture and retain fine-grained information such as hand micro-movement and body orientation constraint, and generate description text that accurately represents the complex semantics of "jumping to the right side without turning the body", which is highly consistent with the real labeled text "a person completes a jump displacement to the right side without turning the body to face the target direction while keeping the body facing forward".

[0092] The above experimental results have proved that the technical solution proposed by the present application, which fuses sequence-level global features and frame-level local fine-grained features, can significantly solve the problem of easy loss of fine-grained action information in the prior art, greatly improves the accuracy and richness of action description while ensuring the smoothness of generated text sentences, and verifies the authenticity and effectiveness of the technical solution of the present application.

[0093] In addition, the terms "upper", "lower", "inner", "outer", "front", "back" are only used for description purposes, and cannot be understood as indicating or implying relative importance. Unless otherwise specified, the relative steps, numerical expressions and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application. Of course, the above description is only a specific embodiment of the present application, and is not intended to limit the scope of the present application. Any equivalent changes or modifications made in accordance with the technical scope of the present application should be included in the scope of the present application.

[0094] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present application, and are used to illustrate the technical solutions of the present application, but are not limiting. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for SMPL-X action-to-text generation with global and local feature fusion, characterized in that, The method comprises the following steps: After preprocessing the input SMPL-X action sequence, encode it into double-stream action features; The encoding into double-stream action features comprises: mapping the preprocessed SMPL-X action sequence into a frame-level embedding sequence through a skeleton embedding layer of an action encoder; concatenating a learnable global mark at the head of the frame-level embedding sequence and adding position encoding to obtain an input sequence; inputting the input sequence into the action encoder for encoding to obtain a top-level hidden state sequence; extracting the feature at the head of the top-level hidden state sequence as a sequence-level global action feature and extracting the features at subsequent positions as frame-level local fine-grained action features, and the sequence-level global action feature and the frame-level local fine-grained action feature are taken as double-stream action features; Through a cross-modal mapping module, the double-stream action features are respectively mapped to the hidden space of a pre-trained large language model through independent projection branches to obtain global conditions and local action prefix embeddings; the independent projection branches comprise a global branch and a local branch; the global branch is used for projecting the sequence-level global action feature in the double-stream action features to obtain global conditions; and the local branch is used for projecting the frame-level local fine-grained action features in the double-stream action features at each time point to obtain local action prefix embeddings; Through a text generation module, a decoder of the pre-trained large language model is taken as a backbone network, a text prompt word embedding is extracted based on a text instruction given by a user, the local action prefix embedding and the text prompt word embedding are concatenated and input into the decoder, the global conditions are injected into each layer of the decoder through a cross-attention mechanism, and description text is generated in a self-recurrent manner; the generation of the description text in the self-recurrent manner comprises: inputting the feature obtained by concatenating the local action prefix embedding and the text prompt word embedding into the decoder, each layer of the decoder sequentially performs self-attention calculation, cross-attention calculation and feedforward network calculation to generate the description text; wherein the self-attention calculation is based on the hidden state output by the previous layer to perform self-attention calculation to obtain a first output; the cross-attention calculation takes the first output of the current layer as a query vector, takes the global conditions as a key vector and a value vector, and performs cross-attention calculation to obtain a second output; and the feedforward network performs nonlinear transformation on the second output to gradually generate the description text.

2. The global and local feature fused SMPL-X action-to-text generation method of claim 1, wherein, The action encoder encodes the input sequence using a VAE encoder to obtain a sequence of top-level hidden states, the encoding process is represented as: , where, is the sequence of top-level hidden states, the frame top-level hidden state, is the total number of top-level hidden states, is the VAE encoder, is the input sequence, is the feature space dimension. 3.The global and local feature fused SMPL-X action-to-text generation method of claim 1, wherein, The independent projection branches each adopt a plurality of multilayer perceptron structures constructed based on a gated linear unit for nonlinear projection, and a gating mechanism followed by the projection branch in the nonlinear transformation process is represented as , for realizing semantic mapping and dimension adaptation from the action feature dimension to the hidden space of the pre-trained large language model; wherein, and are two parts uniformly segmented according to the feature space dimension from the intermediate representation vector output by the same linear layer.

4. The global and local feature fused SMPL-X action-to-text generation method of claim 1, wherein, The extraction of the text prompt word embedding based on the text instruction given by the user comprises: Based on a user-given text instruction obtaining a text prompt word embedding through a text embedding layer of a pre-trained large language model: wherein, is the text prompt word embedding, is an embedding operation of the pre-trained large language model, is a sequence length of the text instruction, is a hidden space feature dimension of the pre-trained large language model.

5. The global and local feature fused SMPL-X action-to-text generation method of claim 1, wherein, The generation process of the description text adopts a three-stage training strategy, which comprises: In the first stage, a reconstruction loss With the KL divergence loss As a basis, a joint rotation reconstruction loss Consistent with the forward kinematics position loss , the first loss function is constructed for optimizing the motion encoder to learn stable dual-stream motion features; In the second stage, the action encoder trained in the first stage is frozen, and a second loss function related to text generation is constructed for optimizing the cross-modal mapping module and the text generation module, so that the global condition and the local action prefix are compatible with the hidden space of the pre-trained large language model in distribution and semantics, and the pre-trained large language model is stably received and decoded. In the third stage, the manner of fixing the global conditions and the feature obtained by concatenating the local action prefix embedding sequence and the text prompt word embedding and injecting it into the decoder is also used to supervise and fine-tune the generation ability of the description text by using the second loss function, so as to stabilize the generation of the description text.

6. The global and local feature fused SMPL-X action-to-text generation method of claim 5, wherein, The first loss function is defined as follows: Among them, reconstruction losses , and The first Frame target motion features and reconstructed motion features, KL divergence loss , Let KL divergence be the KL divergence. For coding distribution, The distribution is normal; joint rotation reconstruction loss Forward kinematic position consistency loss , and The first Frame target rotation features and reconstructed rotation features The total number of frames. It is an L1 norm. and The first Frame number The target 3D coordinates and reconstructed 3D coordinates of each joint The total number of joints. It is an L2 norm; The second loss function is defined as where the vocabulary-level label smoothed cross-entropy loss is is the target text length, is the vocabulary index in the target text length, is the vocabulary set, is the vocabulary subset, is the smoothed target distribution for the vocabulary-level label, is the th real vocabulary of the target text, is the conditional input containing global condition and local action prefix embedding, is the optimization parameter of the large language model; the sentence-level loss is the batch size, and are the reference sentence vector and the generated sentence vector of the th sample, respectively, is the max function, is the reference sentence vector of the th hard negative sample, is the cosine distance calculation, is the margin hyperparameter; the length regression loss is the description text length of the th sample, is the real description text length supervision target of the th sample.​​​ 7. A global and local feature fused SMPL-X action-to-text generation system, comprising: The method for realizing the global and local feature fusion of the SMPL-X action to text generation method in any one of claims 1-6 comprises: an action generation module configured to preprocess an input SMPL-X action sequence and encode it into double-stream action features; A cross-modal mapping module is configured to map the dual-flow action features to the hidden space of a pre-trained large language model through independent projection branches respectively, to obtain global condition and local action prefix embeddings; A text generation module is configured to take the decoder of the pre-trained large language model as a backbone network, extract a text prompt word embedding based on a text instruction given by a user, input the local action prefix embedding and the text prompt word embedding into the decoder after splicing, inject the global condition into each layer of the decoder through a cross-attention mechanism, and generate a description text through a self-recurrent manner.

Images