Human action understanding method based on large language model and adaptive contrast decoding

Abstract

The application belongs to the technical field of artificial intelligence, and discloses a human action understanding method based on a large language model and adaptive contrast decoding, proposes a FLARE framework, introduces a pre-trained frozen large language model LLM and a frozen action encoder, aligns the action and language semantics, and uses an adaptive contrast decoding AdaCD method to reduce excessive dependence on language prior, thereby improving the model's understanding ability of the action, enabling the method to handle various action understanding tasks, improving the generalization ability of the human action understanding method, and reducing the task specificity and computing resource consumption problems in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a human action understanding method based on a large language model and adaptive contrast decoding. BACKGROUND

[0002] Human action understanding is an important research direction in the fields of computer vision and artificial intelligence, aiming to understand and interpret human action behavior through computer algorithms and models. Research in this field involves extracting and analyzing human poses, motion trajectories, and action types from video or sensor data to recognize, classify, and predict human behavior, such as action description, text retrieval, and action recognition tasks.

[0003] Existing human action understanding methods are usually designed and optimized for specific tasks, lacking the ability to generalize across different tasks, i.e., they cannot adapt to multiple different music and dance styles. Moreover, due to excessive reliance on frozen large language model language priors, existing methods may produce hallucination problems that do not match actual actions, resulting in inaccurate understanding of human actions. SUMMARY

[0004] The present application aims to address the above-mentioned problems by providing a human action understanding method based on a large language model and adaptive contrast decoding, which aims to improve the existing human action understanding method's poor generalization ability due to specific task design and insufficient accuracy when generating action descriptions consistent with text prompts.

[0005] The technical solution adopted by the present application is as follows: a human action understanding method based on a large language model and adaptive contrast decoding, the method comprising the following steps: determining the task of human action understanding, combining a frozen autoregressive large language model LLM and a pre-trained action encoder to form a unified FLARE framework, aligning action data and language semantics, and then introducing an adaptive contrast decoding method AdaCD to alleviate hallucination problems by adjusting the weights of relevant markers to optimize model performance; preprocessing action data and language data, using an action encoder to encode human actions, extracting their features and generating corresponding action embeddings, and keeping the pre-trained weights of the action encoder unchanged to reduce computational cost; the LLM uses a Byte-Pair Encoding tokenizer BPE to convert text input into a series of tokens (t1, …, t N ), and keeps the model parameters frozen; aligning the features of actions and language, performing multi-task training, and then performing adaptive contrast decoding AdaCD optimization, and verifying the results through inference and evaluation.

[0006] Further, the FLARE framework comprises the following modules: a language model, which inherits from a recurrent large language model (LLM) pre-trained on large text data and kept frozen in the application, the LLM serializes action descriptions through a BPE tokenizer, aiming to maximize the log-likelihood of these sequences; an action model, which uses a pre-trained action encoder to generate action embeddings from action inputs and converts them into a format consistent with the text embedding dimension through a projection layer; the language model and the action model are used for language and action preprocessing; an action and language integration module, the action embeddings generated by the action encoder are aligned with the text embedding space through a simple linear layer; the action and language integration is used to align language and action.

[0007] Further, the action data and language data preprocessing method comprises: training a recurrent large language model (LLM) on a large text data set, fixing the parameters θ in the recurrent large language model p θ , given an action description, using a Byte-Pair Encoding tokenizer (BPE) to convert the action into a series of tokens (t1,…,t N ) for processing text, and maximizing the optimization of the log-likelihood of the token sequence, the formula is:

[0008]

[0009] Where p θ (t n |t1,…,t n-1 ) represents the probability of the nth token t n appearing under the condition that all previous tokens are given;

[0010] Through the action block method, a pre-trained action backbone model is used to generate action embeddings from <cls>tokenization of the action sequence y into tokens (t1, …, tN) and the text description x into tokens (t1, …, tN) is performed m .

[0011] Further, the action and language feature alignment method comprises: generating an action embedding e m from the action input by the action encoder, and the text of the LLM is aligned in the linear projection layer space, the action encoder first generates an embedding e m from the action input, and then converts it into a format matching the dimension of the text embedding through a linear mapping W c .

[0012] Further, the multi-task training comprises: action-text retrieval, including bidirectional retrieval of action to text M2T and text to action T2M, by embedding the action sequence and the text description into a shared feature space; e t and e m represent the embedding of the text description and the action sequence in the space respectively, and the similarity between them is calculated by cosine similarity:

[0013]

[0014] Action description generation is a task of generating a text description based on action data, given an action y and an action embedding e m , the log-likelihood x of tokenization of (t1, …, tN) is: L

[0015]

[0016] The generation loss function

[0017]

[0018] where N represents N pairs of action-text generation.

[0019] Further, in the training process, the InfoNCE loss is used to optimize the model, and the model is optimized by minimizing the distance between positive samples and maximizing the distance between negative samples, and the InfoNCE loss is defined as follows:

[0020]

[0021] where τ is the temperature parameter, the positive sample is the matched text-action pair, and the negative sample is the unmatched text-action pair;

[0022] Further, the adaptive contrast decoding AdaCD optimization method specifically comprises: setting two kinds of scene prediction logits: with action input, i.e. action description generation, and without action input, i.e. text to action retrieval, and setting the above two kinds of scenes as logits​ cap and logit ret ; define contrastive decoding score: λ CD = logit cap -logit ret ; get λ CD After that, the value of λ CD needs to be compared with a preset threshold γ; when λ CD is lower than the threshold γ, the importance of the token is evaluated using the description logit cap ; when λ CD is negative, the LLM will reduce the sampling probability of the token to avoid generating low-relevant descriptions.

[0023] Further, in the autoregressive large-scale language model LLM <ret>token vocabulary, learning its embeddings while keeping other token embeddings constant during training; at training time, <ret>appended to the end of the input description, this setting generates <ret>, help to integrate action retrieval into generated text; adopt double mapping strategy, use linear mapping W t will be <ret>the hidden representation of the token is converted into a suitable textual embedding e t , another linear mapping W m the action embedding e m is projected into the retrieval space.

[0024] Further, the action description generation formalizes the action description generation as a task of generating text tokens under the condition of action prefix, and converts the action prefix into a format aligned with the text space through a projection layer.

[0025] Further, the reasoning and evaluation verification includes: action retrieval and description generation, through <ret>token, token, and <cls>Token helps to retrieve target actions or generate descriptions consistent with input actions; result evaluation: through multi-task evaluation of action description generation accuracy, action-text retrieval precision and other indicators, verify the performance of the FLARE framework in action understanding tasks.

[0026] In summary, due to the adoption of the technical solutions described above, the beneficial effects of the present application are:

[0027] The present application proposes a FLARE framework, by introducing a pre-trained frozen large language model LLM and a frozen action encoder, aligning action and language semantics, and using an adaptive contrast decoding AdaCD method, reducing the over-reliance on language priors, thereby improving the model's understanding of actions, enabling the method to handle a variety of action understanding tasks, improving the generalization ability of human action understanding methods, and reducing the task-specificity and computational resource consumption problems existing in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the prior art of the present application and the method flowchart of the present application;

[0029] wherein (a) is the method flowchart of the prior art, and (b) is the method flowchart of the present application

[0030] Figure 2 is a schematic diagram of the FLARE framework of the present application;

[0031] wherein (a) is the flow of action description, (b) is the flow of adaptive contrast decoding method, and (c) is the flow of action-text retrieval task;

[0032] Figure 3 is the action description result diagram of the present application;

[0033] Figure 4 is the text-to-action retrieval diagram of the present application;

[0034] wherein (a) is the text-to-action retrieval diagram of the present application, and (b) is the text-to-action retrieval diagram of other methods;

[0035] Figure 5 is the action-to-text retrieval diagram of the present application;

[0036] wherein (a) is the action-to-text retrieval diagram of the present application, and (b) is the action-to-text retrieval diagram of other methods. DETAILED DESCRIPTION

[0037] The present application will be described in detail below with reference to the accompanying drawings.

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] like Figure 1 As shown, existing methods have the following shortcomings when dealing with human action understanding: (1) poor generalization ability due to specific task design; (2) insufficient accuracy in generating action descriptions consistent with text prompts; and (3) high computational resource consumption and training difficulty of complex models.

[0040] To address these issues and improve the accuracy and generalization ability of human action understanding, this invention proposes a unified new framework called FLARE. Compared to existing frameworks, which differ for each task and are limited to specific tasks, the FLARE framework of this invention can handle multiple tasks with a single framework. By introducing a pre-trained frozen large language model (LLM) and a frozen action encoder, it aims to achieve a deeper and broader understanding of actions.

[0041] Because large language models often produce errors (illusions) when applied to action understanding, the contrastive decoding method AdaCD was proposed. The FLARE framework combines the advantages of language models and action encoders to align actions with linguistic semantics. It utilizes the adaptive contrastive decoding method AdaCD to reduce over-reliance on linguistic priors, thereby improving the model's ability to understand actions. The action encoder extracts action features for the decoder, which then decodes these features to reconstruct actions or dance sequences. Specifically, FLARE improves action representation by enhancing the weights of action-related tokens, enabling it to handle various action understanding tasks while reducing the task specificity and computational resource consumption issues present in existing technologies.

[0042] A human action understanding method based on a large language model and adaptive contrastive decoding includes the following steps: Defining the human action understanding task; combining a frozen autoregressive large-scale language model (LLM) and a pre-trained action encoder to form a unified FLARE framework; aligning action data with language semantics; then introducing an adaptive contrastive decoding method, AdaCD, to mitigate the illusion problem and optimize model performance by adjusting the weights of relevant tokens; preprocessing the action and language data; using the action encoder to encode human actions, extracting their features, and generating corresponding action embeddings; keeping the pre-trained weights of the action encoder unchanged to reduce computational cost; and using the Byte-Pair Encoding (BPE) tokenizer (LLM) to convert the text input into a series of tokens (t1,…,t…). N ), and keeping the model parameters frozen, ensuring that the pre-training knowledge of the large-scale language model can be used for this task; aligning the features of actions and language, performing multi-task training, and then performing adaptive contrast decoding AdaCD optimization, and verifying the results through reasoning and evaluation.

[0043] By confirming the task of human action understanding, i.e. action description generation, action-text retrieval and action recognition; redesigning the model framework, i.e. combining the frozen autoregressive large language model LLM and the pre-trained action encoder to form a unified FLARE framework; finally, through the adaptive contrast decoding AdaCD method, the text token closely related to the input action is trained to solve the illusion problem of LLM, i.e. due to the excessive reliance on the language prior of the frozen large language model, the existing method may produce the happy hour problem that does not conform to the actual action, leading to inaccurate understanding of human action, thereby improving the understanding ability of human action.

[0044] As shown in Figure 2 , the FLARE framework includes the following modules:

[0045] Language model, this framework inherits the autoregressive large language model LLM, the above model is pre-trained on large-scale text data, and remains frozen in application, LLM uses BPE tokenizer to serialize the action description, the goal is to maximize the log-likelihood of these sequences;

[0046] Action model, this model uses a pre-trained action encoder to generate action embeddings from action inputs, and converts them into a format consistent with the text embedding dimension through a projection layer;

[0047] The language model and the action model are used for pre-processing of language and action;

[0048] Action and language integration module, the action embedding generated by the action encoder is aligned with the text embedding space through a simple linear layer, thereby enhancing the LLM's understanding ability of action;

[0049] The action and language integration is used to align language and action.

[0050] The pre-processing method of the action data and the language data includes:

[0051] Train the autoregressive large language model LLM on a large-scale text data set, fix the parameters θ in the autoregressive large language model p θ , given an action description, use the Byte-Pair Encoding tokenizer BPE to convert the action into a series of tokens (t1,…,t N ) for processing text, and maximize the log-likelihood of the token sequence, the formula is:

[0052]

[0053] where p θ (t n |t1,...,t n-1 ) represents the probability of the nth token t n to appear given all previous tokens;

[0054] By the action block method, a pre-trained action backbone model is used to extract features from <cls>The token generates an action embedded in e. m .

[0055] The method for aligning action and language features includes:

[0056] To enable LLM models to understand human actions, the action data is converted into a format compatible with LLM models, and the actions are embedded in e m The text in the LLM is aligned spatially through a linear projection layer. The action encoder first generates an embedding e from the action input. m Then through linear mapping W c This is converted into a format that matches the text embedding dimension. This process ensures seamless integration of actions and text and enhances the model's ability to understand actions and text.

[0057] like Figures 3-5 As shown, the multi-task training includes:

[0058] Action-to-text retrieval, including bidirectional retrieval of action-to-text (M2T) and text-to-action (T2M), is aligned by embedding action sequences and text descriptions into a shared feature space; t and e m Let represent the embeddings of text descriptions and action sequences in this space, respectively. Their similarity is calculated using cosine similarity:

[0059]

[0060] Action description generation is the task described in this paper, which is based on action data, given an action y and an action embedding e. m , using (t1,…,t L The log-likelihood x after tokenization is:

[0061]

[0062] Generate loss function

[0063]

[0064] Where N represents N pairs of action-text generation.

[0065] During the training of action-text retrieval, the InfoNCE loss is used to optimize the model. This optimization is achieved by minimizing the distance between positive samples and maximizing the distance between negative samples. The InfoNCE loss is defined as follows:

[0066]

[0067] where τ is a temperature parameter, positive samples are matching text-action pairs, and negative samples are non-matching text-action pairs.

[0068] introducing in said autoregressive large language model LLM <ret>token vocabulary, learning its embeddings during training while keeping other token embeddings constant;

[0069] At training time, <ret>appended to the end of the input description, this setting generates <ret>, to help integrate action retrieval into generated text;

[0070] A double mapping strategy is adopted, using a linear mapping W t will be <ret>The hidden representation of the token is converted into a suitable text embedding e t , another linear mapping W m The action embedding e m is projected into the retrieval space.

[0071] The action description generation formalizes the action description generation as a task of generating text tokens under the condition of action prefix, and converts the action prefix into a format aligned with the text space through a projection layer.

[0072] As Figure 3 shown, input the action, and the embodiment can generate a more comprehensive and accurate description, where the red font is the key feature of the action, and the more key features, the more accurate the description.

[0073] Figure 4 represents the conversion from the input text to the action, Figure 5 represents the conversion from the input action to the text, and through the alignment of the action and the text embedding, the bidirectional retrieval of the action to the text and the text to the action is realized; then use the InfoNCE loss function to optimize the distance of positive and negative samples, to ensure that the matching action-text pair is closer, and the unmatched one is farther away.

[0074] In order to deal with the hallucination problem caused by excessive reliance on frozen LLM language priori, an adaptive contrast decoding AdaCD is proposed, which emphasizes the information related to the action to improve the text-action alignment ability and enhance the understanding of human action.

[0075] The adaptive contrast decoding AdaCD optimization method specifically includes: setting the prediction logits of two scenes: with action input, i.e. action description generation, and without action input, i.e. text-to-action retrieval, setting the above two scenes as logits cap and logits ret respectively; define the contrast decoding score: λ CD = logits cap -logits ret , which can highlight the difference between the prediction logits in the two scenes; get λ CD After that, the value of λ CD needs to be compared with the preset threshold γ; when λ CD is lower than the threshold γ, the description logits cap is used to evaluate the importance of the token; when λ CD is negative, the LLM will reduce the sampling probability of the token, avoiding generating low-relevant descriptions.

[0076] The method takes advantage of the characteristics of the two tasks of having action input and not having action input, only needs a small amount of additional training cost and no additional inference cost.

[0077] The above merely provides the preferred embodiments of the application, and is not intended to limit the application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the application shall fall within the protection scope of the application.< / ret> < / ret> < / ret> < / ret> < / cls> < / cls> < / ret> < / ret> < / ret> < / ret> < / ret> < / cls>

Claims

1. A human action understanding method based on a large language model and adaptive contrastive decoding, characterized in that, The method comprises the following steps: Determine the task of human action understanding, combine the frozen autoregressive large language model LLM and the pre-trained action encoder, form a unified FLARE framework, align the action data and the language semantics, and then introduce an adaptive contrast decoding method AdaCD to alleviate the hallucination problem by adjusting the weight of the relevant mark to optimize the model performance; The action data and the language data are preprocessed, a human body action is encoded using an action encoder, a feature of the human body action is extracted, and a corresponding action embedding is generated, and the action encoder keeps the pre-trained weight unchanged; the LLM converts the text input into a series of and keeps the model parameter frozen state; Align the features of actions and language, perform multi-task training, and then perform adaptive contrast decoding AdaCD optimization, and verify the results through reasoning and evaluation; The FLARE framework comprises the following modules: A language model, the framework inherits the autoregressive large language model LLM, the above-mentioned model is pre-trained on large-scale text data, and remains in a frozen state in application, and the LLM performs sequence processing on action description through a BPE tokenizer, and the goal is to maximize the log-likelihood of these sequences; An action model, the model uses a pre-trained action encoder to generate action embeddings from action input, and converts them into a format consistent with the text embedding dimension through a projection layer; The language model and the action model are used for pre-processing of language and action; An action and language integration module, the action embeddings generated by the action encoder are aligned with the text embedding space through a simple linear layer; The action and language integration is used for aligning language and action; The multi-task training comprises: Action-text retrieval, including bidirectional retrieval of action-to-text M2T and text-to-action T2M, by aligning action sequences and text descriptions by embedding them into a shared feature space; and denote the embedding of the text description and the action sequence in this space, respectively, and the similarity between them is computed by cosine similarity: Action description generation is a task of generating textual descriptions based on action data, given an action and action embeddings , the log-likelihood of tokenization is :​ Generating loss function : wherein, N represents the pair of action-text generation; The adaptive contrast decoding AdaCD optimization method specifically comprises: The prediction logits of two scenarios are set: action input, that is, action description generation, and no action input, that is, text-to-action retrieval. The above two scenarios are set as and respectively. Definition of contrast decoding score: ; The contrast decoding score can highlight the difference between the predicted logits in the two scenarios; obtained After that, the value of the is compared with a preset threshold value; When below a threshold , the importance of the evaluation is described using ; When LLM reduces the sampling probability of the to avoid generating low-relevant descriptions; Introducing vocabulary, learning its embeddings during training while keeping other embeddings constant; At training time, appended to the end of the input description, this setting generates , helping to integrate action retrieval into generated text; Using a double mapping strategy, using linear mapping Converts the hidden representation into a text embedding suitable for retrieval , another linear mapping Projects the action embedding to the retrieval space.

2. The human action understanding method based on a large language model and adaptive contrastive decoding according to claim 1, characterized in that, The pre-processing method of the action data and the language data comprises: training an autoregressive large language model LLM on a large-scale text dataset, the autoregressive large language model parameters fixed, given an action description, converting the action into a sequence of tokens for processing text, and maximizing the optimization log-likelihood of the sequence, which is given by the formula: wherein, represents the probability of the nth token appearing given all the preceding tokens ​ By action block method, action embeddings are generated from pre-trained action backbone models .

3. The human action understanding method based on a large language model and adaptive contrastive decoding according to claim 1, characterized in that, The action and language feature alignment method comprises: Embedding actions And the text of the LLM is aligned through the linear projection layer space, the action encoder first generates an embedding from the action input Then it is converted to a format that matches the dimension of the text embedding through a linear mapping ​ 4. The human action understanding method based on a large language model and adaptive contrastive decoding according to claim 1, characterized in that, During the training process of action-text retrieval, the InfoNCE loss is used to optimize the model, the model is optimized by minimizing the distance between positive samples and maximizing the distance between negative samples, and the InfoNCE loss is defined as follows: wherein is a temperature parameter, positive samples are matching text-action pairs, and negative samples are non-matching text-action pairs.

5. The human action understanding method based on a large language model and adaptive contrastive decoding according to claim 1, characterized in that, The action description generation formalizes the action description generation as the task of generating text under the conditions of an action prefix, converting the action prefix to a format aligned with the text space through a projection layer. ​ 6. The human action understanding method based on a large language model and adaptive contrastive decoding according to any one of claims 1-5, characterized in that, The reasoning and evaluation verification comprises: Action retrieval and description generation, by , and helps to retrieve a target action or generate a description that matches an input action; Result evaluation: the accuracy of action description generation is evaluated through multi-task, the precision index of action-text retrieval is evaluated, and the performance of the FLARE framework on the action understanding task is verified.

Images