Human action understanding method based on large language model and adaptive contrast decoding

Abstract

The invention belongs to the technical field of artificial intelligence, discloses a human action understanding method based on a large language model and self-adaptive contrast decoding, and provides an FLARE framework, which enables actions and language semantics to be aligned by introducing a pre-trained frozen large language model LLM and a frozen action encoder, and utilizes a self-adaptive contrast decoding AdaCD method to understand actions and language semantics. According to the method, excessive dependence on language priori is reduced, so that the understanding ability of the model on actions is improved, the method can process various action understanding tasks, the generalization ability of the human action understanding method is improved, and meanwhile, the problems of task specificity and computing resource consumption in the prior art are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a human action understanding method based on a large language model and adaptive contrast decoding. Background Art

[0002] Human action understanding is an important research direction in the field of computer vision and artificial intelligence, which aims to understand and explain human action behaviors through computer algorithms and models. Research in this field involves extracting and analyzing human postures, action trajectories, and action types from video or sensor data to identify, classify, and predict human behaviors, such as action description, text retrieval, and action recognition tasks.

[0003] Existing methods for understanding human motion are usually designed and optimized for specific tasks, lacking the ability to generalize across different tasks, i.e., unable to adapt to a variety of different music and dance styles. Moreover, due to over-reliance on the language prior of a frozen large language model, existing methods may produce hallucinations that are inconsistent with actual motions, resulting in inaccurate understanding of human motions. Summary of the invention

[0004] The purpose of the present invention is to provide a human action understanding method based on a large language model and adaptive contrast decoding to address the above-mentioned problems, in order to improve the poor generalization ability of existing human action understanding methods due to specific task design and the lack of accuracy in generating action descriptions consistent with text prompts.

[0005] The technical solution adopted by the present invention is as follows: a human action understanding method based on a large language model and adaptive contrastive decoding, the method comprising the following steps: determining a task for human action understanding, combining a frozen autoregressive large-scale language model LLM and a pre-trained action encoder to form a unified FLARE framework, aligning action data and language semantics, and then introducing an adaptive contrastive decoding method AdaCD to reduce the hallucination problem and optimize model performance by adjusting the weights of related tags; preprocessing action data and language data, using an action encoder to encode human actions, extracting their features and generating corresponding action embeddings, and the action encoder keeps the pre-trained weights unchanged to reduce the computational cost; LLM uses a Byte-Pair Encoding tokenizer BPE to convert text input into a series of tokens (t1, ..., t N ) and keep the model parameters frozen; align the features of action and language, perform multi-task training, and then perform adaptive contrast decoding AdaCD optimization, and verify the results through reasoning and evaluation.

[0006] Furthermore, the FLARE framework includes the following modules: a language model, which inherits the autoregressive large-scale language model LLM. The above model is pre-trained on large-scale text data and kept frozen in the application. The LLM serializes the action descriptions through the BPE word segmenter, and the goal is to maximize the log-likelihood of these sequences; an action model, which uses a pre-trained action encoder to generate action embeddings from action inputs and converts them into a format consistent with the text embedding dimension through a projection layer; the language model and action model are used to preprocess language and action; an action and language integration module, the action embeddings generated by the action encoder are aligned with the text embedding space through a simple linear layer; the action and language integration is used to align language and action.

[0007] Furthermore, the method for preprocessing the action data and the language data includes: training an autoregressive large language model LLM on a large text data set, and θ The parameter θ in the example is fixed. Given an action description, the Byte-Pair Encoding tokenizer BPE is used to convert the action into a series of tokens (t1,…,t N ) is used to process text and maximize the logarithmic likelihood of the token sequence. The formula is:

[0008]

[0009] Among them, p θ (t n |t1,…,t n-1 ) means that given all the previous tokens, the nth token t n Probability of occurrence;

[0010] Through the action block method, using the pre-trained action backbone model, <cls>Generate action embedding in token m .

[0011] Furthermore, the method for aligning the action and the language features includes: embedding the action into the m The text of LLM is spatially aligned through a linear projection layer. The action encoder first generates an embedding e from the action input. m , and then through the linear mapping W c Convert it to a format that matches the text embedding dimensions.

[0012] Furthermore, the multi-task training includes: action-text retrieval, including bidirectional retrieval from action to text (M2T) and text to action (T2M), by aligning action sequences and text descriptions by embedding them into a shared feature space; e t and e m Respectively represent the embedding of text description and action sequence in this space, and the similarity between them is calculated by cosine similarity:

[0013]

[0014] Action description generation is the task of generating the description in this paper based on action data. Given an action y and an action embedding e m , using (t1,…,t L ) The log-likelihood x of tokenization is:

[0015]

[0016] Generating loss function

[0017]

[0018] Here, N represents N pairs of action-text generation.

[0019] Furthermore, during the training process, the InfoNCE loss is used to optimize the model by minimizing the distance between positive samples and maximizing the distance between negative samples. The InfoNCE loss is defined as follows:

[0020]

[0021] Where τ is the temperature parameter, the positive sample is the matching text-action pair, and the negative sample is the unmatched text-action pair;

[0022] Furthermore, the adaptive contrastive decoding AdaCD optimization method specifically includes: setting prediction logits for two scenarios: one with action input, i.e., action description generation, and one without action input, i.e., text-to-action retrieval, and setting the above two scenarios as logits cap and logit ret ; Define the contrast decoding score: λ CD =logit cap -logit ret ; get λ CD After that, we need to CD The value of is compared with the preset threshold γ; when λ CD Below the threshold γ, the descriptive logit is used cap To evaluate the importance of a token; when λ CD When it is negative, LLM will reduce the sampling probability of the token to avoid generating low-relevant descriptions.

[0023] Furthermore, the autoregressive large-scale language model LLM is introduced <ret>token vocabulary, learn its embedding during training, while keeping other token embeddings unchanged; during training, <ret>Appended to the end of the input description, this setting is generated at inference time <ret>, helps integrate action retrieval into the generated text; adopts a dual mapping strategy, using the linear mapping W t Will <ret>The hidden representation of the token is converted into a text embedding suitable for retrieval. t , another linear map W m Embed actions into m Projected into this retrieval space.

[0024] Furthermore, the action description generation formalizes the action description generation into a task of generating text tokens under the condition of an action prefix, and converts the action prefix into a format aligned with the text space through a projection layer.

[0025] Furthermore, the reasoning and evaluation verification includes: action retrieval and description generation, through <ret>token, token sum <cls>The token helps to retrieve the target action or generate a description that matches the input action. Result evaluation: Through multi-task evaluation of indicators such as the accuracy of action description generation and the accuracy of action-text retrieval, the performance of the FLARE framework in the action understanding task is verified.

[0026] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0027] The present invention proposes a FLARE framework, which introduces a pre-trained frozen large language model (LLM) and a frozen action encoder to align action and language semantics, and uses the adaptive contrastive decoding (AdaCD) method to reduce excessive reliance on language priors, thereby improving the model's ability to understand actions. This method can handle a variety of action understanding tasks, improves the generalization ability of human action understanding methods, and reduces the task specificity and computing resource consumption problems existing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of the prior art of the present invention and the method of the present invention;

[0029] Among them, (a) is a flowchart of the method of the prior art, and (b) is a flowchart of the method of the present invention.

[0030] Figure 2 is a schematic diagram of the FLARE framework of the present invention;

[0031] Among them, (a) is the process of action description, (b) is the process of adaptive contrast decoding method, and (c) is the process of action-text retrieval task;

[0032] Figure 3 It is an action description result diagram of the present invention;

[0033] Figure 4 is the text-to-action retrieval graph of the present invention;

[0034] Wherein, (a) is the text-to-action retrieval graph of the present invention, and (b) is the text-to-action retrieval graph of other methods;

[0035] Figure 5 It is the action-to-text retrieval graph of the present invention;

[0036] Among them, (a) is the action-to-text retrieval graph of the present invention, and (b) is the action-to-text retrieval graph of other methods. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] like Figure 1 As shown in the figure, existing methods have the following shortcomings when dealing with human motion understanding: (1) poor generalization ability due to specific task design; (2) insufficient accuracy in generating action descriptions consistent with text prompts; and (3) high computing resource consumption and training difficulty of complex models.

[0040] In order to solve these problems and improve the accuracy and generalization ability of human motion understanding, the present invention proposes a unified new framework, called FLARE. Compared with the existing framework, that is, each task has a different framework and can only do specific tasks, the FLARE framework of the present invention can handle multiple tasks in one framework, and is committed to achieving deeper and more extensive motion understanding by introducing a pre-trained frozen large language model LLM and a frozen motion encoder.

[0041] Since large language models will produce many errors, i.e. hallucinations, when applied to action understanding, the contrastive decoding AdaCD method is proposed; the FLARE framework combines the advantages of language models and action encoders to align actions with language semantics, and uses the adaptive contrastive decoding AdaCD method to reduce excessive reliance on language priors, thereby improving the model's ability to understand actions; the action encoder is used to extract action features for the decoder, and the decoder restores the features to actions or dances through decoding. Specifically, FLARE improves the representation of actions by enhancing the weights of action-related tokens, enabling it to handle a variety of action understanding tasks, while reducing the task specificity and computing resource consumption problems existing in the prior art.

[0042] A method for human action understanding based on a large language model and adaptive contrastive decoding, the method comprising the following steps: determining a task for human action understanding, combining a frozen autoregressive large-scale language model (LLM) and a pre-trained action encoder to form a unified FLARE framework, aligning action data and language semantics, and then introducing an adaptive contrastive decoding method AdaCD to reduce the hallucination problem and optimize model performance by adjusting the weights of related tags; pre-processing action data and language data, using an action encoder to encode human actions, extracting their features and generating corresponding action embeddings, and the action encoder keeps the pre-trained weights unchanged to reduce the computational cost; LLM uses a Byte-Pair Encoding tokenizer (BPE) to convert text input into a series of tokens (t1, ..., t N ) and keep the model parameters frozen to ensure that the pre-trained knowledge of the large-scale language model can be used for the task; align the features of the action and language, perform multi-task training, and then perform adaptive contrastive decoding AdaCD optimization, and verify the results through reasoning and evaluation.

[0043] By identifying the tasks of human motion understanding, namely action description generation, action-text retrieval and action recognition; redesigning the model framework, that is, combining the frozen autoregressive large-scale language model (LLM) and the pre-trained action encoder to form a unified FLARE framework; finally, the adaptive contrastive decoding (AdaCD) method is used to train text tokens closely related to the input action to solve the LLM hallucination problem. That is, due to over-reliance on the language prior of the frozen large language model, the existing methods may produce results that are inconsistent with the actual actions, resulting in inaccurate understanding of human motion, thereby improving the ability to understand human motion.

[0044] like Figure 2 As shown, the FLARE framework includes the following modules:

[0045] Language model, this framework inherits the autoregressive large-scale language model LLM, which is pre-trained on large-scale text data and kept frozen in the application. LLM serializes action descriptions through the BPE tokenizer, and the goal is to maximize the log-likelihood of these sequences;

[0046] The action model uses a pre-trained action encoder to generate action embeddings from action inputs and transforms them into a format consistent with the text embedding dimension through a projection layer;

[0047] The language model and action model are used to pre-process the language and action;

[0048] The action and language integration module, where the action embedding generated by the action encoder is aligned with the text embedding space through a simple linear layer, thereby enhancing the LLM's ability to understand actions;

[0049] The action and language integration is used to align language and action.

[0050] The method for preprocessing the motion data and the language data comprises:

[0051] Train the autoregressive large language model LLM on a large text dataset and transform the autoregressive large language model p θ The parameter θ in the example is fixed. Given an action description, the Byte-Pair Encoding tokenizer BPE is used to convert the action into a series of tokens (t1,…,t N ) is used to process text and maximize the logarithmic likelihood of the token sequence. The formula is:

[0052]

[0053] Among them, p θ (t n |t1,...,t n-1 ) means that given all the previous tokens, the nth token t n Probability of occurrence;

[0054] Through the action block method, using the pre-trained action backbone model, <cls>Generate action embedding in token m .

[0055] The method for aligning the action and language features includes:

[0056] In order to enable the LLM model to understand human actions, the action data is converted into a format compatible with the LLM model and the actions are embedded in the m The text of LLM is spatially aligned through a linear projection layer. The action encoder first generates an embedding e from the action input. m , and then through the linear mapping W c Convert it to a format that matches the text embedding dimensions. This process is used to ensure seamless integration of actions and text and improve the model's ability to understand actions and text.

[0057] like Figure 3-5 As shown, the multi-task training includes:

[0058] Action-text retrieval, including bidirectional retrieval from action to text (M2T) and text to action (T2M), aligns action sequences and text descriptions by embedding them into a shared feature space; e t and e m Respectively represent the embedding of text description and action sequence in this space, and the similarity between them is calculated by cosine similarity:

[0059]

[0060] Action description generation is the task of generating the description in this paper based on action data. Given an action y and an action embedding e m , using (t1,…,t L ) The log-likelihood x of tokenization is:

[0061]

[0062] Generating loss function

[0063]

[0064] Here, N represents N pairs of action-text generation.

[0065] During the training process of action-text retrieval, InfoNCE loss is used to optimize the model by minimizing the distance between positive samples and maximizing the distance between negative samples. The InfoNCE loss is defined as follows:

[0066]

[0067] Where τ is the temperature parameter, positive samples are matching text-action pairs, and negative samples are unmatched text-action pairs.

[0068] Introduced in the autoregressive large-scale language model LLM <ret>Token vocabulary, whose embedding is learned during training while keeping other token embeddings unchanged;

[0069] During training, <ret>Appended to the end of the input description, this setting is generated at inference time <ret>,helps integrate action retrieval into the generated text;

[0070] Adopting the dual mapping strategy, using the linear mapping W t Will <ret>The hidden representation of the token is converted into a text embedding suitable for retrieval. t , another linear map W m Embed actions into m Projected into this retrieval space.

[0071] The action description generation formalizes the action description generation as the task of generating text tokens under the condition of action prefixes, and converts the action prefixes into a format aligned with the text space through a projection layer.

[0072] like Figure 3 As shown, the input action, this embodiment can generate a more comprehensive and accurate description, wherein the red font is the key features of the action, the more key features, the more accurate the description.

[0073] Figure 4 Indicates that the input text is converted into an action. Figure 5 It means that the input action is converted into text, and the action-to-text and text-to-action bidirectional retrieval is achieved by aligning the action and text embeddings. The InfoNCE loss function is then used to optimize the distance between positive and negative samples to ensure that the matching action-text pairs are closer and the mismatched ones are farther away.

[0074] To address the hallucination problem caused by over-reliance on frozen LLM language priors, adaptive contrastive decoding (AdaCD) is proposed, which emphasizes action-related information to improve text-action alignment and enhance human action understanding.

[0075] The adaptive contrastive decoding AdaCD optimization method specifically includes: setting prediction logits for two scenarios: one with action input, i.e., action description generation, and one without action input, i.e., text-to-action retrieval, and setting the above two scenarios as logits respectively cap and logit ret ; Define the contrast decoding score: λ CD =logit cap -logit ret , this contrast decoding score can highlight the difference between the predicted logits in these two scenarios; get λ CD After that, we need to CD The value of is compared with the preset threshold γ; when λ CD Below the threshold γ, the descriptive logit is used cap To evaluate the importance of a token; when λ CD When it is negative, LLM will reduce the sampling probability of the token to avoid generating low-relevant descriptions.

[0076] This method takes advantage of the characteristics of both tasks with and without action input, requiring only a small amount of additional training cost and no additional inference cost.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / ret> < / ret> < / ret> < / ret> < / cls> < / cls> < / ret> < / ret> < / ret> < / ret> < / ret> < / cls>

Claims

1. A human action understanding method based on a large language model and adaptive contrastive decoding, characterized in that: The method comprises the following steps: We identify the task of human motion understanding, combine the frozen autoregressive large-scale language model (LLM) and the pre-trained motion encoder to form a unified FLARE framework, align motion data and language semantics, and then introduce an adaptive contrastive decoding method (AdaCD) to reduce the hallucination problem and optimize model performance by adjusting the weights of related tags. The action data and language data are preprocessed, and the action encoder is used to encode the human action, extract its features and generate the corresponding action embedding. The action encoder keeps the pre-trained weights unchanged; LLM uses the Byte-PairEncoding tokenizer BPE to convert the text input into a series of tokens (t 11 ,…,t N ) and keep the model parameters frozen; The features of actions and languages ​​are aligned, multi-task training is performed, and then adaptive contrast decoding AdaCD optimization is performed, and the results are verified through reasoning and evaluation.

2. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 1, characterized in that: The FLARE framework includes the following modules: Language model, this framework inherits the autoregressive large-scale language model LLM, which is pre-trained on large-scale text data and kept frozen in the application. LLM serializes action descriptions through the BPE tokenizer, and the goal is to maximize the log-likelihood of these sequences; The action model uses a pre-trained action encoder to generate action embeddings from the action input and converts them into a format consistent with the text embedding dimension through a projection layer; The language model and action model are used to pre-process the language and action; An action and language integration module, where the action embedding generated by the action encoder is aligned with the text embedding space through a simple linear layer; The action and language integration is used to align language and action.

3. The human action understanding method based on large language model and adaptive contrastive decoding according to claim 2, characterized in that: The method for preprocessing the motion data and the language data comprises: Train the autoregressive large language model LLM on a large text dataset and transform the autoregressive large language model p θ The parameter θ in the example is fixed. Given an action description, the Byte-Pair Encoding tokenizer BPE is used to convert the action into a series of tokens (t1,…,t N ) is used to process text and maximize the logarithmic likelihood of the token sequence. The formula is: Among them, p θ (t n |t1,…,t n-1 ) means that given all the previous tokens, the nth token t n Probability of occurrence; Through the action block method, using the pre-trained action backbone model, <cls>Generate action embedding in token m .< / cls> 4. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 2, characterized in that: The method for aligning the action and language features includes: Embed actions into m The text of LLM is spatially aligned through a linear projection layer. The action encoder first generates an embedding e from the action input. m , and then through the linear mapping W c Convert it to a format that matches the text embedding dimensions.

5. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 1, characterized in that: The multi-task training includes: Action-text retrieval, including bidirectional retrieval from action to text (M2T) and text to action (T2M), aligns action sequences and text descriptions by embedding them into a shared feature space; e t and e m Respectively represent the embedding of text description and action sequence in this space, and the similarity between them is calculated by cosine similarity: Action description generation is the task of generating the description in this paper based on action data. Given an action y and an action embedding e m , using (t1,…,t L ) The log-likelihood x of tokenization is: Generating loss function Here, N represents N pairs of action-text generation.

6. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 5, characterized in that: During the training process of action-text retrieval, InfoNCE loss is used to optimize the model by minimizing the distance between positive samples and maximizing the distance between negative samples. The InfoNCE loss is defined as follows: Where τ is the temperature parameter, positive samples are matching text-action pairs, and negative samples are unmatched text-action pairs.

7. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 1, characterized in that: The adaptive contrast decoding AdaCD optimization method specifically includes: Set the prediction logits for two scenarios: with action input, i.e., action description generation, and without action input, i.e., text-to-action retrieval. Set the above two scenarios as logits cap and logit ret ; Define the contrast decoding score: λ CD =logit cap -logit ret ; Get λ CD After that, we need to CD The value of is compared with the preset threshold value γ; When CD Below the threshold γ, the descriptive logit is used cap To evaluate the importance of tokens; When CD When it is negative, LLM will reduce the sampling probability of the token to avoid generating low-relevant descriptions.

8. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 3, characterized in that: Introduced in the autoregressive large-scale language model LLM <ret> Token vocabulary, whose embedding is learned during training while keeping other token embeddings unchanged;< / ret> During training, <ret>Appended to the end of the input description, this setting is generated at inference time <ret> ,helps integrate action retrieval into the generated text;< / ret> < / ret> Adopting the dual mapping strategy, using the linear mapping W t Will <ret>The hidden representation of the token is converted into a text embedding suitable for retrieval. t , another linear map W m Embed actions into m Projected into this retrieval space.< / ret> 9. The method for human action understanding based on large language model and adaptive contrastive decoding according to claim 5, characterized in that: The action description generation formalizes the action description generation as the task of generating text tokens under the condition of action prefixes, and converts the action prefixes into a format aligned with the text space through a projection layer.

10. The method for human action understanding based on large language model and adaptive contrastive decoding according to claims 1-9, characterized in that: The reasoning and evaluation verification includes: Action retrieval and description generation, through <ret>token, token sum <cls> The token helps to retrieve the target action or generate a description that matches the input action;< / cls> < / ret> Result evaluation: Through multi-task evaluation of indicators such as the accuracy of action description generation and the accuracy of action-text retrieval, the performance of the FLARE framework in the action understanding task is verified.

Images