A small sample behavior recognition method of a visual language pre-training large model
By fine-tuning a visual language pre-trained model on a base class dataset and utilizing meta-learning and Hardman product to fuse similarity, the problem of knowledge transfer in small-sample behavior recognition using large visual language models is solved, achieving efficient model adaptation and generalization.
Patent Information
- Application Number
- CN202410153249.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2044-02-02
AI Technical Summary
Existing technologies cannot efficiently transfer prior knowledge from large visual language models to small-sample behavior recognition tasks, and existing methods suffer from high memory consumption and wasted computational resources.
We employ a meta-learning approach to fine-tune a visual language pre-trained model on a base class dataset. By constructing support sets and query sets, we utilize the Hardman product to fuse visual and semantic similarity for few-sample behavior recognition.
It effectively solves the overfitting problem, retains the prior knowledge of the original model, reduces memory consumption and computational resources, and improves the model's generalization ability in small sample tasks.
Smart Images
Figure CN118070878B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The embodiment of the present disclosure relates to the technical field of machine learning, in particular to a small sample behavior recognition method of a visual language pre-training large model. BACKGROUND
[0002] In recent years, deep learning has gradually entered the era of large models, and pre-training large models such as ChatGPT and DaLL-E have actually affected people's daily life. The foundation supporting these large models is large-scale artificial collection of training data. Without sufficient labeled training data, there is no large model that performs surprisingly. However, in practice, it is not easy to collect and label data, which requires a lot of manpower and resources. Moreover, for some special tasks such as the identification of endangered protected animals, it is difficult to collect a large amount of relevant task data for training. Therefore, when the available training samples of the target task are relatively scarce (one or a few), how to obtain a neural network model that performs well on the target task is a very important problem. Small sample behavior recognition is a task of identifying and classifying the behavior of a person in a video in a sample-scarce scenario.
[0003] Previous work on small sample behavior recognition mostly focuses on how to align the temporal information between videos. With the outstanding performance of the visual language pre-training large model CLIP in multiple downstream task scenarios, more and more researchers have begun to focus on how to better transfer the prior knowledge contained in it to the small sample behavior recognition task. CLIP-FSAR additionally introduces a semantic-guided prototype modulation module to enhance the temporal expression ability of video features. The entire CLIP model is used as a feature extractor and is completely fine-tuned on the small sample behavior recognition dataset. MA-CLIP proposes to introduce an additional adapter between each layer of CLIP. Only the parameters of the adapter need to be updated, and all parameters of the original CLIP model are frozen. However, the full fine-tuning strategy requires a large amount of video memory and is expensive, and it will also cause "catastrophic forgetting", which will destroy the prior knowledge of the original model. Introducing additional adapter parameters will also increase the computational cost of the model. The existing methods cannot efficiently transfer the prior knowledge of the visual language large model to the small sample behavior recognition task. SUMMARY
[0004] In order to avoid the shortcomings of the prior art, the present application provides a small sample behavior recognition method of a visual language pre-training large model, which solves the problem that the prior art cannot efficiently transfer the prior knowledge of the visual language large model to the small sample behavior recognition task.
[0005] According to the embodiment of the present disclosure, a small sample behavior recognition method of a visual language pre-training large model is provided, which comprises:
[0006] constructing a visual language pre-training model;
[0007] selecting behavior recognition data of N categories in a base class dataset, and selecting K+C video samples in the behavior recognition data of each category; wherein the first K video samples form a first support set, and the last C video samples form a first query set;
[0008] respectively inputting the first support set and the first query set into an image feature extractor in the visual language pre-training model to obtain first support set video features and first query set video features, and respectively performing average pooling on the first support set video features and the first query set video features;
[0009] performing intra-class average pooling on the first support set video features after average pooling to obtain first representative class prototype features of N categories;
[0010] respectively fitting the text labels of the first support set into a prompt template to obtain a first text label prompt set, and inputting the first text label prompt set into a frozen text feature extractor in the visual language pre-training model to obtain first semantic features of N categories;
[0011] calculating a first visual similarity according to the first query set video features and the first representative class prototype features;
[0012] calculating a first semantic similarity according to the first query set video features and the first semantic features;
[0013] fusing the first visual similarity and the first semantic similarity by using a Hadamard product to obtain a prediction result of the first query set video features.
[0014] Further, the step of fusing the visual similarity and the semantic similarity by using a Hadamard product to obtain the prediction result of the first query set video features further comprises:
[0015] calculating a classification loss according to the prediction result, calculating a gradient of the classification loss by back propagation according to the classification loss, and updating the classification loss according to the gradient of the classification loss;
[0016] determining whether the number of training rounds reaches a preset number, and if so, ending the training;
[0017] Otherwise, continue to select behavior recognition data of N categories in the base class dataset, and select K+C video samples in the behavior recognition data of each category.
[0018] Further, the method further comprises:
[0019] inputting the data to be recognized into the frozen trained visual language pre-training model to obtain a recognition result of the data to be recognized.
[0020] Further, in the step of inputting the data to be recognized into the frozen trained visual language pre-training model to obtain a recognition result of the data to be recognized, the step comprises:
[0021] selecting N categories in the data to be recognized, and selecting K+C samples in the data to be recognized of each category; wherein the first K samples form a second support set, and the last C samples form a second query set;
[0022] inputting the second support set and the second query set into the frozen image feature extractor of the trained visual language pre-training model respectively to obtain second support set video features and second query set video features respectively, and performing average pooling on the second support set video features and the second query set video features respectively;
[0023] performing intra-class average pooling on the second support set video features after average pooling to obtain N second representative class prototype features;
[0024] suits the text labels of the second support set into a prompt template respectively to obtain a second text label prompt set, and inputs the second text label prompt set into the frozen text feature extractor to obtain N second semantic features;
[0025] calculating a second visual similarity according to the second query set video features and the second representative class prototype features;
[0026] calculating a second semantic similarity according to the second query set video features and the second semantic features;
[0027] fusing the second visual similarity and the second semantic similarity by using Hadamard product to obtain a prediction result of the second query set video features.
[0028] Further, the expression of the first visual similarity is:
[0029]
[0030] In the formula, sim vis is the first visual similarity, F Q is the first support set video features after average pooling, is the first representative class prototype feature.
[0031] Further, the expression of the first semantic similarity is:
[0032] sim text = cosine < F Q , F τ >, sim text ∈ R NC×N (2)
[0033] wherein sim text is the first semantic similarity, F τ is the first semantic feature.
[0034] Further, the expression of the prediction result is:
[0035] sim fuse = sim vis ⊙ sim text (3)
[0036] wherein sim fuse is the prediction result.
[0037] The technical scheme provided by the embodiments of the present disclosure can include the following beneficial effects:
[0038] In the embodiments of the present disclosure, by using the small sample behavior recognition method of the visual language pre-training large model, on the one hand, the meta-learning method is used, that is, the small sample task form on the new class is imitated to complete the fine-tuning on the base class dataset. In the meta-learning stage, N-way K-shot meta-learning tasks are randomly extracted and constructed from the base class dataset each time: N classes of behavior recognition data are randomly selected, and then K+C video samples of each class of behavior recognition data are randomly extracted, wherein the first K video samples are used as a small sample support set, and the last C videos are used as a query set. The classification loss of all NxC query videos in each meta-learning task is used as the optimization target, and the visual language pre-training large model is gradually adapted to the small sample behavior recognition task domain through a certain number of randomly extracted meta-learning tasks. On the other hand, this method does not require a large amount of video memory consumption, and does not need to introduce additional computing resources, which well preserves the prior knowledge of the original model, and is simple and effective. This method only needs to fine-tune the visual language pre-training model on the base class dataset with sufficient data, rather than directly fine-tuning the visual language pre-training model on the small sample target dataset, effectively solving the overfitting problem caused by direct fine-tuning, and fully tapping the generalization potential of the visual language pre-training model itself. BRIEF DESCRIPTION OF DRAWINGS
[0039] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the disclosure. It is to be noted, as evident from the following description, that the drawings need not be to scale as compared to the actual implementation, and in particular, the dimensions of the elements in the figures can be arbitrarily expanded to make the drawing legible, as compared to the actual implementation.
[0040] Figure 1 A step diagram of the small sample behavior recognition method of the visual language pre-training large model in the exemplary embodiment of the present disclosure is shown;
[0041] Figure 2 A schematic diagram of the meta-learning stage of the small sample behavior recognition method of the visual language pre-training large model in the exemplary embodiment of the present disclosure is shown;
[0042] Figure 3 A schematic diagram of the meta-testing stage of the small sample behavior recognition method of the visual language pre-training large model in the exemplary embodiment of the present disclosure is shown;
[0043] Figure 4 A graph showing the results of the present method in the small sample standard test of the data sets Kinetics, HMDB51 and UCF101 in the exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0044] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.
[0045] In addition, the accompanying drawings are included to provide a thorough understanding of embodiments of the present disclosure and are not intended to be in any way limiting. The same reference numerals in different drawings identify the same or similar elements.
[0046] In the present example implementation, a small sample behavior recognition method of a visual language pre-training large model is first provided. Referring to FIG. 1, the small sample behavior recognition method of the visual language pre-training large model can include steps S101-S108. Figure 1
[0047] Step S101: Construct a visual language pre-training model;
[0048] Step S102: selecting N categories of behavior recognition data in a base class dataset, and selecting K+C video samples in the behavior recognition data of each category; wherein the first K video samples form a first support set, and the last C video samples form a first query set;
[0049] Step S103: inputting the first support set and the first query set into an image feature extractor in the visual language pre-training model respectively to obtain first support set video features and first query set video features, and performing average pooling on the first support set video features and the first query set video features respectively;
[0050] Step S104: performing intra-class average pooling on the first support set video features after average pooling to obtain first representative class prototype features of N categories;
[0051] Step S105: fitting the text labels of the first support set into a prompt template respectively to obtain a first text label prompt set, and inputting the first text label prompt set into a frozen text feature extractor in the visual language pre-training model to obtain first semantic features of N categories;
[0052] Step S106: calculating a first visual similarity according to the first query set video features and the first representative class prototype features;
[0053] Step S107: calculating a first semantic similarity according to the first query set video features and the first semantic features;
[0054] Step S108: fusing the first visual similarity and the first semantic similarity by using a Hadamard product to obtain a prediction result of the first query set video features.
[0055] The method for few-shot behavior recognition using the pre-trained large-scale visual language model described above employs a meta-learning approach. This involves mimicking the form of few-shot tasks on new categories to fine-tune the model on the base class dataset. In the meta-learning phase, an N-way K-shot meta-learning task is constructed by randomly selecting data from the base class dataset: N categories of behavior recognition data are randomly selected, and then K+C video samples are randomly extracted from each category. The first K video samples serve as the few-shot support set, and the last C videos form the query set. The classification loss of all N×C query videos in each meta-learning task is used as the optimization objective. Through a certain number of randomly selected meta-learning tasks, the pre-trained large-scale visual language model gradually adapts to the few-shot behavior recognition task domain. Furthermore, this method does not require significant GPU memory consumption or introduce additional computational resources, effectively preserving the prior knowledge of the original model, and is simple and effective. This method only requires fine-tuning the visual language pre-trained model on a sufficiently large base class dataset, rather than directly fine-tuning the visual language pre-trained model on a small target dataset. This effectively solves the overfitting problem caused by direct fine-tuning and fully taps the generalization potential of the visual language pre-trained model itself.
[0056] Below, we will refer to Figures 1 to 4 The steps of the small-sample behavior recognition method of the above-described visual language pre-trained large model in this example embodiment will be described in more detail.
[0057] In step S101, a visual language pre-trained model is constructed.
[0058] Specifically, for the large visual language pre-trained model CLIP, only the parameters of its last neural network layer are set to be updatable, while the parameters of all other layers are frozen.
[0059] The formulaic description is as follows:
[0060] Let the CLIP image feature extractor be F(Θ)∶R. B×C×H×W →R B×d This can convert a batch of images into corresponding high-dimensional image features, where Θ represents all the learnable parameters of the model, and Θ = [θ1, θ2, ..., θ]. M ], θ i This represents the parameters of the i-th layer, and M represents the total number of layers in the CLIP model.
[0061] For methods that fine-tune the entire model, the feature extractor (i.e., the image feature extractor) is a function F(Θ) of all parameters in the entire model. For methods that only fine-tune the additional adapter parameters, after inserting additional adapter parameters between each layer, Θ becomes... However, since only the adapter parameters are involved in the update, the original model parameters remain unchanged, and the feature extractor is only a function of the adapter parameters introduced additionally For part of the fine-tuning methods provided in the present application, the feature extractor is made to be a function F(θ M ) only related to the last layer model parameters. With the increasing number of layers of the neural network, the encoded features are more abstract. The strategy of updating only the last layer parameters only requires the model to adjust the encoding method of high-level features, has less impact on the original model features, and is sufficient to achieve efficient transfer on small sample datasets with few samples.
[0062] In step S102, behavior recognition data of N categories are selected in the base class dataset, and K+C video samples are selected in the behavior recognition data of each category; wherein the first K video samples form a first support set, and the last C video samples form a first query set.
[0063] Specifically, K+C video samples are randomly selected in each category, and the first K samples form a support set Wherein x i and y i represent the video sample and the corresponding category text label respectively, and the last C samples form a query set
[0064] In step S103, the first support set and the first query set are respectively input into the image feature extractor in the visual language pre-training model, and the first support set video feature and the first query set video feature are respectively obtained, and the first support set video feature and the first query set video feature are respectively averaged.
[0065] Specifically, the video samples of the first support set and the first query set are input into the partially fine-tuned feature extractor F(θ M ), and the support set video feature F S ∈R NK×T×d (i.e. the first support set video feature) and the query set video feature F Q ∈R NC×T×d (i.e. the first query set video feature) are obtained respectively; all video features are averaged along the time dimension T to obtain the averaged support set video feature F S ∈R NK×d and the averaged query set video feature F Q ∈R NC×d .
[0066] In step S104, the first support set video feature after the average pooling is in-class average pooled to obtain N first representative class prototype features.
[0067] Specifically, the K sample feature averages of each category are supported to obtain the first representative class prototype features of N categories
[0068] In step S105, the text labels of the categories of the first support set are respectively fitted into the prompt template to obtain the first text label prompt set, and the first text label prompt set is input into the frozen text feature extractor of the visual language pre-training model to obtain the first semantic features of N categories.
[0069] Specifically, the text labels of the N support set categories are respectively fitted into the fixed prompt template "a photo about {action category}", and then input into the frozen CLIP text encoder to obtain the first semantic features F of N categories τ ∈R N×d .
[0070] In step S106, the first visual similarity is calculated according to the first query set video features and the first representative class prototype features.
[0071] Specifically, the similarity between the query video and each category representative class prototype is calculated as the visual similarity:
[0072]
[0073] In step S107, the first semantic similarity is calculated according to the first query set video features and the first semantic features.
[0074] Specifically, the similarity between the query video and each category semantic feature is calculated as the semantic similarity:
[0075] sim text =cosine<F Q ,F τ >,sim text ∈R NC×N (2)
[0076] In step S108, the first visual similarity and the first semantic similarity are fused by using the Hadamard product to obtain the prediction result of the first query set video features.
[0077] Specifically, the visual and semantic similarities are fused by the Hadamard product as the N-class prediction result of the query video:
[0078] sim fuse =sim vis ⊙sim text (3)
[0079] Further, the method further comprises: calculating a classification loss according to the prediction result, calculating a gradient of the classification loss according to the classification loss, and updating the classification loss according to the gradient of the classification loss; judging whether the number of rounds of training reaches a preset number, and if yes, ending the training; otherwise, continuing to select N classes of behavior recognition data in the base class data set, and selecting K+C video samples in each class of behavior recognition data.
[0080] Further, the method further comprises: inputting the to-be-recognized data into the frozen trained visual language pre-training model to obtain a recognition result of the to-be-recognized data.
[0081] Specifically, for a new sample with a small amount of behavior categories, the entire model is frozen and no longer participates in updating, and directly serves as a feature extractor of a new category. For a new class query sample to be classified, only a small amount of labeled samples of the new category are used as a support set, and the steps S103 to S108 are used to complete the behavior recognition task of the new category.
[0082] More specifically, only a small amount of labeled samples of the new category are used as a second support set, and the new class query sample to be classified is used as a second query set.
[0083] The video samples of the second support set and the second query set are both input into the frozen feature extractor F to obtain second support set video features and second query video features, respectively.
[0084] The video features are averaged along the time dimension T to obtain averaged second support set video features and averaged second query video features.
[0085] The K sample features of each category in the second support set are averaged to obtain second representative class prototype features of N categories.
[0086] The text labels of the N second support set categories are respectively fitted into a fixed prompt template “a photo about {action category}”, and then input into the frozen CLIP text encoder to obtain second semantic features of N categories.
[0087] The similarity between the query video and each category representative class prototype is calculated as a visual similarity.
[0088] The similarity between the query video and each category semantic feature is calculated as a semantic similarity.
[0089] The visual and semantic similarities are fused as an N-class prediction result of the query video by using a Hadamard product.
[0090] In one specific embodiment, the results obtained by using three international public data sets Kinetics, HMDB51 and UCF101 for small sample standard testing are as followsFigure 4 The 5-way 5-shot test accuracy on the three datasets reaches 96.9%, 88.7% and 99.3% respectively.
[0091] The application provides an efficient parameter updating strategy for adapting visual language pre-training large models to small sample behavior recognition tasks, which does not require large memory consumption compared to the full fine-tuning method, does not introduce additional computing resources compared to the fine-tuning of additional adapter parameters, well preserves the prior knowledge of the original model, and is simple and effective.
[0092] The application proposes fine-tuning the pre-training model only on the base dataset with sufficient data, rather than directly fine-tuning the pre-training model on the small sample target dataset, effectively solving the overfitting problem caused by direct fine-tuning, and fully tapping the generalization potential of the pre-training large model itself.
[0093] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.
[0094] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present specification.
[0095] Other embodiments of the present disclosure will be apparent to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles thereof and include the general knowledge or custom of the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A small sample behavior recognition method of a visual language pre-training large model, characterized in that, The method comprises: constructing a visual language pre-training model; wherein the parameters of the last layer of neural network of the visual language pre-training model are set as updateable parameters, and the parameters of other layers of neural network are set as frozen parameters; selecting behavior recognition data of N categories in a base class data set, and selecting K+C video samples in the behavior recognition data of each category; wherein the first K video samples form a first support set, and the last C video samples form a first query set; inputting the first support set and the first query set into an image feature extractor in the visual language pre-training model respectively to obtain first support set video features and first query set video features, and performing average pooling on the first support set video features and the first query set video features respectively; performing intra-class average pooling on the first support set video features after average pooling to obtain first representative class prototype features of N categories; folding the text labels of the first support set into a prompt template respectively to obtain a first text label prompt set, and inputting the first text label prompt set into a frozen text feature extractor in the visual language pre-training model to obtain first semantic features of N categories; calculating first visual similarity according to the first query set video features and the first representative class prototype features; calculating first semantic similarity according to the first query set video features and the first semantic features; fusing the first visual similarity and the first semantic similarity by using Hadamard product to obtain a prediction result of the first query set video features.
2. The small sample behavior recognition method of the visual language pre-training large model according to claim 1, characterized in that, After the step of fusing the visual similarity and the semantic similarity by using Hadamard product to obtain the prediction result of the first query set video features, the method further comprises: calculating a classification loss according to the prediction result, and then calculating the gradient of the classification loss by back propagation according to the classification loss, and updating the classification loss according to the gradient of the classification loss; determining whether the number of training rounds reaches a preset number, and if so, ending the training; otherwise, continue to select behavior recognition data of N categories in the base class data set, and select K+C video samples in the behavior recognition data of each category.
3. The small sample behavior recognition method of the visual language pre-training large model according to claim 2, characterized in that, The method further comprises: inputting the to-be-identified data into the frozen trained visual language pre-training model to obtain an identification result of the to-be-identified data.
4. The small sample behavior recognition method of the visual language pre-training large model according to claim 3, characterized in that, In the step of inputting the to-be-identified data into the frozen trained visual language pre-training model to obtain an identification result of the to-be-identified data, the method comprises: setting the parameters of the neural network of all layers of the visual language pre-training model as the frozen parameters; selecting N categories in the to-be-identified data, and selecting K+C samples in the to-be-identified data of each category; wherein the first K samples form a second support set, and the last C samples form a second query set; inputting the second support set and the second query set into the image feature extractor of the visual language pre-training model which is trained and frozen respectively to obtain second support set video features and second query set video features, and performing average pooling on the second support set video features and the second query set video features respectively; performing intra-class average pooling on the second support set video features after average pooling to obtain second representative class prototype features of N categories; sleeving text labels of the second support set into a prompt template respectively to obtain a second text label prompt set, and inputting the second text label prompt set into the text feature extractor which is frozen to obtain second semantic features of N categories; calculating a second visual similarity according to the second query set video features and the second representative class prototype features; calculating a second semantic similarity according to the second query set video features and the second semantic features; fusing the second visual similarity and the second semantic similarity by using Hadamard product to obtain a prediction result of the second query set video features.
5. The small sample behavior recognition method of the visual language pre-training large model according to claim 1, characterized in that, an expression of the first visual similarity is: （1） In the formula, is the first visual similarity, is the first support set video feature after average pooling, is the first representative class prototype feature.
6. The small sample behavior recognition method of the visual language pre-training large model according to claim 5, characterized in that, an expression of the first semantic similarity is: （2） In the formula, is the first semantic similarity, is the first semantic feature.
7. The small sample behavior recognition method of the visual language pre-training large model according to claim 6, characterized in that, an expression of the prediction result is: （3） In the formula, is the predicted result.