A test-time adaptive small-sample incremental action recognition method and device

Abstract

This invention discloses a few-sample incremental action recognition method and apparatus with test-time adaptive capability, relating to the field of video action recognition technology. The invention is based on a two-stage structure of "base class learning—new class adaptation": in the base class stage, the model learns general spatiotemporal features through sufficient samples and introduces a multi-head self-attention mechanism to improve generalization ability; in the new class stage, a hierarchical test-time adaptive mechanism is used to achieve self-updating of the model during inference. Using this invention, not only is the model's responsiveness to new class samples improved, but catastrophic forgetting and bias accumulation caused by noisy samples are also effectively suppressed, thereby achieving a stable and efficient incremental learning process, and thus improving the accuracy and efficiency of few-sample action recognition.

Description

Technical Field

[0001] This invention relates to the field of video action recognition technology, and in particular to a small-sample incremental action recognition method and apparatus with adaptive test time. Background Technology

[0002] Incremental Few-Shot Action Recognition (IFSAR) aims to enable models to continuously learn new classes while maintaining their ability to recognize old classes under limited sample conditions. It is a highly challenging research direction in dynamic artificial intelligence systems. Because new class samples are extremely scarce during the incremental phase, the extremely limited training samples are insufficient to fully represent the complete information of a class, leading to inaccuracies when using prototypes to represent classes. Specifically, this strategy inevitably results in models exhibiting better generalization and robustness in scenarios with sufficient samples, while in incremental scenarios with scarce samples, they are susceptible to noise and interference, leading to lower accuracy in few-shot action recognition. Summary of the Invention

[0003] To address the low accuracy of small-sample action recognition in existing technologies, this invention provides a small-sample incremental action recognition method and apparatus with adaptive testing time. The technical solution is as follows:

[0004] On the one hand, a test-time adaptive few-sample incremental action recognition method is provided, which is implemented by a test-time adaptive few-sample incremental action recognition device, and the method includes:

[0005] S01. Obtain a training set, a test set, and a few-shot action recognition model to be trained. The training set includes base class training samples and new class training samples. The few-shot action recognition model includes a video feature extraction backbone, a classifier, and a multi-head self-attention module.

[0006] S02. Based on the base class training samples, train the video feature extraction backbone and classifier to obtain the trained video feature extraction backbone and first classifier.

[0007] S03. Construct pseudo-incremental samples based on the base class training samples, and train the multi-head self-attention module based on the pseudo-incremental samples to obtain the initially trained multi-head self-attention module.

[0008] S04. Based on the video features of all base class training samples, determine the prototype features of each action category, and use the prototype features as the weights of the new classifier to obtain the second classifier.

[0009] S05. Extract the backbone based on the new class training samples and the trained video features, obtain the video features of the new class training samples, and then obtain the prototype features of each new class. Add the prototype features of the new class to the second classifier to obtain the third classifier.

[0010] S06. Classify the prototype features in the third classifier and determine the prototype category of each prototype feature; based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier, determine the pseudo-label and confidence of each test sample; based on the confidence of each test sample and the prototype category of the prototype features of the pseudo-label, classify the test samples and determine the sample category of each test sample.

[0011] S07. Update the prototype features corresponding to the pseudo-labels of the test samples based on the prototype category of the prototype features, the sample category of the test samples, the prototype features of the pseudo-labels of the test samples, and the video features of the test samples.

[0012] S08. Update the pre-trained multi-head self-attention module based on the test samples of the preset sample categories, cross-entropy loss, and entropy regularization loss.

[0013] S09. Repeatedly execute S05-S08 to obtain the trained multi-head self-attention module, and then obtain the trained few-sample action recognition model.

[0014] S10. Input the video to be classified into the trained few-sample action recognition model to obtain the action recognition result of the video to be classified.

[0015] On the other hand, a test-time adaptive few-sample incremental action recognition device is provided, which is applied to the test-time adaptive few-sample incremental action recognition method. The device includes:

[0016] The acquisition unit is used to acquire a training set, a test set, and a few-shot action recognition model to be trained. The training set includes base class training samples and new class training samples. The few-shot action recognition model includes a video feature extraction backbone, a classifier, and a multi-head self-attention module.

[0017] The first training unit is used to train the video feature extraction backbone and classifier based on the base class training samples, so as to obtain the trained video feature extraction backbone and the first classifier.

[0018] The second training unit is used to construct pseudo-incremental samples based on the base class training samples, and to train the multi-head self-attention module based on the pseudo-incremental samples to obtain the initially trained multi-head self-attention module.

[0019] The first update unit is used to determine the prototype features of each action category based on the video features of all base class training samples, and use the prototype features as new classifier weights to obtain the second classifier.

[0020] The second update unit is used to extract the backbone based on the new class training samples and the trained video features, obtain the video features of the new class training samples, and then obtain the prototype features of each new class. The prototype features of the new class are added to the second classifier to obtain the third classifier.

[0021] The classification unit is used to classify the prototype features in the third classifier and determine the prototype category of each prototype feature. Based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier, it determines the pseudo-label and confidence of each test sample. Based on the confidence of each test sample and the prototype category of the prototype features of the pseudo-label, it classifies the test samples and determines the sample category of each test sample.

[0022] The third update unit is used to update the prototype features corresponding to the pseudo-labels of the test samples based on the prototype category of the prototype features, the sample category of the test samples, the prototype features of the pseudo-labels of the test samples, and the video features of the test samples.

[0023] The third training unit is used to update the initially trained multi-head self-attention module based on the test samples of the preset sample categories, cross-entropy loss, and entropy regularization loss.

[0024] The repeated execution unit is used to iteratively execute the second update unit-third training unit multiple times to obtain the trained multi-head self-attention module, and then obtain the trained few-shot action recognition model;

[0025] The action recognition unit is used to input the video to be classified into a trained few-sample action recognition model to obtain the action recognition result of the video to be classified.

[0026] On the other hand, a test time-adaptive small sample incremental action recognition device is provided, the test time-adaptive small sample incremental action recognition device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for test time-adaptive small sample incremental action recognition.

[0027] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described small sample incremental action recognition methods with adaptive test time.

[0028] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0029] To address the issues of scarce new class samples, feature bias, and classifier imbalance in incremental action recognition with few samples, a multi-level adaptive test time framework is proposed. This method centers on a two-stage structure of "base class learning—new class adaptation": in the base class stage, the model learns general spatiotemporal features through sufficient samples and introduces a multi-head self-attention mechanism to improve generalization ability; in the new class stage, the hierarchical adaptive test time mechanism enables the model to self-update during inference. Overall, incremental action recognition with few samples is achieved by jointly updating the prototype and network parameters. To update test samples more accurately, this embodiment of the invention designs a multi-level update module, enabling the network to selectively update key samples while eliminating interference samples that may mislead the model.

[0030] This not only improves the model's ability to respond to new types of samples, but also effectively suppresses catastrophic forgetting and bias accumulation caused by noisy samples, thereby achieving a stable and efficient incremental learning process, and thus improving the accuracy and efficiency of small sample action recognition. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of a small-sample incremental action recognition method with adaptive test time provided by an embodiment of the present invention;

[0033] Figure 2 This is a block diagram of a small-sample incremental action recognition device with adaptive test time provided in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the structure of a small sample incremental action recognition device with adaptive test time provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0036] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0037] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0038] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0039] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0040] This invention provides a test-time adaptive small-sample incremental action recognition method, which can be implemented by a test-time adaptive small-sample incremental action recognition device, which can be a terminal or a server. Figure 1 The flowchart shown is for a small-sample incremental action recognition method with adaptive test time. The processing flow of this method may include the following steps:

[0041] S01. Obtain the training set, test set, and small sample action recognition model to be trained.

[0042] The training set includes base class training samples and new class training samples. The few-shot action recognition model includes a video feature extraction backbone, a classifier, and a multi-head self-attention module.

[0043] In one feasible implementation, it is assumed that the small sample incremental action recognition task has a total of Base class sessions and A new type of session, represented as ,in For the base class Session, For the new type of session, the task's tag set is represented as follows: ,satisfy The training set and the test set are represented as follows: and For the conversation , Its training set is ,in The sample refers to a set of video frames. This indicates the label corresponding to the sample, specifically the action category of the video, and The test set is ,Label Unknowable, and .

[0044] This invention primarily consists of two phases: a base class phase and a new class phase. The base class phase aims to learn general spatiotemporal feature representations using large-scale known class data and establish a stable prototype space and classification structure. It focuses on achieving adaptive model updates within the testing time, enabling the model to quickly absorb semantic features of new classes while maintaining the stability of old class knowledge. Structurally, the model comprises three key components: a video feature extraction backbone network F, responsible for extracting spatiotemporal feature representations from action sequences; a classifier G, which replaces traditional linear weights with class center prototypes for easy and rapid expansion; and a multi-head self-attention module H, which leverages meta-learning methods to enhance the model's generalization ability.

[0045] In the following steps:

[0046] S02-S04 constitute the base class stage. The goal of the base class stage is to learn a general spatiotemporal action feature representation using sufficient base class data, and to establish a stable representation space and classification structure for subsequent incremental learning stages. This stage includes three key steps: backbone network and classifier training, self-attention module training, and classifier prototype replacement. These three steps are performed sequentially to gradually build a unified recognition framework consisting of the backbone network F, the classifier G, and the self-attention module H.

[0047] S05-S09 represent the new class stage. In the small-sample incremental action recognition task, the goal of the new class stage is to enable the model to quickly adapt to the semantic features of new class samples while maintaining the stability of old class knowledge. To achieve this goal, this embodiment of the invention adopts a test-time adaptive mechanism. First, a multi-level update module is designed. Through multi-scale screening and confidence control mechanisms at the sample and prototype levels, low-quality or interfering samples are effectively filtered, thereby improving the stability of the update process and the efficiency of sample utilization. This module establishes a refined update foundation at the sample level, enabling the model to accurately identify key samples and perform targeted feature adjustments in complex scenes. Based on this, this embodiment of the invention further performs joint optimization and update of prototype and network parameters, realizing the dynamic migration and self-evolution of the model in each new class session. This mechanism utilizes the sample distribution information in the test phase to simultaneously optimize high-level semantic parameters and prototype representations, enabling the model to continuously absorb new class features while maintaining old class cognition. Finally, in each incremental session, the model forms an adaptive closed loop driven by multi-level feature updates and global parameter optimization, constructing an optimization system that is both dynamically adaptable and stable, thereby achieving a balanced fusion and efficient transfer of old and new class knowledge.

[0048] S02. Based on the base class training samples, train the video feature extraction backbone and classifier to obtain the trained video feature extraction backbone and first classifier.

[0049] In one feasible implementation, the model is first trained extensively using large-scale data with known categories to obtain feature representations with good generalization ability. Let the input sample be x, and TSM-ResNet50 be used as the backbone for video feature extraction, denoted as... Its characteristics can be obtained through forward propagation as follows:

[0050]

[0051] Subsequently, embodiments of the present invention introduce a classifier based on cosine similarity. To calculate sample features The similarity between the data and the weight vectors of each category is denoted as the classification score. Based on the true label y of the sample, supervised training is performed using cross-entropy loss:

[0052]

[0053] in The temperature coefficient is minimized. The network can be optimized simultaneously. and This makes the features extracted from the backbone more separable in the cosine space, thus obtaining a more discriminative semantic representation. However, since direct training of classifier weights in small-sample incremental tasks may lead to incompatibility with subsequent new classes, this embodiment of the invention will replace the classifier weights with class feature prototypes in subsequent steps to enhance the scalability and stability of the model.

[0054] S03. Construct pseudo-incremental samples based on the base class training samples, and train the multi-head self-attention module based on the pseudo-incremental samples to obtain the initially trained multi-head self-attention module.

[0055] In one feasible implementation, after the backbone network has been trained and has acquired preliminary representation capabilities, this embodiment of the invention further introduces a multi-head self-attention module H to enhance the model's generalization ability through a simulated meta-learning strategy.

[0056] Optionally, the specific operation of S03 may include the following steps S031-S036:

[0057] S031. Randomly select multiple action categories from the base class training samples, and randomly select one sample from each action category to form a pseudo-base class support set. Perform random angle rotation on the base class training samples in the pseudo-base class support set to construct a pseudo-incremental support set.

[0058] S032. Select another sample from the selected multiple action categories to form a pseudo base class query set, and apply the same rotation transformation as the pseudo incremental support set to the base class training samples in the pseudo base class query set to obtain the pseudo incremental query set.

[0059] S033. Input the query samples from the pseudo-base class query set and pseudo-increment query set into the trained video feature extraction backbone to obtain the video features of the query samples.

[0060] S034. Input the pseudo-base class support samples from the pseudo-base class support set into the trained video feature extraction backbone to obtain the video features of the pseudo-base class support samples.

[0061] S035. Input the pseudo-incremental support samples from the pseudo-incremental support set into the trained video feature extraction backbone to obtain the video features of the pseudo-incremental support samples.

[0062] S036. Based on the video features of the query sample, the video features of the pseudo-base class support sample, the video features of the pseudo-increment support sample, and the cross-entropy supervised loss function, the multi-head self-attention module is trained to obtain the initially trained multi-head self-attention module.

[0063] In one feasible implementation, from the base class tag set Random selection For each of the three classes, one sample is randomly selected to form the pseudo-base class support set. Then, the support set samples are rotated at random angles to construct a pseudo-incremental support set. The number of samples in the pseudo-increment support set is the same as the number of samples in the pseudo-base class support set to simulate the change in class distribution during the increment phase. Simultaneously, another sample is selected from the same class to form the pseudo-base class query set. And apply the same rotation transformation to it to obtain the pseudo-incremental query set. In this way, each sample in the query set can find a corresponding category in the support set. For each query sample in the pseudo-base class query set and the pseudo-incremental query set... Combine its video features with support set features to form And input multi-head attention module ,get ,make ,calculate and Cosine similarity of various types CrossEntropy loss:

[0064]

[0065] in For temperature coefficient, The number of categories to select can be set to 10. During training, In update At the same time, it will also update with a small learning rate. . The main update provides enhanced discriminative power, while the backbone... Fine-tuning is performed with a small learning rate to maintain feature consistency. This process enables the network to maintain stable representational capabilities under different perspectives and deformations, laying the foundation for adaptive incremental stages.

[0066] S04. Based on the video features of all base class training samples, determine the prototype features of each action category, and use the prototype features as the weights of the new classifier to obtain the second classifier.

[0067] In one feasible implementation, after completing self-attention training, to improve the system's compatibility in the incremental phase, this embodiment of the invention will classify the classifier. The weights are replaced with the class prototypes calculated based on the training samples.

[0068] Optionally, the specific operations of S04 may include:

[0069] The video features of all base class training samples are classified according to action categories. The feature values ​​of all video features corresponding to each action category are added together according to the dimension and then averaged to obtain the prototype features of each action category. The prototype features are used as the weights of the new classifier to obtain the second classifier.

[0070] In one feasible implementation, the base class training set Extract video features from all samples The prototype feature set for each category is obtained by averaging the features by category. The specific method for averaging by category is to add the feature values ​​of multiple video features according to their dimensions, and then average them according to their dimensions. For example, if there are 10 video features, and each video feature has 1024 dimensions, then add the 10 feature values ​​of the first dimension of the 10 video features, and then divide by 10 to get the feature value of the first dimension of the prototype feature. This process is repeated to calculate the feature value of each dimension of the prototype feature, thus obtaining the prototype feature.

[0071] These prototype features reflect the feature distribution of the class centers. They are then used as the new classifier weights. The parameters in the parameter space are used to achieve the transformation from a parametric classifier to a non-parametric recognition based on the metric space. This replacement mechanism not only preserves the discriminativeness of the backbone feature space, but also provides a natural interface for subsequent expansion of new classes and prototype updates.

[0072] S05. Extract the backbone based on the new class training samples and the trained video features, obtain the video features of the new class training samples, and then obtain the prototype features of each new class. Add the prototype features of the new class to the second classifier to obtain the third classifier.

[0073] In one feasible implementation, the process begins with the injection of new class knowledge, and embodiments of the present invention utilize new class sessions. A small number of labeled samples were provided. Video features are extracted through the backbone network F, and the prototype feature representation of each new class is calculated. These new class prototypes are directly added to the second classifier, together with the historical prototypes, to form an expanded classification space. This process gives the model the initial ability to recognize new classes. This step lays the foundation for subsequent adaptive testing time.

[0074] S06. Classify the prototype features in the third classifier and determine the prototype category of each prototype feature. Based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier, determine the pseudo-label and confidence level of each test sample. Based on the confidence level and the prototype category of the prototype features of the pseudo-label, classify the test samples and determine the sample category of each test sample.

[0075] In one feasible implementation, to achieve accurate processing of test samples, this embodiment of the invention proposes a multi-level discrimination module. This module first discriminates the prototype type. Then, the module discriminates the test samples.

[0076] Optionally, the specific operation of S06 may include the following steps S061-S066:

[0077] S061. For any prototype feature of an action category in the third classifier, calculate the cosine similarity between the prototype feature of the action category and the prototype features of each other action category to obtain multiple similarity values. Compare each of the multiple similarity values ​​with a preset similarity threshold to determine the number of similarity values ​​greater than the preset similarity threshold. Based on the number and the preset prototype classification rules, determine the prototype category corresponding to the prototype feature of the action category.

[0078] In one feasible implementation, prototypes can be categorized into three types: misclassifiable prototypes, commonly misclassifiable prototypes, and misclassifiable prototypes. If a prototype has very low cosine similarity to prototypes of other categories, then the category corresponding to that prototype is less likely to be misclassified; therefore, this prototype is called a misclassifiable prototype. In practice, by calculating the cosine similarity among all prototypes and analyzing their distribution in the feature space, prototype classification is performed based on the number of prototypes whose cosine similarity to this prototype is greater than several preset thresholds.

[0079] Optionally, the preset prototype classification rule can be: if the number of similarity values ​​is greater than the preset similarity threshold and is greater than or equal to the first threshold, then it is classified as a mistake-prone prototype; if the number is less than the first threshold but greater than or equal to the second threshold, then it is classified as a normal mistake-prone prototype; if the number is less than the second threshold, then it is classified as a non-mistake-prone prototype; wherein the first threshold is greater than the second threshold.

[0080] S062. Based on the video features of the test samples and the pre-trained multi-head self-attention module, obtain the attention features of the test samples.

[0081] S063. Based on the prototype features of each action category in the third classifier and the pre-trained multi-head self-attention module, obtain the attention features of the prototype features of each action category.

[0082] S064. Calculate the score for each test sample for each action category based on the cosine similarity between the attention features of the test sample and the prototype features of the attention features of the new training sample.

[0083] S065. The action category corresponding to the highest score of each test sample is determined as the pseudo-label of the test sample, and the highest score of each test sample is determined as the confidence level of each test sample.

[0084] S066. Based on the confidence level of each test sample, the prototype category of the pseudo-label's prototype features, and the preset sample classification rules, classify the test samples to obtain the sample category for each test sample. The sample categories include high-quality samples, medium-quality samples, and low-quality samples.

[0085] In one feasible implementation, the sample categories can be divided into high-quality samples, medium-quality samples, and low-quality samples.

[0086] For two different test samples, the test sample features are obtained through the test sample, the backbone network, the prototype features in classifier G, the multi-head self-attention network, and the prototype in classifier G. and the prototype features of very few new class training samples. ,Should The output of the pseudo-multi-head self-attention network. Calculate the cosine similarity between the sample feature and the prototype feature. And calculate the score for the sample for each class:

[0087]

[0088] The class with the highest score is the pseudo-label of the test sample. The confidence level of the corresponding score for that sample is denoted as . :

[0089]

[0090] This multi-level discrimination mechanism provides a precise basis for decision-making in subsequent differentiated update strategies.

[0091] Optionally, the core of the preset sample classification rule is as follows: after the model outputs two different pseudo-labels and two different confidence levels, the prototype type corresponding to the confidence level and the pseudo-label needs to be comprehensively considered. That is, test samples classified as ordinary error-prone prototype pseudo-labels need to have higher confidence levels to be classified as high-level samples compared to test samples classified as non-error-prone prototype pseudo-labels. High-level samples have greater weight in the subsequent optimization network compared to medium-level samples. In other words, non-error-prone prototypes need lower confidence levels to be classified as high-level samples; ordinary error-prone prototypes need medium confidence levels to be classified as high-level samples; and error-prone prototypes need higher confidence levels to be classified as high-level samples.

[0092] A feasible pre-defined sample classification rule could be:

[0093] When the pseudo-label of the test sample is a prototype that is not prone to error classification, it is classified as a high-level sample when its confidence level is greater than or equal to the first confidence threshold; it is classified as a medium-level sample when its confidence level is less than the first confidence threshold but greater than or equal to the second confidence threshold; and it is classified as a low-level sample when its confidence level is less than the second confidence threshold.

[0094] When the pseudo-label of a test sample is a common error-prone prototype, it is classified as a high-quality sample if its confidence level is greater than or equal to the third confidence threshold; it is classified as a medium-quality sample if its confidence level is less than the third confidence threshold but greater than or equal to the fourth confidence threshold; and it is classified as a low-quality sample if its confidence level is less than the fourth confidence threshold.

[0095] When the pseudo-label of a test sample is a misclassifiable prototype, it is classified as a high-level sample if its confidence level is greater than or equal to the fifth confidence threshold; it is classified as a medium-level sample if its confidence level is less than the fifth confidence threshold but greater than or equal to the sixth confidence threshold; and it is classified as a low-level sample if its confidence level is less than the sixth confidence threshold.

[0096] The first confidence threshold is greater than the second confidence threshold, the third confidence threshold is greater than the fourth confidence threshold, and the fifth confidence threshold is greater than the sixth confidence threshold; and the fifth confidence threshold > the third confidence threshold > the first confidence threshold, and the sixth confidence threshold > the fourth confidence threshold > the second confidence threshold.

[0097] S07. Based on the prototype category of the prototype features, the sample category of the test sample, the prototype features of the pseudo-label of the test sample, and the video features of the test sample, update the prototype features corresponding to the pseudo-label of the test sample.

[0098] In one feasible implementation, the core of this method lies in the joint optimization and updating of key components of the model during testing. This includes two aspects: First, performing prototype evolution based on sample type.

[0099] Optionally, the specific operation of S07 may include the following steps S071-S072:

[0100] S071. Determine the sample category weight of each test sample based on the number of sample categories corresponding to each test sample, and determine the pseudo-label prototype category weight of each test sample based on the number of prototype categories corresponding to the pseudo-labels of each test sample.

[0101] S072. For any test sample, the ratio of the pseudo-label prototype class weight to the sum of the pseudo-label prototype class weight and the sample class weight is determined as the prototype feature weight of the test sample's pseudo-label. The ratio of the sample class weight to the sum of the pseudo-label prototype class weight and the sample class weight is determined as the video feature weight of the test sample. The product of the prototype feature weight and the prototype feature is added to the product of the video feature weight and the video feature to obtain the updated prototype feature corresponding to the pseudo-label of the test sample. The sum of the pseudo-label prototype class weight and the sample class weight is used as the updated prototype feature weight of the pseudo-label to update the third classifier.

[0102] In one feasible implementation, appropriate update weights are dynamically assigned to each test sample based on its sample category and the prototype type of the prototype feature corresponding to its pseudo-label. And update the corresponding class prototype using the moving average formula:

[0103]

[0104]

[0105] in To test the prototype features of the pseudo-labels of the samples, Training samples were processed by backbone The obtained video features The weights are set to the pseudo-label prototypes for the test samples, with the initial value defaulting to the number of training samples for that category. The weights are updated for each test sample. The determination of the prototype means that a high-level sample corresponding to a non-mistakeable classification prototype will be given a high update weight to correct the prototype; while a low-level sample corresponding to a mistakeable classification prototype will have a very low update weight, thus avoiding misleading updates to the robust prototype.

[0106] S08. Update the initially trained multi-head self-attention module based on the test samples of the preset sample categories, cross-entropy loss, and entropy regularization loss.

[0107] In one feasible implementation, the second step of the joint optimization update involves performing fine-tuning of the self-attention network for robust optimization.

[0108] Optionally, the specific operations of S08 may include:

[0109] Test samples of the high-level sample category are selected. Cross-entropy loss is calculated based on the confidence of each high-level sample. Entropy regularization loss is calculated based on the score of each high-level sample for each action category. Composite loss function is calculated based on the sum of cross-entropy loss and entropy regularization loss. The multi-head self-attention module that has been initially trained is then trained based on composite loss function.

[0110] In one feasible implementation, in order to simultaneously optimize network parameters, embodiments of the present invention utilize high-quality samples and set a scaling factor. Calculate a composite loss function:

[0111]

[0112] in, This is the scaling factor, which can be set to 1 / 15.

[0113] This loss combines cross-entropy loss and entropy regularization loss, aiming to reinforce correct classification while encouraging the model to make more decisive predictions. By backpropagating this loss, embodiments of the invention fine-tune the parameters of the self-attention network H online. This joint update of the prototype and network parameters constitutes the core of the test-time adaptive method, enabling the model to continuously learn safely and efficiently from the test stream.

[0114] S09. Repeatedly execute S05-S08 to obtain a trained multi-head self-attention module, and then obtain a trained few-sample action recognition model.

[0115] In one feasible implementation, in each new class session In this embodiment of the invention, the method executes a complete closed-loop process to achieve adaptive learning of test time.

[0116] S10. Input the video to be classified into the trained few-sample action recognition model to obtain the action recognition result of the video to be classified.

[0117] The method based on embodiments of the present invention was evaluated through extensive experiments on three types of public datasets. Furthermore, comprehensive ablation experiments were conducted on each proposed module to verify its effectiveness. The experimental process and results are described below:

[0118] 1. Dataset

[0119] UCF-101: This dataset is a human behavior recognition dataset containing over 13,000 video clips from YouTube, totaling approximately 27 hours in length. The dataset covers 101 categories, with 51 categories set as base classes and the remaining 50 as new classes. In a 5-way 5-shot setup, these 50 new classes are divided into 10 incremental sessions—each session containing 5 categories, with each category having 5 training samples in the incremental session.

[0120] Kinetics-100: This dataset is constructed from 100 randomly selected categories from Kinetics-400, with 100 samples in each category. For the FSCIL task, the 100 categories are divided into 60 classes and 40 new classes. The 40 new classes are evenly distributed across 8 sessions, with each session containing 5 classes. The experiments also follow a 5-way 5-shot setup.

[0121] HMDB-5: This human behavior recognition dataset contains 51 categories and includes 7,000 segments extracted from multi-source videos. The 51 categories are divided into 31 base classes and 20 new classes. The 20 new classes are evenly distributed across 10 sessions, with each session containing 2 new classes.

[0122] 2. Baseline

[0123] LIMIT: This method is based on a meta-learning paradigm, generalizing the feature space of new tasks by sampling multi-stage pseudo-incremental tasks. Furthermore, LIMIT proposes a calibration module that balances old and new classes through contextualized query instances and reorders the output probabilities of all known classes.

[0124] FACT: This method compresses the embedding space of known classes by pre-allocating virtual prototypes, while reserving embedding capacity for new classes. Virtual samples are generated through instance mixing to predict the possible distribution characteristics of new classes.

[0125] NC-FSCIL: Inspired by the neural collapse phenomenon, this method avoids catastrophic forgetting by addressing the mismatch between features and classifiers. Specifically, NC-FSCIL employs a novel loss function that predefines the optimal feature-classifier alignment as a fixed optimization objective, thereby eliminating optimization conflicts between sessions.

[0126] ALICE: This method re-evaluates the current benchmark settings for the FSCIL task, improving it in two dimensions: introducing angle-penalized loss into the FSCIL framework and enhancing model generalization ability through data analysis. To ensure fairness in the comparison and avoid the time consumption of video data augmentation, the data augmentation module was removed in the reproduction.

[0127] DSN: Employs a node self-activation mechanism to generate sparse indicator vectors, compressing and expanding the network during training. Simultaneously, by sampling feature vectors from the class distribution, it supports feature distribution preservation, effectively avoiding catastrophic forgetting and overfitting problems.

[0128] CEC: It avoids catastrophic forgetting by storing old knowledge through prototyping. Furthermore, it utilizes an attention network to optimize the similarity relationship between sample embeddings and the prototype, making them more adaptable to new classes.

[0129] STOP: Employs a spatiotemporal attention routing mechanism and orthogonal projection capsule layers. By encoding spatiotemporal transformation information and maintaining the distance between new and old class prototypes, it effectively prevents catastrophic forgetting while avoiding overfitting.

[0130] 3. Implementation details

[0131] In the experiments, ResNet50 was used as the backbone network, and a TSM structure (described in the literature TSM: Temporal Shift Module for Efficient Video Understanding) was adopted on this basis to process the input format of video frames. The backbone network was built based on the PyTorch library and trained using the SGD optimizer with a learning rate set to 0.001.

[0132] During the training phase of the self-attention module, a basic self-attention structure consisting of key-value pairs K, query Q, value V, layer normalization LN, and fully connected layers FC is used for the UCF-101 and Kinetics-100 datasets; while for the HMDB-51 dataset, a residual structure based on the input and output of the basic self-attention structure is used. New category samples are synthesized by randomly rotating video frames from angles of 90°, 180°, and 270°. Both the self-attention module and the backbone network are trained with a learning rate of 0.0005, which decays to half its original value every 20 training epochs.

[0133] Regarding adaptive testing time, prototypes are divided into three categories: error-prone prototypes, ordinary prototypes, and difficult-to-classify prototypes. Let n be the total number of prototypes in the current session, and [nc0.3, nc0.5, nc0.7] represent the number of prototypes with a cosine similarity greater than [0.3, 0.5, 0.7] to prototypes of category c. If [nc0.3, nc0.5, nc0.7] is greater than [0.5... n, 0.15 n, 0.06 If n], then the prototype of category c is defined as the error-prone classification prototype; if it is less than [0.2], then the prototype of category c is defined as the error-prone classification prototype. n, 0.05 n, 0.01335 If the prototype categories are [mistake-prone prototype, normal prototype, and non-mistake-prone prototype], then they are defined as error-prone prototypes; the rest are ordinary prototypes. Based on the prototype categories and confidence levels of the predicted categories of the test samples, the samples can be divided into three categories: high-quality samples, ordinary samples, and low-quality samples. Let s be the current session number. When the prototype categories of the predicted categories are [mistake-prone prototype, ordinary prototype, and error-prone prototype], high-quality samples must satisfy a confidence level higher than [0.95-s]. 0.0075, 0.95-s 0.015, 0.7-s [0.015]; Inferior samples must meet the requirement that the confidence level is below [0.85-s]. 0.0175, 0.8-s 0.02, 0.55-s [0.02]; the rest are regular samples. High-quality samples and regular samples will be used to update the prototype, with high-quality samples having a weight of 1, and regular samples assigned weights of [0.3, 0.65, 1] ​​based on their prototype category [misclassifiable prototype, regular prototype, difficult-to-misclassify prototype]. Only high-quality samples participate in model optimization. A smaller learning rate lr1 is used to optimize the K, Q, and V parts, and a larger learning rate lr2 is used to optimize the LN and FC parts. Both lr1 and lr2 increase with increasing sessions according to a cosine curve: lr1 ranges [1e-9, 1.2e-7] and lr2 ranges [1e-7, 5e-5] for the UCF-101 dataset; lr1 ranges [1e-9, 1e-5] and lr2 ranges [5e-6, 5e-4] for the Kinetics-100 dataset; and lr1 ranges [5e-8, 1.2e-7] and lr2 ranges [5e-9, 5e-5] for the HMDB-51 dataset. Prototype updates and model optimizations are performed only once per batch of test data.

[0134] 4. Comparison with advanced methods

[0135] In this section, the experimental results and various baselines of the method will be presented. As shown in Table 1, in the experiments on the UCF101 dataset, a 10-step, 5-way, 5-shot setup was used, achieving a highest average accuracy of 69.26% and a lowest PDR (performance dropping rate) of 22.27%. Specifically, the method of this embodiment improves upon the DSN and FACT methods by 2.63% and 5.38%, respectively. For CEC, the average accuracy of the method is 1.56% higher, and the accuracy drop is significantly reduced in the last 5 sessions, with even the accuracy in session 7 being higher than that in session 6. Compared with previous methods, the method of this embodiment corrects each prototype during testing based on the confidence level of the test samples and the type of the prototype. This effectively alleviates the drawback of insufficient training samples for synthesizing incremental class prototypes in the small increment problem, leading to insufficient representativeness of the incremental class prototypes. Specifically, the prototype type division mainly targets the feature space of the prototype. The addition of new classes causes partial crowding in the originally evenly divided feature space, and prototypes in crowded regions are more prone to misclassification than other prototypes. The revised prototype weights reflect a comprehensive consideration of prototype type and sample confidence. Maximizing the influence of high-level samples and revising new class prototypes while accepting a small number of misclassified samples yields better results. Furthermore, using high-level samples to fine-tune the entire attention module, rather than just a specific layer, further optimizes the similarity relationship between test samples and all prototypes. Its learning rate increases with each session in a cosine curve manner, ensuring that the latest class also has an influence on module fine-tuning and avoiding fitting and catastrophic forgetting in the early stages of testing.

[0136] Table 1

[0137]

[0138] The experimental results of the method of this invention on the Kinetics-100 dataset are shown in Table 2. The method of this invention achieved a maximum average accuracy of 56.79%. Kinetics-100 is more challenging than UCF101, partly because it contains fewer videos with extremely similar content, increasing the difficulty of model training. However, based on the test data, the method of this invention can still improve the average accuracy, specifically, by 2.45% and 3.46% compared to the DSN and FACT methods, respectively.

[0139] Table 2

[0140]

[0141] Table 3 shows the experimental results of the method of this embodiment on the HMDB-51 dataset. The method of this embodiment achieved the best accuracy in each session, with a highest average accuracy of 39.44%. Due to the relatively small size of the dataset, the method of fitting the prototype endows the model with strong invariance, resulting in higher accuracy in the first few sessions. Furthermore, the reasonable use of test data to refine the model also alleviated the problem of limited data, thus achieving good results. However, there were instances where the accuracy of later sessions was higher than that of earlier sessions; the accuracy of sessions 2 and 9 was higher than that of sessions 1 and 8, respectively.

[0142] Table 3

[0143]

[0144] 5. Ablation test

[0145] This section evaluates the effectiveness of the algorithm and examines the characteristics of each component through a series of experiments. Specifically, it mainly presents experimental results based on the UCF-101 dataset.

[0146] 5.1 Effectiveness Analysis of Each Component

[0147] Based on the UCF-101 dataset, Table IV explores the contributions of the prototype module and loss function to model performance. Table 4 lists the results of the TSM-Resnet50 backbone network and self-attention module under three evaluation metrics: final session accuracy, average accuracy, and performance degradation rate. After adopting the prototype module, compared with the backbone network, the final session accuracy improved by 0.9%, the average accuracy improved by 1.15%, and the performance degradation rate decreased by 0.46%. This is because a large amount of test data compensated for the insufficient training samples for incremental classes, both increasing the number of samples used to construct the prototype to effectively correct it and fine-tuning the base class prototype, thereby improving the overall performance. When the model introduced the loss function, the average accuracy improved by 0.23%. By adopting an incremental learning rate strategy, making minor adjustments in the initial session and implementing moderate optimization in subsequent sessions, the model's adaptability was effectively enhanced.

[0148] Table 4

[0149]

[0150] Simultaneous application of the prototyping module and loss function yields optimal performance, achieving a final session accuracy of 60.03%, an average accuracy of 69.25%, and a performance degradation rate of 22.27%, all superior to other configurations. These results fully demonstrate the positive impact of the prototyping module and loss function on performance improvement.

[0151] 5.2 Compare the impact of different thresholds for determining prototype types

[0152] Based on the number of prototypes with high similarity to a given prototype, a threshold is set to classify prototypes into error-prone, normal, and non-error-prone types. As shown in Table 5, changing this threshold, which modifies the determination of prototype type, will significantly affect the experimental results. If the condition for becoming a non-error-prone prototype is relaxed, with the addition of new classes and the crowding of the prototype pool, more non-error-prone prototypes mean greater tolerance for model fine-tuning, which may lead to more misclassified samples negatively adjusting the model. Conversely, if more prototypes are judged as error-prone, it means that the fine-tuning conditions for the model are more stringent, preventing the full potential of the TTA effect from being realized. A more moderate threshold selection will yield better results. Furthermore, it is not possible to determine the prototype type solely based on the number of prototypes with similarity greater than a single similarity; multiple similarity values ​​need to be set when necessary. Some prototypes, although they do not have other extremely similar prototypes, may still be judged as error-prone prototypes if they have multiple prototypes with moderate similarity. The model in this embodiment counts the number of prototypes with similarity greater than 0.3, 0.5, and 0.7 to improve the prototype discrimination effect.

[0153] Table 5

[0154]

[0155] Note: esp represents the error-prone classification prototype, lesp represents the non-error-prone classification prototype, op represents the normal prototype, nps-0.3 represents the number of prototypes with a similarity greater than 0.3, and s represents the current session ID. Specifically, under low threshold conditions: for the error-prone classification prototype, the number of prototypes with similarities exceeding 0.3, 0.5, and 0.7 should be greater than 0.4s, 0.12s, and 0.048, respectively. s.

[0156] 5.3 Compare the impact of different thresholds for determining sample types

[0157] Based on the confidence level of the test samples and the prototype type of their predicted classes, a threshold is set for the confidence level to categorize the test samples into high-quality, average, and low-quality samples. Setting different thresholds will affect the number of samples in each category. As shown in Table 6, a higher threshold will reduce the number of high-quality samples, ensuring a higher accuracy rate for them, but it will weaken the fine-tuning of the model. Conversely, a lower threshold will allow for more significant model adjustments, resulting in more misclassified samples being included. Balancing the threshold to maintain a reasonable ratio between correctly classified and misclassified samples will have a significant impact on the final result. Furthermore, as new classes are added, the confidence level of the test samples generally decreases, therefore the threshold should also be adjusted downwards accordingly.

[0158] Table 6

[0159]

[0160] 5.4 Compare the impact of different prototype update weights

[0161] This study investigated the impact of adjusting the update weights of the prototypes using test samples on the model. Each class's prototype initially consisted of training samples, each with a weight of 1, which were then fine-tuned using test samples with varying weights. The high-quality test samples consisted almost entirely of correctly classified samples, effectively correcting the model, while the low-quality test samples contained a large number of misclassified samples, almost entirely requiring discarding. Table 7 shows that setting the weight of the high-quality test samples to 1 to fully utilize their effectiveness, setting the weight of the low-quality test samples to 0 to isolate them from most misclassified samples, and appropriately adjusting the weights of the medium-quality test samples according to the prototype type yielded the best results.

[0162] Table 7

[0163]

[0164] 5.5 Comparison with other TTA methods

[0165] This paper selects four TTA methods: TENT, SAR, SHOT, and T3A. TENT relies on calculating the entropy of the prediction results of test samples to select samples with low entropy, i.e., high prediction accuracy, for fine-tuning the Batch Normalization (BN) layer of the model. SAR, building on TENT, further selects samples with small gradients from those with low entropy to fine-tune the LN layer of the model. SHOT adaptively adjusts the feature extraction module of the network based on the entropy of the prediction results of a single sample and the probability distribution of the prediction results of all samples, achieving alignment between the source and target domains. T3A selects test samples with low entropy for the prediction results of the source domain classifier, embeds and normalizes them, collects them, and uses their centroids as the classifier for the target domain samples, then performs secondary prediction to achieve domain adaptation. Due to model limitations, SAR and TENT fine-tune the LN layer of the attention module in the experiment, while SHOT fine-tunes the backbone. As shown in Table 8, SAR can select higher-quality test samples for model updates compared to TENT, thus achieving higher accuracy. Compared to SAR, the method in this embodiment of the invention can effectively correct new class prototypes composed of small samples and fine-tune the entire attention module including the LN layer, making the relationships between various prototypes clearer, thus achieving better results. For SHOT, a smaller learning rate is used to avoid catastrophic forgetting, but fine-tuning the backbone still violates the invariance of the prototypes, and the indiscriminate selection of all test samples for fine-tuning also greatly limits the performance of the method. Compared to T3A, the method in this embodiment of the invention significantly increases the weight of the training sample prototypes and assigns different weights to different test samples to enhance the contribution of the training samples in small incremental learning. Therefore, the method in this embodiment of the invention has better performance even when using only prototype correction.

[0166] Table 8

[0167]

[0168] In this embodiment of the invention, addressing the issues of scarce new class samples, feature bias, and classifier imbalance in incremental action recognition with few samples, a multi-level adaptive test time framework is proposed. This method is centered on a two-stage structure of "base class learning—new class adaptation": in the base class stage, the model learns general spatiotemporal features through sufficient samples and introduces a multi-head self-attention mechanism to improve generalization ability; in the new class stage, the hierarchical adaptive test time mechanism enables the model to self-update during inference. Overall, incremental action recognition with few samples is achieved by jointly updating the prototype and network parameters. To update test samples more accurately, this embodiment of the invention designs a multi-level update module, enabling the network to selectively update key samples while eliminating interference samples that may mislead the model.

[0169] This not only improves the model's ability to respond to new types of samples, but also effectively suppresses catastrophic forgetting and bias accumulation caused by noisy samples, thereby achieving a stable and efficient incremental learning process, and thus improving the accuracy and efficiency of small sample action recognition.

[0170] Figure 2 This is a block diagram of a time-adaptive small-sample incremental action recognition device provided in an embodiment of the present invention. This device is used for testing a time-adaptive small-sample incremental action recognition method. (Refer to...) Figure 2 The device includes:

[0171] The acquisition unit 2010 is used to acquire a training set, a test set, and a few-shot action recognition model to be trained. The training set includes base class training samples and new class training samples. The few-shot action recognition model includes a video feature extraction backbone, a classifier, and a multi-head self-attention module.

[0172] The first training unit 2020 is used to train the video feature extraction backbone and classifier based on the base class training samples, so as to obtain the trained video feature extraction backbone and the first classifier.

[0173] The second training unit 2030 is used to construct pseudo-incremental samples based on the base class training samples, and to train the multi-head self-attention module based on the pseudo-incremental samples to obtain the initially trained multi-head self-attention module.

[0174] The first update unit 2040 is used to determine the prototype features of each action category based on the video features of all base class training samples, and use the prototype features as new classifier weights to obtain the second classifier.

[0175] The second update unit 2050 is used to extract the backbone based on the new class training samples and the trained video features, obtain the video features of the new class training samples, and then obtain the prototype features of each new class. The prototype features of the new class are added to the second classifier to obtain the third classifier.

[0176] The classification unit 2060 is used to classify the prototype features in the third classifier and determine the prototype category of each prototype feature. Based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier, it determines the pseudo-label and confidence of each test sample. Based on the confidence of each test sample and the prototype category of the prototype features of the pseudo-label, it classifies the test samples and determines the sample category of each test sample.

[0177] The third update unit 2070 is used to update the prototype features corresponding to the pseudo-labels of the test samples based on the prototype category of the prototype features, the sample category of the test samples, the prototype features of the pseudo-labels of the test samples, and the video features of the test samples.

[0178] The third training unit 2080 is used to update the initially trained multi-head self-attention module based on the test samples of the preset sample categories, cross-entropy loss, and entropy regularization loss.

[0179] The repeated execution unit 2090 is used to iteratively execute the second update unit 2050 to the third training unit 2080 multiple times to obtain the trained multi-head self-attention module, and then obtain the trained few-sample action recognition model.

[0180] The action recognition unit 2100 is used to input the video to be classified into a trained few-sample action recognition model to obtain the action recognition result of the video to be classified.

[0181] Figure 3 This is a schematic diagram of the structure of a small-sample incremental action recognition device with adaptive test time provided in an embodiment of the present invention, as shown below. Figure 3 As shown, a small-sample incremental action recognition device with adaptive test time can include the above-mentioned Figure 2 The illustrated test-time adaptive small-sample incremental action recognition device 310 may optionally include a first processor 2001.

[0182] Optionally, the small-sample incremental action recognition device 310 with adaptive test time may also include a memory 2002 and a transceiver 2003.

[0183] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0184] The following is combined with Figure 3 The following is a detailed introduction to the various components of the small-sample incremental action recognition device 310 with adaptive testing time:

[0185] The first processor 2001 is the control center of the test time adaptive small sample incremental action recognition device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0186] Optionally, the first processor 2001 can perform various functions of the small sample incremental motion recognition device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0187] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0188] In a specific implementation, as one example, the small-sample incremental action recognition device 310 with adaptive test time may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0189] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0190] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the test time adaptive small sample incremental action recognition device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0191] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0192] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0193] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently, and its interface circuit can be used with the small sample incremental motion recognition device 310 with test time adaptive small sample incremental motion recognition device. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0194] It should be noted that, Figure 3 The structure of the test time adaptive small sample incremental motion recognition device 310 shown does not constitute a limitation on the router. Actual test time adaptive small sample incremental motion recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0195] Furthermore, the technical effect of the test time adaptive small sample incremental action recognition device 310 can be referred to the technical effect of the test time adaptive small sample incremental action recognition method described in the above method embodiments, and will not be repeated here.

[0196] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0197] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0198] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0199] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0200] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0201] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0202] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0203] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0204] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0205] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0207] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0208] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A small-sample incremental action recognition method with adaptive test time, characterized in that, The method includes: S01. Obtain a training set, a test set, and a few-shot action recognition model to be trained. The training set includes base class training samples and new class training samples. The few-shot action recognition model includes a video feature extraction backbone, a classifier, and a multi-head self-attention module. S02. Based on the base class training samples, train the video feature extraction backbone and classifier to obtain the trained video feature extraction backbone and first classifier. S03. Construct pseudo-incremental samples based on the base class training samples, and train the multi-head self-attention module based on the pseudo-incremental samples to obtain the initially trained multi-head self-attention module. S04. Based on the video features of all base class training samples, determine the prototype features of each action category, and use the prototype features as the weights of the new classifier to obtain the second classifier. S05. Extract the backbone based on the new class training samples and the trained video features, obtain the video features of the new class training samples, and then obtain the prototype features of each new class. Add the prototype features of the new class to the second classifier to obtain the third classifier. S06. Classify the prototype features in the third classifier and determine the prototype category of each prototype feature; based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier, determine the pseudo-label and confidence of each test sample; based on the confidence of each test sample and the prototype category of the prototype features of the pseudo-label, classify the test samples and determine the sample category of each test sample. S07. Update the prototype features corresponding to the pseudo-labels of the test samples based on the prototype category of the prototype features, the sample category of the test samples, the prototype features of the pseudo-labels of the test samples, and the video features of the test samples. S08. Update the pre-trained multi-head self-attention module based on the test samples of the preset sample categories, cross-entropy loss, and entropy regularization loss. S09. Repeatedly execute S05-S08 to obtain a trained multi-head self-attention module, and then obtain a trained few-sample action recognition model. S10. Input the video to be classified into the trained few-sample action recognition model to obtain the action recognition result of the video to be classified.

2. The small-sample incremental action recognition method with adaptive test time according to claim 1, characterized in that, S03 involves constructing pseudo-incremental samples based on base class training samples, and training the multi-head self-attention module using these pseudo-incremental samples to obtain a pre-trained multi-head self-attention module, including: S031. Randomly select multiple action categories from the base class training samples, and randomly select one sample for each action category to form a pseudo base class support set; rotate the base class training samples in the pseudo base class support set at random angles to construct a pseudo incremental support set; S032. Select another sample from the selected multiple action categories to form a pseudo base class query set, and apply the same rotation transformation as the pseudo incremental support set to the base class training samples in the pseudo base class query set to obtain the pseudo incremental query set. S033. Input the query samples from the pseudo-base class query set and the pseudo-increment query set into the trained video feature extraction backbone to obtain the video features of the query samples. S034. Input the pseudo-base class support samples in the pseudo-base class support set into the trained video feature extraction backbone to obtain the video features of the pseudo-base class support samples. S035. Input the pseudo-incremental support samples in the pseudo-incremental support set into the trained video feature extraction backbone to obtain the video features of the pseudo-incremental support samples. S036. Based on the video features of the query sample, the video features of the pseudo-base class support sample, the video features of the pseudo-increment support sample, and the cross-entropy supervised loss function, the multi-head self-attention module is trained to obtain the initially trained multi-head self-attention module.

3. The small-sample incremental action recognition method with adaptive test time according to claim 1, characterized in that, S04 determines the prototype features of each action category based on the video features of all base class training samples, and uses the prototype features as new classifier weights to obtain a second classifier, including: The video features of all base class training samples are classified according to action categories. The feature values ​​of all video features corresponding to each action category are added together according to the dimension and then averaged to obtain the prototype features of each action category. The prototype features are used as the weights of the new classifier to obtain the second classifier.

4. The small-sample incremental action recognition method with adaptive test time according to claim 3, characterized in that, The step S06 involves classifying the prototype features in the third classifier and determining the prototype category corresponding to each prototype feature, including: S061. For the prototype feature of any action category in the third classifier, calculate the cosine similarity between the prototype feature of the action category and the prototype features of each other action category to obtain multiple similarity values. Compare each of the multiple similarity values ​​with a preset similarity threshold to determine the number of similarity values ​​greater than the preset similarity threshold. Based on the number and the preset prototype classification rules, determine the prototype category corresponding to the prototype feature of the action category.

5. The small-sample incremental action recognition method with adaptive test time according to claim 4, characterized in that, S06 determines the pseudo-label and confidence level of each test sample based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier. Based on the confidence level of each test sample and the prototype category of each prototype feature in the third classifier, the test samples are classified, including: S062. Based on the video features of the test samples and the pre-trained multi-head self-attention module, obtain the attention features of the test samples. S063. Based on the prototype features of each action category in the third classifier and the pre-trained multi-head self-attention module, obtain the attention features of the prototype features of each action category. S064. Calculate the score for each test sample for each action category based on the cosine similarity between the attention features of the test sample and the prototype features of the attention features of the new training sample. S065. The action category corresponding to the highest score of each test sample is determined as the pseudo-label of the test sample, and the highest score of each test sample is determined as the confidence level of each test sample. S066. Based on the confidence level of each test sample, the prototype category of the prototype feature of the pseudo-label, and the preset sample classification rules, classify the test samples to obtain the sample category of each test sample; the sample category includes high-level samples, medium-level samples, and low-level samples.

6. The small-sample incremental action recognition method with adaptive test time according to claim 1, characterized in that, S07 updates the prototype features corresponding to the pseudo-labels of the test samples based on the prototype category of the prototype features, the sample category of the test samples, the prototype features of the pseudo-labels of the test samples, and the video features of the test samples, including: S071. Determine the sample class weight of each test sample based on the number of sample classes corresponding to each test sample, and determine the pseudo-label prototype class weight of each test sample based on the number of prototype classes corresponding to each pseudo-label. S072. For any test sample, the ratio of the pseudo-label prototype class weight to the sum of the pseudo-label prototype class weight and the sample class weight is determined as the prototype feature weight of the test sample's pseudo-label; the ratio of the sample class weight to the sum of the pseudo-label prototype class weight and the sample class weight is determined as the video feature weight of the test sample; the product of the prototype feature weight and the prototype feature is added to the product of the video feature weight and the video feature to obtain the updated prototype feature corresponding to the pseudo-label of the test sample; and the sum of the pseudo-label prototype class weight and the sample class weight is used as the updated prototype feature weight of the pseudo-label to update the third classifier.

7. The small-sample incremental action recognition method with adaptive test time according to claim 5, characterized in that, S08 updates the initially trained multi-head self-attention module based on test samples of preset sample categories, cross-entropy loss, and entropy regularization loss, including: Test samples of the high-level sample category are selected. Cross-entropy loss is calculated based on the confidence of each high-level sample. Entropy regularization loss is calculated based on the score of each high-level sample for each action category. Composite loss function is calculated based on the sum of cross-entropy loss and entropy regularization loss. The multi-head self-attention module that has been initially trained is then trained based on composite loss function.

8. A test-time adaptive small-sample incremental action recognition device, wherein the test-time adaptive small-sample incremental action recognition device is used to implement the test-time adaptive small-sample incremental action recognition method as described in any one of claims 1-7, characterized in that, The device includes: The acquisition unit is used to acquire a training set, a test set, and a few-shot action recognition model to be trained. The training set includes base class training samples and new class training samples. The few-shot action recognition model includes a video feature extraction backbone, a classifier, and a multi-head self-attention module. The first training unit is used to train the video feature extraction backbone and classifier based on the base class training samples, so as to obtain the trained video feature extraction backbone and the first classifier. The second training unit is used to construct pseudo-incremental samples based on the base class training samples, and to train the multi-head self-attention module based on the pseudo-incremental samples to obtain the initially trained multi-head self-attention module. The first update unit is used to determine the prototype features of each action category based on the video features of all base class training samples, and use the prototype features as new classifier weights to obtain the second classifier. The second update unit is used to extract the backbone based on the new class training samples and the trained video features, obtain the video features of the new class training samples, and then obtain the prototype features of each new class. The prototype features of the new class are added to the second classifier to obtain the third classifier. The classification unit is used to classify the prototype features in the third classifier and determine the prototype category of each prototype feature. Based on the video features of the test samples, the pre-trained multi-head self-attention module, and the prototype features of each action category in the third classifier, it determines the pseudo-label and confidence of each test sample. Based on the confidence of each test sample and the prototype category of the prototype features of the pseudo-label, it classifies the test samples and determines the sample category of each test sample. The third update unit is used to update the prototype features corresponding to the pseudo-labels of the test samples based on the prototype category of the prototype features, the sample category of the test samples, the prototype features of the pseudo-labels of the test samples, and the video features of the test samples. The third training unit is used to update the initially trained multi-head self-attention module based on the test samples of the preset sample categories, cross-entropy loss, and entropy regularization loss. The repeated execution unit is used to iterate S05-S08 multiple times to obtain a trained multi-head self-attention module, and then to obtain a trained few-sample action recognition model. The action recognition unit is used to input the video to be classified into a trained few-sample action recognition model to obtain the action recognition result of the video to be classified.

9. A small-sample incremental action recognition device with adaptive test time, characterized in that, The test-time adaptive small-sample incremental action recognition device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images