A small sample action recognition method, device, equipment and medium

Abstract

The application discloses a small sample action recognition method and device, equipment and medium, and relates to the technical field of action recognition. The application utilizes limited samples to establish a support set and a query set, combines the samples in the support set and the samples in the query set, and injects the category semantic and timing characteristic information of the support set into the timing modeling process of the query set samples through a cross-sample attention mechanism, thereby enhancing the discrimination ability of the query features. In the training process using each sample in the query set, the knowledge of each sample in the support set can be applied, thereby improving the feature quantity participating in recognition, achieving high-precision and high-generalization ability training in the case of small samples, and in application, the semantic and timing information of the support set is injected into the frame-level features of the query video relying on the category prototypes and historical knowledge learned by the support set, thereby enhancing the representation of the query video to each action category and improving the recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of action recognition technology, and in particular to a method, apparatus, device and medium for small sample action recognition. Background Technology

[0002] Few-Shot Action Recognition (FSAR) aims to accurately identify new action categories using a small number of samples and is widely used in video surveillance, human-computer interaction, and smart healthcare. Existing FSAR methods typically rely on metric learning and meta-learning frameworks. The basic idea is to map query videos and supporting videos to a shared feature space and match them using predefined or adaptive similarity metrics. To model temporal features, existing techniques mainly employ temporal representation learning, spatiotemporal attention mechanisms, and feature alignment strategies. Temporal representation learning models action features by capturing the temporal dependencies between video frames. Classical methods such as OTAM (Ordered Temporal Alignment Module) use Dynamic Time Warping (DTW) to finely align video frame sequences, while HyRSM (Hybrid Relation Guided Set Matching) uses a bidirectional average Hausdorff space metric to calculate the temporal matching score between videos. Spatiotemporal attention mechanisms enhance the expressive power of local spatiotemporal features through multi-scale spatiotemporal attention modules or by combining multimodal information (such as depth, optical flow, etc.). Feature alignment strategies improve the accuracy of similarity measurements by aligning features between query videos and supporting videos at the frame or tuple level.

[0003] Existing technologies mainly involve: first, extracting frame-level features from videos using a backbone network (such as 3D-CNN (3D Convolutional Neural Network)); then, independently learning the temporal features of each video; and finally, completing the classification task by calculating the similarity between the query and support samples. However, despite significant progress in few-shot action recognition tasks, existing methods still have many limitations that affect accuracy and generalization ability.

[0004] However, existing methods typically map query videos and supporting videos into a discriminative feature space, then perform supporting video matching based on a predefined or learned distance metric. While these methods achieve impressive accuracy, they still face two major limitations: firstly, they learn frame-level features independently for each video, neglecting to model relationships between videos and tasks, thus failing to extract effective information across videos and tasks; secondly, they fail to capture temporal dependencies between and within videos, leading to poor performance on data-scarce tasks. Therefore, existing methods utilize the temporal features of each video independently for modeling, ignoring explicit associations between videos and tasks. As a result, shared temporal patterns across videos cannot be effectively captured. For example, the temporal evolution of the same action may be similar in different videos (e.g., the take-off, flight, and landing phases of a "jump"), but existing methods fail to extract these shared temporal patterns. Simultaneously, the models fail to reuse temporal knowledge learned from previous tasks, requiring the relearning of temporal features each time a new task is encountered, thus reducing data utilization efficiency.

[0005] While some methods improve model generalization through task-level feature alignment, these methods still maintain an interaction granularity at the video level, failing to achieve finer-grained frame-level inter-video interactions. For example, certain keyframes in the supporting video (such as the release moment of a "shooting" action) may have potential semantic relationships with multiple frames in the query video, but existing methods cannot capture this fine-grained temporal interaction, thus affecting the accuracy and effectiveness of the overall alignment.

[0006] Furthermore, existing technologies typically handle each task in isolation, lacking cross-task knowledge transfer mechanisms, which prevents the effective reuse of temporal patterns from historical tasks. For example, the temporal features of racket swing learned in the "sports action recognition" task can provide a reference for subsequent "fitness action recognition" tasks, but existing methods have failed to establish a cross-task knowledge base and lack effective knowledge transfer mechanisms, resulting in insufficient support for new tasks and affecting the model's transferability.

[0007] The root cause of these problems lies in the fact that existing technologies generally focus only on temporal modeling of a single video, failing to model the multi-level temporal relationships between frames, videos, and tasks from a global perspective. Existing methods fail to fully explore the potential semantic associations across videos and tasks, resulting in models struggling to effectively improve recognition accuracy and generalization ability when data is scarce. Summary of the Invention

[0008] The purpose of this application is to provide a method, apparatus, device, and medium for action recognition with a small sample size, so as to improve the accuracy and generalization ability of action recognition when the sample size is small.

[0009] To achieve the above objectives, this application provides the following solution.

[0010] In a first aspect, this application provides a method for few-sample action recognition, comprising the following steps:

[0011] Construct a support set and a query set; the support set includes N×K first video samples, and the query set includes L second video samples; where N is the number of action types, K is the number of second video samples for each action type, and L is the number of second video samples.

[0012] Each second video sample is combined with all the first video samples in the support set to obtain L sample combinations;

[0013] The video encoder model is trained using L combinations of samples to obtain the trained video encoder as an action recognition model.

[0014] The video to be identified is combined with all the first video samples in the support set to obtain the video combination to be identified;

[0015] The video to be identified is combined and input into the action recognition model to obtain the action type corresponding to the video to be identified.

[0016] Secondly, this application provides a few-sample action recognition device, which applies the above-described few-sample action recognition method. The few-sample action recognition device includes:

[0017] A dataset construction module is used to construct a support set and a query set; the support set includes N×K first video samples, and the query set includes L second video samples; where N is the number of action types, K is the number of second video samples for each action type, and L is the number of second video samples.

[0018] The sample stitching module is used to combine each second video sample with all the first video samples in the support set to obtain L sample combinations.

[0019] The model training module is used to train the video encoder model using L sample combinations to obtain the trained video encoder as an action recognition model.

[0020] The video combination module is used to combine the video to be identified with all the first video samples in the support set to obtain the video combination to be identified;

[0021] The action recognition module is used to input the combined video to be recognized into the action recognition model to obtain the action type corresponding to the video to be recognized.

[0022] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described small sample action recognition method.

[0023] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned small sample action recognition method.

[0024] According to the specific embodiments provided in this application, this application has the following technical effects.

[0025] This application provides a method, apparatus, device, and medium for small-sample action recognition. This application utilizes a limited set of samples to establish a support set and a query set, and combines samples from the support set and the query set. This allows the knowledge of samples from the support set to be applied during training using samples from the query set, thereby increasing the number of features involved in recognition and achieving high-precision and high-generalization training even with small samples. In specific applications, the knowledge of samples from the support set is also added to the video to be recognized to improve recognition accuracy.

[0026] This application also utilizes prototypes obtained from each training or recognition to expand and update the knowledge base, which can further improve the generalization ability of action recognition. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a small-sample action recognition method provided in an embodiment of this application.

[0029] Figure 2 This is a schematic diagram of a small sample action recognition method provided in an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of a video encoder model provided in one embodiment of this application.

[0031] Figure 4 This is a schematic diagram illustrating the construction of a mask matrix according to an embodiment of this application.

[0032] Figure 5 This is a flowchart illustrating the workflow of a video encoder model provided in one embodiment of this application.

[0033] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] This application aims to address the following key issues existing in few-shot action recognition (FSAR) technology:

[0037] First, existing few-shot action recognition (FSAR) methods primarily rely on learning independent video features, neglecting explicit relationship modeling between videos and tasks. This makes it difficult to effectively capture the temporal patterns of the same action across different videos, especially when actions exhibit similar temporal evolution across videos; existing methods cannot effectively extract shared temporal features. Furthermore, temporal knowledge from past tasks cannot be effectively reused, requiring relearning of temporal features each time a new task is encountered, thus reducing data utilization efficiency. Therefore, this application proposes a few-shot action recognition method, apparatus, device, and medium, which is a hierarchical temporal relationship modeling framework, such as... Figure 2 As shown, the aim is to jointly model the temporal correlation between videos and the knowledge transfer between tasks, improve the ability to capture shared temporal patterns across videos, and effectively reuse the temporal knowledge of historical tasks, thereby improving the model's recognition accuracy and data utilization efficiency.

[0038] Secondly, although existing technologies have attempted to improve the generalization ability of models through task-level feature alignment, these methods remain at the video level in terms of interaction granularity, failing to achieve finer-grained frame-level inter-video interactions. To address this issue, this application enhances the temporal alignment accuracy of the model through fine-grained cross-video frame-level interactions (inter-video semantic association, ISC), thereby improving the model's ability to capture keyframes and temporal changes.

[0039] Furthermore, existing technologies lack effective cross-task temporal knowledge transfer mechanisms, making it impossible for temporal patterns from historical tasks to support new tasks. To overcome this problem, this invention proposes an inter-task knowledge transfer (IKT) strategy. By establishing relationships and a knowledge base between tasks, it stores and reuses diverse temporal patterns from historical tasks, further improving the efficiency of learning new tasks and the overall generalization ability of the model.

[0040] Therefore, the technical solution of this application effectively overcomes many limitations of the prior art by hierarchically modeling the temporal relationship between videos, knowledge transfer between tasks, and fine-grained cross-video interaction, and significantly improves the accuracy, efficiency and generalization ability of small sample action recognition tasks under data-scarce conditions.

[0041] In one exemplary embodiment, a few-sample action recognition method is provided, such as... Figure 1 As shown, it includes the following steps 101-105.

[0042] Step 101: Construct a support set and a query set; the support set includes N×K first video samples, and the query set includes L second video samples; where N is the number of action types, K is the number of second video samples for each action type, and L is the number of second video samples.

[0043] Step 102: Combine each second video sample with all the first video samples in the support set to obtain L sample combinations.

[0044] Step 103: Train the video encoder model using L sample combinations to obtain the trained video encoder as the action recognition model.

[0045] Step 104: Combine the video to be identified with all the first video samples in the support set to obtain the combined video to be identified.

[0046] Step 105: Input the video combination to be identified into the action recognition model to obtain the action type corresponding to the video to be identified.

[0047] The aforementioned few-sample action recognition method aims to address the problems in existing technologies, such as insufficient fine-grained action recognition capability, poor reusability of temporal knowledge, and weak generalization ability of long sequences, caused by neglecting cross-video and cross-task semantic associations, by unifying the relationship modeling between frames, videos, and tasks.

[0048] In another exemplary implementation, step 101 above is a data preprocessing stage, which divides the training dataset into a support set (K labeled videos sampled for each category) and a query set (containing samples from the remaining categories) by constructing an N-way K-shot scenario task.

[0049] The N-way K-shot task involves identifying real-world scenarios with limited sample sizes. Specifically, the model faces a new N-way K-shot classification problem in each sub-task. In reality, there are often only a very small number of labeled samples, but the model needs to quickly generalize to new categories. By repeatedly subjecting the model to the N-way K-shot task, the model learns how to construct category representations using limited samples for classification.

[0050] This application uses features of the same category but at different frame levels extracted multiple times as samples for the support set and query set. The support set randomly selects N categories, and then randomly selects K labeled videos from these N categories. The query set is constructed by selecting the N categories chosen from the support set, and then extracting M videos from the remaining videos, where L = N*M are unlabeled videos.

[0051] For example, the input is an N-way K-shot scenario task, where the support set contains N types of actions, with K labeled videos for each type, and the query set contains L unlabeled videos. Each video is sampled uniformly for T = 8 frames, the frame resolution is adjusted to 224×224, and the frames are preprocessed and normalized to the range [0,1].

[0052] In another exemplary embodiment, such as Figure 3 and Figure 5 As shown, Figure 3 (a), (b), and (c) in the diagram represent the overall flowchart, the cross-video semantic association flowchart, and the cross-task knowledge transfer flowchart, respectively. The video encoder model in this application includes: a visual encoder, an inter-frame temporal modeling module, an inter-video semantic association module, an inter-task knowledge transfer module, and an OTAM calculation module. This application introduces an inter-video semantic association module (ISC) and an inter-task knowledge transfer module (IKT) to model task-specific temporal information at multiple levels, and improves model performance by optimizing computational efficiency. This not only enhances the accuracy of recognition but also strengthens the model's generalization ability to new action categories. Figure 3As shown, the visual encoder extracts features from each video frame in the first and second video samples of each sample combination to obtain frame-level features. The inter-frame temporal modeling module concatenates the frame-level features of each video sample and the second video sample in each sample combination to obtain the joint features of each sample combination. The inter-video semantic association module enhances the joint features of each sample combination with attention features to obtain the first-level enhanced features of each sample combination. The inter-task knowledge transfer module enhances the first-level enhanced features of each sample combination with inter-task association features to obtain the second-level enhanced features of each sample combination. The OTAM calculation module performs OTAM calculation on the second-level enhanced features of each sample combination to determine the predicted action category corresponding to the second video sample in each sample combination. The OTAM calculation process is as follows: calculate the inter-frame similarity matrix between the support set features and the query set features in the joint features after the second-level enhancement. Convert the similarity to distance. Minimize the distance and then use softmax to convert it to the probability of each category. This determines which category of action each frame feature in the query set belongs to in the existing support set features.

[0053] In this embodiment, a CLIP visual encoder is used to extract video frame-level features, and the model parameters are optimized using a parameter efficient fine-tuning (PEFT) strategy to reduce computational overhead. Specifically, for each frame image I... norm Frame-level features are extracted using a pre-trained CLIP ViT-B / 16 model. The features of the first video sample in the support set are integrated as follows: The features of the second video sample in the query set are integrated into Where NK is the total number of samples in the support set (N classes × K samples).

[0054] In this embodiment of the application, the support set feature X S and query set features X q Standardization is performed to ensure temporal alignment and amplitude uniformity. Specifically, frame-by-frame Z-score standardization is applied to the support set, queries, and features. Where, μ frame This represents the channel-level mean vector calculated from the features of all video frames in the support set, ensuring brightness consistency across videos, δ. frame This represents a channel-level standard deviation vector that supports all frame features, eliminating feature scale differences caused by different video acquisition devices.

[0055] Next, a semantic association module between videos is introduced to enhance task-specific features through fine-grained cross-video interactions, as follows:

[0056] First, fine-grained cross-video interaction, such as Figure 3As shown in (a) and (b), the support set features and query set features are concatenated along the sample dimension to form joint features. Reshape the spliced ​​features into Preserving temporal independence to enable frame-level interaction across videos, a multi-head self-attention mechanism is used for cross-video frame-level interaction to generate task-specific features. and The calculation process is as follows:

[0057] Employing a multi-head self-attention mechanism for cross-line video feature interaction: Wherein, projection matrix As learnable parameters, the input features are mapped to query, key, and value spaces. The attention weight matrix is ​​then calculated. Here, D represents the feature dimension of each attention head, used to scale the dot product to prevent gradient explosion. The output yields the interactive features. Finally, the multi-head outputs are concatenated along the feature dimensions and then restored to the original dimensions using linear projection.

[0058] To further suppress noise interference, an adaptive interaction masking mechanism is designed to selectively retain support-support intra-class associations and query-support inter-class associations, thereby suppressing redundant interactions.

[0059] Design a mask matrix The query video is restricted to interacting only with supported videos. Specifically, the attention matrix A is dynamically adjusted using element-wise multiplication, formula: F. S ' =A⊙F S A is the adaptive interaction mask. It significantly enhances intra-class associations (e.g., pose consistency for the same action) and inter-class distinctions (e.g., speed differences for different actions) within the support set. Frame-level visual features and corresponding text features are concatenated along the temporal dimension and input into a temporal Transformer for inter-frame temporal modeling. This captures the temporal dependencies between video frames and yields enhanced support video and query video features. The specific steps include:

[0060] Constructing a binary mask matrix like Figure 4 As shown, the mask value between support set samples (first NK rows and columns) is 1, allowing interaction; the mask value between query-support samples (last L rows and first NK columns) is 1, allowing the query video to aggregate information from the support videos. The mask value between query-query samples (last rows and columns) is 0, disallowing redundant interaction, such as... Figure 4 As shown, Figure 4(a), (b), (c), and (d) in the diagrams are schematic diagrams of the binary mask matrix obtained by adaptive interactive masking, the binary mask matrix obtained under the condition of allowing support-support interaction, the binary mask matrix obtained under the condition of allowing query-support interaction, and the binary mask matrix obtained under the condition of allowing full interaction, respectively.

[0061] like Figure 3 As shown in (a) and (c), the mask matrix forces the model to focus only on task-relevant cross-sample interactions. Applying mask suppression J to the attention weight matrix A suppresses invalid interactions. The attention weights at invalid locations (J=0) are brought close to zero by adding a maximum negative value log 0 = -∞. This is used to enhance the support set features after output mask adjustment. Enhanced features of query sets

[0062] In summary, this application decouples dense attention into inter-frame interaction and ISC modules to obtain temporal features of the supporting and query videos. After reshaping these features, a multi-head attention mechanism and a sample attention matrix are used for cross-video interaction. By focusing on all frame features at the same time point, task-specific temporal features of the supporting and query videos are obtained. Due to the differences between the supporting and query videos, an adaptive interaction mask is proposed.

[0063] The purpose of introducing the aforementioned video-to-video semantic association module in this application is to enable the processing of all features at each time step. Within the same time step, information exchange between frames of different videos is required, the purpose of which is to support knowledge transfer to the query. This is not a feature pair, and the concatenated result is not a single feature, because the number of frames is fixed. Instead, the frame-level features of each video in each category of the support set are concatenated with the frame-level features of each video in the query set, according to the frame dimension. After concatenation, at each time step, the frame features of all videos form an attention token sequence. This results in a feature set formed by concatenating multiple frame features, rather than a single feature. Interaction after concatenation completes the knowledge transfer process where labeled support features guide query features.

[0064] The concatenated feature set has a time series length of T and a structure of (NK+L) tokens in each frame.

[0065] In this embodiment, the query set is trained without labels. During the training process, the model cannot obtain the labels of the query set samples, but it can obtain the labels of the support set samples. In this embodiment, the support set is used to guide the query set to learn features and make classification predictions.

[0066] Moreover, each few-show task is a new task. Unlike traditional full-data training that directly relies on global semantic features, in each task, after the support set and query set are concatenated, the attention considers both the support set and the query set.

[0067] The purpose of attention calculation in this embodiment is to allow each frame feature to interact with other frame features, and additionally, to allow the support set to guide the query set in learning. During the attention process on the frame features of the support set, the most relevant frames can be automatically found from the query set; this is the alignment point, and also the positive effect of attention. Frames of the same type will reinforce each other, while frames of different types will be suppressed by attention.

[0068] In another exemplary embodiment, to address the lack of cross-task transfer capabilities in existing methods, an inter-task knowledge transfer (IKT) module is designed.

[0069] First, a time-series knowledge base is constructed to store diverse time-series patterns M = m1, m2, ..., m of historical tasks. G ]∈R G×G Where G is the knowledge base capacity, in this embodiment, after frame-level temporal modeling and support-query interaction, it is still necessary to further aggregate information at the task level. Therefore, in addition to adjusting the fine-grained relationships between samples, it is also necessary to aggregate similar features. The initial knowledge base is constructed based on the support set. For each category in the current task, the frame-level features of all supporting videos of that category are first averaged along the sample dimension and the time frame dimension to obtain a vector of length D, and P is used as the prototype representation of that category. Each prototype P encodes the temporal evolution pattern of the action category. The formula for calculating the temporal prototype is as follows: Where f i,t This represents a single enhanced feature. The feature vector of the i-th video frame t, N c For one enhancement feature The number of videos included, where T is the number of augmented features. The number of frames in each video. Obtain the original temporal prototype P, and retrieve semantically similar historical knowledge from the knowledge base. Initially, all prototypes m... g Set it to the zero vector.

[0070] In the embodiments of this application, M is a memory matrix of fixed size, which contains G knowledge units. The dimension of each knowledge unit is consistent with the prototype dimension, representing the action sequence pattern vector extracted from the historical task, that is, a knowledge base that remembers many common action patterns.

[0071] In another exemplary embodiment, the inter-task knowledge transfer module includes a prototype calculation module, a historical prototype query module, a prototype enhancement module, and a knowledge base update module.

[0072] The prototype computation module aims to utilize a "prototype-to-frame" attention mechanism to process the learnable prototype vector P. i It dynamically smooths and updates the temporal features of all frames in the support set. Specifically, the module first performs dynamic smoothing and updating on each prototype vector P. i Through linear mapping W q Convert to Query vector Q=P i W q Simultaneously, all frame features of the support set after the first enhancement are mapped to key and value vectors, respectively. The model calculates the click attention between each prototype query and the key vectors of all frames in the support set, and normalizes it along the frame dimension. Finally, the obtained attention weights are weighted and aggregated onto the value vectors to obtain the temporal dynamic information most relevant to the prototype. The module also uses residual connections to add the aggregated information back to the original prototype, resulting in an updated prototype representation.

[0073] The historical prototype query module is used to find multiple historical prototypes similar to the current prototype from the knowledge base, and performs Top-K calculations on the retrieved historical prototypes to obtain the historical knowledge prototype. This module updates the prototype. Retrieving relevant time-series knowledge from a knowledge base based on cosine similarity: The topk function filters historical knowledge most relevant to the semantics of the current task. The Top-K filtering strategy first calculates the previous prototype. With the historical prototype m in the knowledge base g The cosine similarity is calculated, and then the top 40% of the most similar m values ​​are selected. g The data is aggregated based on similarity weights. The hyperparameter k = O / G determines the number of searches. P' represents historical tasks related to the current task, i.e., the retrieved memory prototypes. In this embodiment, the historical prototype query module retrieves the Top-K historical prototypes most relevant to the current prototype from the knowledge base, and performs a weighted summation of these K prototypes based on their respective attention weights to obtain the historical knowledge prototype.

[0074] The prototype enhancement module is used to enhance the primary enhancement feature based on the historical knowledge prototype to obtain the secondary enhancement feature, that is, to compare the retrieved memory prototype P' with the updated prototype. Fusion, generating task-specific enhanced prototypes The specific process is as follows: Retrieving historical knowledge P' and comparing it with the current task prototype. Element-wise addition generates an enhanced prototype: Then support set enhancement features Enhanced prototype injection of support set features is achieved through a cross-attention mechanism: This involves fusing knowledge retrieved from historical tasks with features of the current task to enhance knowledge transfer capabilities between tasks, such as... Figure 3 As shown in (b),

[0075] Knowledge base update module, such as Figure 3 As shown in (a), after each task training is completed, the temporal prototype of the current task is stored in the knowledge base. For new tasks added to the knowledge base, after the task is completed, the current prototype p is stored in the knowledge base. i (Representing the i-th prototype) is written to the knowledge base. If the knowledge base is not full, it is directly inserted; if it is full, the historical prototype that is most similar to the current prototype (highest cosine similarity) is replaced. A momentum update strategy is used to dynamically adjust the prototypes, where: m t ←μm t +(1-μ)p i .

[0076] In this embodiment, since each class has only a few samples, it is necessary to use similar experiences from historical tasks to strengthen the prototype of the current small sample. This makes the prototype features closer to the characteristics of the real class. The enhanced prototype is then injected into the support features after the first enhancement. The support features are adjusted according to the prototype, making the adjustments in the support set closer to the prototype features, thus facilitating matching with the query set.

[0077] In another exemplary embodiment, the model is guided to determine the category of the query sample using samples from the support set. Here, the final distance matrix is ​​calculated as logits, and the final query sample category prediction result is output. During the training phase, the temporal relationship between the support set and the query set is jointly explored, and the distance distribution between them is calculated using the optimal transit alignment metric (OTAM).

[0078] Specifically, after modeling various relationships, enhanced support features were obtained. and query features In this process, the i-th frame-level features in the input support set Features at the j-th frame level in the query set The data is input into the OTAM metric, and the query-supported distance distribution is calculated step by step: Metric, by default, represents the OTAM metric.

[0079] Subsequently, the calculated support-query distance is used as the log-odds input, and the model is trained under supervised supervision using the cross-entropy loss function. This enables the model to accurately capture the temporal correspondence between the query video and the correctly supporting videos. Specifically, the training process flowchart is as follows: Figure 4 As shown, the features obtained by the visual encoder CLIP are regarded as the i-th frame-level support features of each query sample and support sample, respectively. and the j-th frame-level query feature The temporal alignment distance is calculated using OTAM, and the distance matrix D is transformed into classification logits. Here, NK support samples are mapped to N classes, and cross-entropy loss is performed under supervision. The backpropagation process optimizes only the ISC and IKT modules, fixes the parameters of the CLIP feature extractor, and performs a knowledge base update after each episode is trained.

[0080] During the inference phase, the model directly uses the distance D calculated above as the log-probability output prediction result to achieve efficient identification of unknown action categories. Through this training objective, this method gradually optimizes the temporal alignment between support and query features, thereby significantly improving the model's ability to learn fine-grained dynamic patterns and its generalization ability to new tasks.

[0081] In another exemplary embodiment, the following example is provided to illustrate the specific implementation of this application.

[0082] Applying the methods described in the above embodiments to intelligent security scenarios, the method proposed in this application can quickly identify abnormal actions (such as "climbing" and "falling") based on a small number of labeled samples, and significantly improve the recognition accuracy through hierarchical relationship modeling and knowledge transfer mechanisms. The following uses "detecting wall climbing behavior" as an example to explain in detail steps 1-8.

[0083] Step 1: Task Construction and Data Input. The support set contains 5 action categories ("climbing", "normal walking", "running", "stationary", and "carrying objects"), with one labeled video for each category (5-way 1-shot task). The query set contains 10 surveillance video clips, including one clip of "climbing" behavior, with the rest representing normal activities. Features are extracted from the support set videos using the CLIPViT-B encoder.

[0084] Step 2: In the Inter-Video Semantic Association (ISC) module, the features of the support set and query set are concatenated and reshaped, and then used for frame-level interaction through multi-head self-attention. Only queries are allowed to interact with "climbing" type samples from the support set, suppressing interference from other irrelevant categories.

[0085] Step 3: Output enhanced features. The "climbing" related frame features of the query video are significantly enhanced, and the distance with the "climbing" class in the support set is reduced to 0.15 (the average distance of other classes is >0.35).

[0086] Step 4: In the inter-task knowledge transfer module, retrieve historical "climbing" class prototypes (such as "climbing over railings" and "climbing stairs") from the knowledge base, and generate enhanced prototypes after weighted aggregation.

[0087] Step 5: Inject the enhanced prototype into the support set features through cross attention to improve the discriminativeness of the "climbing" class.

[0088] Step 6: The current task's "climbing" prototype is updated to the knowledge base with momentum μ = 0.99 for reuse in future tasks.

[0089] Step 7: Perform temporal matching and classification: The bidirectional Hausdorff distance between the query video and the supporting samples of the "climbing" category is 0.12, significantly lower than other categories (e.g., the distance for "normal walking" is 0.41). Softmax probability distribution shows that the confidence level for the "climbing" category will be much higher than that for other categories.

[0090] Step 8: Real-time output and alarms. If the system detects "climbing" behavior in the queried video, it triggers an alarm signal in real time and transmits it to the security platform via API. The alarm latency is ≤10ms (NVIDIA 3090 GPU), and the video stream processing frame rate is ≥30FPS, meeting real-time monitoring requirements.

[0091] Through the above examples, this application demonstrates efficient and accurate abnormal action recognition capabilities in intelligent security scenarios, providing reliable technical support for the public safety field.

[0092] In summary, this application proposes a hierarchical relationship-enhanced few-shot action recognition method (HR). 2 G-shot significantly improves fine-grained action recognition accuracy, cross-task generalization ability, and long-sequence stability by unifying the modeling of multi-level semantic associations between frames, videos, and tasks. This technology achieves frame-level interaction across videos through the Inter-Video Semantic Association (ISC) module, capturing task-specific features; combined with the Inter-Task Knowledge Transfer (IKT) module, it reuses the temporal knowledge base of historical tasks, effectively solving the problems of missing semantic associations and low knowledge reuse efficiency caused by isolated modeling of video features in traditional methods. Its technological achievements can be widely applied in fields such as intelligent monitoring, virtual reality, human-computer interaction, and robot behavior understanding, providing an efficient and reliable solution for action understanding and decision-making in low-resource scenarios. It has significant academic value and industrialization potential, specifically offering the following advantages:

[0093] 1. Significantly improved accuracy in fine-grained motion recognition

[0094] This invention enables fine-grained interaction across videos at the frame level through the Inter-Video Semantic Association (ISC) module, effectively capturing subtle differences between similar actions (such as the speed variation between "waving" and "waving") and distinguishable features of different types of actions (such as the temporal patterns between "running" and "jumping").

[0095] 2. Significantly enhanced cross-task generalization ability

[0096] Through the inter-task knowledge transfer (IKT) module, this invention can reuse the temporal knowledge base of historical tasks, reducing the dependence of new tasks on labeled data.

[0097] 3. Improved stability of long sequence action recognition

[0098] By modeling hierarchical relationships, this invention effectively mitigates recognition errors caused by temporal misalignment or noise accumulation in long sequences.

[0099] 4. Expanded applicability across multiple scenarios

[0100] This invention demonstrates broad applicability in fields such as intelligent monitoring, human-computer interaction, and virtual reality. For example, in intelligent security scenarios, it can quickly identify abnormal actions (such as "climbing" or "falling") based on a small number of samples; in VR interaction scenarios, the generated action recognition results can drive virtual characters in real time, significantly improving the user experience.

[0101] 5. Balancing training and reasoning efficiency

[0102] Incremental learning is supported by optimizing knowledge retrieval and aggregation mechanisms. The inference phase directly utilizes the pre-computed knowledge base, eliminating the need for repeated training and significantly reducing deployment costs.

[0103] Based on the same inventive concept, this application also provides a small sample action recognition device for implementing the small sample action recognition method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more small sample action recognition device embodiments provided below can be found in the limitations of the small sample action recognition method above, and will not be repeated here.

[0104] In one exemplary embodiment, a small sample action recognition device is provided, comprising:

[0105] A dataset construction module is used to construct a support set and a query set; the support set includes N×K first video samples, and the query set includes L second video samples; where N is the number of action types, K is the number of second video samples for each action type, and L is the number of second video samples.

[0106] The sample stitching module is used to combine each second video sample with all the first video samples in the support set to obtain L sample combinations.

[0107] The model training module is used to train the video encoder model using L sample combinations to obtain the trained video encoder as an action recognition model.

[0108] The video combination module is used to combine the video to be identified with all the first video samples in the support set to obtain the video combination to be identified;

[0109] The action recognition module is used to input the combined video to be recognized into the action recognition model to obtain the action type corresponding to the video to be recognized.

[0110] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a small-sample action recognition method.

[0111] Those skilled in the art will understand that Figure 6 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0112] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0114] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0115] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for recognizing actions in a small sample size, characterized in that, The few-sample action recognition method includes the following steps: Construct a support set and a query set; the support set includes The query set includes L first video samples and L second video samples; where N is the number of action types, K is the number of first video samples for each action type, and L is the number of second video samples. Each second video sample is combined with all the first video samples in the support set to obtain L sample combinations; The video encoder model is trained using L combinations of samples to obtain the trained video encoder as an action recognition model. The video to be identified is combined with all the first video samples in the support set to obtain the video combination to be identified; The video to be identified is combined and input into the action recognition model to obtain the action type corresponding to the video to be identified; The video encoder model includes: a visual encoder, an inter-frame temporal modeling module, an inter-video semantic association module, an inter-task knowledge transfer module, and an OTAM calculation module; The visual encoder is used to extract features from each video frame in the first video sample and the second video sample in each sample combination to obtain frame-level features. The inter-frame temporal modeling module is used to perform temporal dependency modeling on each frame-level feature of the first video sample and the second video sample in each sample combination, obtain the temporal features of the first video sample and the second video sample in each sample combination, and then splice them together to obtain the joint features of each sample combination. The video semantic association module is used to perform attention feature enhancement on the joint features of each sample combination to obtain the enhanced features of each sample combination. The inter-task knowledge transfer module is used to perform inter-task association enhancement on the primary enhanced features of each sample combination to obtain secondary enhanced features for each sample combination. The OTAM calculation module is used to perform OTAM calculation on the secondary enhanced features of each sample combination to determine the predicted action category corresponding to the second video sample in each sample combination.

2. The small sample action recognition method according to claim 1, characterized in that, In terms of enhancing the joint features of each sample combination with attention features to obtain the enhanced features of each sample combination, the inter-video semantic association module is specifically used for: By constructing a binary mask matrix and employing a multi-head self-attention mechanism to enhance the joint features, a first-order enhanced feature is obtained, as shown in the following formula: ； ； in, This is the attention weight matrix after binary masking. , , These are the query matrix, key matrix, and value matrix, respectively. , , All are constructed based on joint features, with the superscript T indicating transpose. This represents the feature dimension of each attention head. For one enhancement feature, It is a binary mask matrix. The elements in the representation indicate whether the features between video frames can interact.

3. The small sample action recognition method according to claim 1, characterized in that, The inter-task knowledge transfer module includes a prototype calculation module, a historical prototype query module, and a prototype enhancement module. The prototype calculation module is used to calculate the prototype features of the first enhancement feature, which serves as the current prototype; The historical prototype query module is used to find multiple historical prototypes similar to the current prototype from the knowledge base, and to perform a weighted summation of the multiple historical prototypes obtained from the query to obtain the historical knowledge prototype. The prototype enhancement module is used to enhance the first-stage enhanced features based on the historical knowledge prototype to obtain the second-stage enhanced features.

4. The small sample action recognition method according to claim 3, characterized in that, The formula for enhancing the first-order feature based on the historical knowledge prototype is as follows: ； ； in, This is a secondary enhancement feature. For one enhancement feature, To enhance the prototype, For the current prototype, As a prototype of historical knowledge, 、 、 These are the linear projection matrices of the query matrix, key matrix, and value matrix, respectively.

5. The small sample action recognition method according to claim 3, characterized in that, The inter-task knowledge transfer module also includes a knowledge base update module; The knowledge base update module is used to directly add the current prototype to the knowledge base when the number of historical prototypes in the knowledge base is less than a preset threshold, and to replace the historical prototype with the highest similarity to the current prototype in the knowledge base with the current prototype when the number of historical prototypes in the knowledge base reaches the preset threshold.

6. The small sample action recognition method according to claim 1, characterized in that, The video encoder model is trained using a combination of L samples. The loss function used in the process of obtaining the trained video encoder as an action recognition model is the cross-entropy loss function.

7. A small sample action recognition device, characterized in that, The small sample action recognition device applies the small sample action recognition method according to any one of claims 1-6, and the small sample action recognition device includes: The dataset construction module is used to build the support set and query set; the support set includes... The query set includes L first video samples and L second video samples; where N is the number of action types, K is the number of first video samples for each action type, and L is the number of second video samples. The sample stitching module is used to combine each second video sample with all the first video samples in the support set to obtain L sample combinations. The model training module is used to train the video encoder model using L sample combinations to obtain the trained video encoder as an action recognition model. The video combination module is used to combine the video to be identified with all the first video samples in the support set to obtain the video combination to be identified; The action recognition module is used to input the combined video to be recognized into the action recognition model to obtain the action type corresponding to the video to be recognized.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the small sample action recognition method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the small sample action recognition method according to any one of claims 1-6.

Images