Progressive alternative decoupling small sample behavior recognition method and system
By employing a progressively decoupled few-shot action recognition method, and utilizing a dual-path architecture and cross-set attention interaction, the problem of insufficient spatiotemporal feature modeling in few-shot action recognition is solved, achieving higher accuracy and robust action recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINGDAO UNIV OF SCI & TECH
- Filing Date
- 2025-12-23
- Publication Date
- 2026-07-24
AI Technical Summary
Existing methods for behavior recognition with few samples lack sufficient spatiotemporal feature modeling and have weak prototype discrimination power, making it difficult to achieve accurate recognition under conditions of very few samples.
A progressively alternating decoupled few-sample behavior recognition method is adopted. Visual features are processed along the "spatial first, temporal second" and "temporal first, spatial third" paths respectively through a dual-path architecture. Combined with a dynamic sensitive differential aggregator, a diffuse long temporal adapter, and bidirectional cross-set attention interaction, discriminative coupled spatiotemporal representations are generated.
It improves the accuracy and robustness of behavior recognition in small samples, effectively resists changes in perspective and background interference, generates more discriminative coupled spatiotemporal representations, and improves recognition accuracy under conditions of very few samples.
Smart Images
Figure CN121746798B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of few-sample recognition technology, specifically a method and system for recognizing few-sample behaviors using a progressively alternating decoupling approach. Background Technology
[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.
[0003] Few-Shot Recognition is a type of task in machine learning. Its core objective is to enable the model to accurately identify unknown new samples when there are only a very small number of labeled samples for each target class. Its key difference from conventional recognition tasks lies in its extremely low requirement for labeled data, thus addressing the bottleneck problem of scarce training data caused by difficulties in data collection and high labeling costs in various scenarios.
[0004] In the specific application scenario of behavior recognition, few-shot recognition specifically refers to the ability of a model to learn to recognize similar actions in unknown videos using only a few labeled action video examples. For example, human actions in videos, such as running, waving, and playing ball, can occur in any size frame. However, for a new action category (such as "practicing Tai Chi"), if only a very small number of example videos are provided (say, 5), the model needs to be able to identify which actions are "practicing Tai Chi" from a large number of videos. This type of problem is called "few-shot behavior recognition."
[0005] Mainstream few-shot behavior recognition methods primarily follow a meta-learning paradigm based on metric learning. In this paradigm, the model first undergoes meta-training on a base dataset containing a large number of categories to learn a general deep feature embedding space. Then, when faced with a novel N-way K-shot task (i.e., containing N new categories, with only K supporting samples for each category), the model predicts the category affiliation of the query sample by calculating the distance between the features of the query sample and the prototype supporting samples of each category (usually the mean of the features of samples of the same category) in this embedding space.
[0006] Most existing methods follow a single modeling paradigm of "space first, time later," that is, first using convolutional neural networks to extract the spatial appearance features of each frame of image, and then aggregating temporal information through mechanisms such as temporal pooling, recurrent networks, or 3D convolution. This serial and fixed processing approach makes it difficult to fully capture the complex and complementary interaction between spatial structural information and multi-level temporal dynamic information in human actions. Moreover, the fine-grained categories of actions often depend on both key poses (spatial) and specific motion patterns (temporal). Existing methods are insufficient in modeling this joint dependency, resulting in weak discriminative power and poor generalization ability of video feature representations learned from very few samples. Summary of the Invention
[0007] This invention provides a progressively alternating decoupled method and system for small-sample behavior recognition, which solves the problems of insufficient spatiotemporal feature modeling and weak prototype discrimination under small-sample conditions, thereby reducing the dependence on a large amount of labeled data and improving the recognition accuracy and robustness of the model in data-scarce scenarios.
[0008] The first aspect of this invention discloses a method for identifying few-sample behaviors using a progressively alternating decoupling approach, comprising the following steps: Obtain the video samples to be identified and use them as the query set. Use the support set, which contains N categories and K labeled video samples in each category, where N and K are positive integers. Visual features of all video samples in the support set and query set are extracted, and spatiotemporal features are decoupled and fused along the first path and the second path, respectively. In the first path, the visual features are first subjected to spatial context enhancement processing, and then the short-time and long-time motion features are fused to obtain the first coupled feature; In the second path, visual features are first fused with short-time and long-time motion features, and then spatial context enhancement is performed to obtain the second coupled feature. The first coupling feature and the second coupling feature are fused to obtain the coupled spatiotemporal representation of the corresponding video sample. Further, through bidirectional cross-set attention interaction, the enhanced representation corresponding to the support set and the query set is obtained. The enhanced representation is constructed by subsequence and distance metric to calculate the distance between the video sample of the query set and the video samples of each category in the support set, and the category of each video sample to be identified in the query set is determined based on the obtained distance.
[0009] Furthermore, the fusion of short-time and long-time motion features is achieved through a "dynamically sensitive differential aggregator" and a "diffusion-heuristic long-time adapter," including the following steps: Short-term motion features of visual features are extracted through differential computation and gating mechanisms; Long-term temporal motion features of visual features are extracted using dilated convolution and temporal attention mechanisms; The short-time motion features and long-time motion features are weighted and summed according to learnable weights to obtain the temporal fusion features.
[0010] Furthermore, the spatial context enhancement processing steps are implemented through the "progressive spatial optimization module," including the following steps: Large-scale local spatial features in visual features are extracted using depthwise separable convolution. Two consecutive 1×1 convolutions are used to expand and restore the channels of local spatial features to improve representation capabilities; Channel attention and spatial attention are applied sequentially to the restored features to extract spatial contextual information; The attention-processed features are residually connected to the input features to preserve the original spatial details, resulting in enhanced spatial features.
[0011] Furthermore, the spatial context enhancement process is executed multiple times in a stacked manner to enhance the spatial representation capability in an iterative way.
[0012] Furthermore, short-term motion features are extracted through differential computation and gating mechanisms, specifically: The inter-frame motion feature sequences are passed through a gated network containing layer normalization and linear layers to generate a weight mask; The weight mask is combined with the original features using the Hadamard product, and further enhanced by the channel attention mechanism to obtain short-term motion features.
[0013] Furthermore, long-term temporal motion features are extracted through dilated convolution and temporal attention mechanisms, specifically: Multi-scale temporal modeling of visual features is performed using depthwise separable convolutions with increasing dilation factors. The features output by the convolution are input into the temporal Transformer block, and the multi-head attention mechanism is used to capture long-distance temporal dependencies. The output of the Transformer block is subjected to one-dimensional convolution and layer normalization to generate long-term temporal motion features.
[0014] Furthermore, the bidirectional cross-set attention interaction step is implemented through a "bidirectional cross-attention module," which includes the following steps: The spatiotemporal representations of the support set and query set are reconstructed into token sequences, and the most discriminative key tokens are selected based on attention scores. Using the filtered support set key tokens and query set key tokens as input, perform cross-attention calculations for "support to query" and "query to support" respectively; The results of cross-attention output are processed by layer normalization and feedforward network to generate enhanced representations corresponding to the support set and query set.
[0015] A second aspect of the present invention discloses a small sample behavior recognition system with progressive alternating decoupling, comprising: The data input module is configured to: acquire video samples to be identified and use them as a query set, and use a support set containing N categories, each category containing K labeled video samples, where N and K are positive integers; The feature acquisition and enhancement module is configured to: extract the visual features of all video samples in the support set and query set, and perform spatiotemporal feature decoupling and fusion along the first path and the second path respectively; In the first path, the visual features are first subjected to spatial context enhancement processing, and then short-time and long-time motion features are fused to obtain the first coupled feature; In the second path, visual features are first fused with short-time and long-time motion features, and then spatial context enhancement is performed to obtain the second coupled feature; The fusion output module is configured as follows: the first coupling feature and the second coupling feature are fused to obtain the coupled spatiotemporal representation corresponding to the corresponding video sample. Further, through bidirectional cross-set attention interaction, the enhanced representation corresponding to the support set and the query set is obtained. The enhanced representation is constructed by subsequence and distance metric to calculate the distance between the video sample of the query set and the video samples of each category in the support set. Based on the obtained distance, the category of each video sample to be identified in the query set is determined.
[0016] A third aspect of the present invention discloses a computer program product including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the aforementioned progressively alternating decoupled few-sample behavior recognition method.
[0017] A fourth aspect of the present invention discloses an electronic device including at least one processor and a memory connected to the processor, the memory being used to store a computer program; the processor being used to execute the computer program, enabling the electronic device to implement the above-described progressively alternating decoupled small sample behavior recognition method.
[0018] Compared with existing technologies, one or more of the above technical solutions have the following beneficial effects: 1. An alternating decoupled dual-path modeling architecture was designed. By processing the same visual feature in parallel along two paths—"spatial first, then temporal" and "temporal first, then spatial"—the rigid pattern of a single processing order is broken. This allows the model to jointly explore the deep dependencies between key poses (spatial) and motion patterns (temporal) in the video from two complementary perspectives. The features output from the two paths are then fused, forcing the model to simultaneously focus on spatial details and the dynamic changes at multiple temporal scales (short-term motion and long-term patterns), thereby generating a more discriminative and comprehensive coupled spatiotemporal representation. This makes the feature representations learned from a very small number of samples more resistant to noise such as viewpoint changes and background interference, improving the recognition accuracy of behaviors with few samples.
[0019] 2. When fusing temporal features, a complementary modeling approach of "short-time" and "long-time" methods is combined. Specifically, subtle, local motion differences between frames are explicitly captured through differential computation and gating mechanisms (short-time), while dilated convolution and temporal Transformers are used to capture global temporal dependencies and evolution patterns spanning multiple time steps (long-time). These two types of features are adaptively fused through a learnable weight, allowing the model to dynamically balance the importance of instantaneous changes and long-term patterns based on the characteristics of the action itself. This mechanism effectively addresses the temporal diversity of different action categories, enabling the model to achieve stable and reliable recognition even in complex scenarios with varying action execution speeds and cycle lengths.
[0020] 3. After obtaining the coupled spatiotemporal representation, a bidirectional cross-set attention interaction mechanism is introduced. This mechanism first selects the most discriminative key feature fragments from the support set and query set, and then performs bidirectional attention interaction between them (support → query, query → support). This process allows query samples to actively "refer" to the discriminative context of the support set to optimize their own expression, while also allowing the support set prototype to be fine-tuned according to the characteristics of the query sample, thereby generating a more targeted and robust enhanced representation. Finally, a fine-grained measurement strategy based on subsequences further enhances the accuracy of matching. This method effectively alleviates the prototype noise problem caused by the extremely small number of support samples (e.g., 1-shot), improves the reliability of the measurement process, and makes the final category determination more accurate. Attached Figure Description
[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0022] Figure 1 A schematic diagram of a small sample behavior recognition process with progressive alternating decoupling provided for one or more embodiments of the present invention; Figure 2 A schematic diagram of the architecture of a progressively alternating decoupled few-sample behavior recognition method provided in one or more embodiments of the present invention; Figure 3 A schematic diagram of a progressive space optimization module provided in one or more embodiments of the present invention; Figure 4 A schematic diagram of a dynamic sensitive differential aggregator provided in one or more embodiments of the present invention; Figure 5 A schematic diagram of a diffusion-heuristic long-time adapter provided for one or more embodiments of the present invention. Detailed Implementation
[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0025] As introduced in the background section, mainstream few-shot behavior recognition methods mainly follow a meta-learning paradigm based on metric learning. In this paradigm, the model is first meta-trained on a basic dataset containing a large number of categories to learn a general deep feature embedding space. Then, when faced with a brand-new N-way K-shot task (i.e., containing N new categories, with only K supporting samples for each category), the model predicts the category affiliation of the query sample by calculating the distance between the features of the query sample and the prototype of the supporting samples of each category (usually the mean of the features of samples of the same category) in this embedding space.
[0026] Most existing few-sample behavior recognition methods follow a single modeling paradigm of "space first, time later," that is, first using convolutional neural networks to extract spatial appearance features of each frame of image, and then aggregating temporal information through mechanisms such as temporal pooling, recurrent networks, or 3D convolution. This serial and fixed processing method is difficult to fully capture the complex and complementary interaction between spatial structural information and multi-level temporal dynamic information in human actions. Moreover, the fine category of actions often depends on both key poses (spatial) and specific motion patterns (temporal). Existing methods are insufficient in modeling this joint dependency, resulting in weak discriminative power and poor generalization ability of video feature representations learned from very few samples.
[0027] After obtaining video features, existing methods typically generate a prototype representation for each category through simple averaging, and use this as the benchmark for measurement. In cases of extremely scarce samples (e.g., 1-shot), noise or bias from a single support sample can directly contaminate the entire class prototype. Simultaneously, the rich contextual relationships between query and support samples are not effectively mined, resulting in insufficient information in the final matching measurement process. This raises the extension question: how can we more robustly construct class prototypes and enhance the information interaction between the support and query sets during the measurement phase to improve the discriminativeness and reliability of the measurement?
[0028] In summary, existing few-shot behavior recognition technologies have fundamental limitations in the sufficiency and flexibility of spatiotemporal feature modeling, and are insufficient in prototype construction and measurement. This solution provides a progressively alternating decoupled few-shot behavior recognition method and system. By designing a dual-path alternating decoupling architecture of "space → time" and "time → space," it utilizes progressive spatial optimization, differential short-time series aggregation, and diffusion-type long-time series adaptation modules to achieve multi-level complementary enhancement of spatiotemporal features. Then, a discriminative prototype is generated through bidirectional cross-attention interaction, and finally, accurate recognition is achieved through subsequence measurement.
[0029] like Figure 1 As shown, the progressively alternating decoupling few-sample behavior recognition method includes the following steps: Obtain the video samples to be identified and use them as the query set. Use the support set, which contains N categories and K labeled video samples in each category, where N and K are positive integers. Visual features of all video samples in the support set and query set are extracted, and spatiotemporal features are decoupled and fused along the first path and the second path, respectively. In the first path, the visual features are first subjected to spatial context enhancement processing, and then the short-time and long-time motion features are fused to obtain the first coupled feature; In the second path, visual features are first fused with short-time and long-time motion features, and then spatial context enhancement is performed to obtain the second coupled feature. The first coupling feature and the second coupling feature are fused to obtain the coupled spatiotemporal representation of the corresponding video sample. Further, through bidirectional cross-set attention interaction, the enhanced representation corresponding to the support set and the query set is obtained. The enhanced representation is constructed by subsequence and distance metric to calculate the distance between the video sample of the query set and the video samples of each category in the support set, and the category of each video sample to be identified in the query set is determined based on the obtained distance.
[0030] This example provides a progressively alternating decoupled few-shot behavior recognition method inspired by the diffusion model. It obtains video samples containing the video to be recognized, constructs a support set and a query set based on the few-shot learning task, and inputs the video samples of the support set and the video to be recognized into the trained behavior recognition model to obtain the query category prediction result. The trained behavior recognition model extracts features from both the support set and query set videos to obtain the visual features of each video. It then performs spatial-temporal and temporal-spatial modeling on the visual features of each video to obtain coupled spatial-temporal and coupled temporal-spatial features. A cross-attention operation is then performed on the coupled features of the support set and the query set to obtain the final fused features of the support set and the query set. Finally, the final fused features of the support set and the query set are used to construct subsequences, and the distance between the query set subsequences and the support set class subsequences is calculated to obtain the final category prediction result for the query set videos.
[0031] The overall architecture of this solution is as follows: Figure 2 As shown, for a few-shot recognition task constructed from video samples in the support set and query set, the following steps are taken: First, visual features are extracted using a visual encoder with shared parameters. These features are then input into a decoupled spatial-temporal modeling branch and a decoupled temporal-spatial modeling branch, respectively. These branches output coupled spatial-temporal features and coupled temporal-spatial features, respectively. Next, the coupled spatial-temporal and coupled temporal-spatial features of the support set and query set are input into a bidirectional cross-attention module to generate the final fused features of the support set and query set. Finally, the fused features of the support set and query set are input into a CrossTransformer-based metric module to predict the category of each query set sample.
[0032] The training process for an action recognition model includes: Construct a dataset consisting of video frame images with known category labels; The training set is input into the behavior recognition model to train the model. In each round of training, a small sample learning task is constructed from the training set. The model predicts the label of the query set by learning the knowledge of the support set and updates the parameters according to the prediction loss. When the loss function of the model no longer decreases, or when the number of iterations reaches the set number, training stops, and the trained behavior recognition model is obtained.
[0033] The training set is a video dataset, which includes several videos, each containing several video frames. The few-sample behavior recognition method in this scheme uses a sparse temporal sampling strategy to sample video frame sequences as output, and outputs the classification result of the video to be recognized.
[0034] "Few-shot learning task" refers to sampling a few samples from a given dataset. -way -shot task, which consists of a support set and a query set; Support set includes There are 10 categories, each with 10 categories. One video sample; The query set consists of several unlabeled video samples from the same category as the support set; The model learns knowledge from the support set to predict the labels of the query set and updates its parameters based on the prediction loss.
[0035] During the training phase, an episode is sampled from the training set. An episode refers to a small training unit that simulates a single test task. An episode is constructed according to the following process: (1) Randomly sample N categories from the training set; (2) Sample K samples from each category as support; (3) Sample Q queries for each category, and these queries do not overlap with the support queries. Here, Q is usually set to 5, 10, or 15.
[0036] Once the behavior recognition model is trained, determine whether user input is required for the support set based on the categories to be recognized. (1) When the categories are fixed: For example, if 20 action categories are identified online, the categories remain unchanged. In this case, the support set can be prepared and fixed in advance by the system: directly store several representative video features of each class, or store the "prototype" (class center vector) of each class, or store a small support library, from which the nearest neighbor is selected online. In this case, the user does not need to input the support set, but only needs to input the query; (2) When the category is not fixed: For example, if the user wants the system to temporarily recognize "a new action" that the system has not seen before, then the user needs to provide a support set.
[0037] The sparse temporal sampling strategy is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Randomly divided into The system uses 8 video clips (the default setting is 8 clips), and then randomly selects one frame from each video clip to construct a video frame sequence. .
[0038] The trained behavior recognition model, such as Figure 2 As shown, it includes: A shared-parameter visual encoder takes a sequence of video frames as input to a support set and a query set, and outputs the visual features of the corresponding samples. ,in, Indicates batch size, and Representing the time dimension and feature channels, and Corresponding spatial dimensions; visual features of each video sample The input is fed into the input terminal of the alternating decoupling module; The output of the alternating decoupling module is connected to the input of the bidirectional cross-attention module; The output of the bidirectional cross-attention module is the final fused feature of the support set and the query set; the final fused feature of the support set and the query set is measured, and the prediction result of the query sample is output.
[0039] like Figure 2 As shown, the alternating decoupling module has two parallel branches: a decoupling spatial-temporal modeling branch and a decoupling temporal-spatial modeling branch; both branches include a progressive spatial optimization module and two parallel modules, namely a dynamic sensitive differential aggregator and a diffusion-heuristic long-time adapter. In the decoupled spatial-temporal modeling branch, features from the support set and query set are first input into the progressive spatial optimization module. The output of the progressive spatial optimization module is connected to the input of the dynamic sensitive difference aggregator and the diffusion heuristic long temporal adapter, respectively. The outputs of the two parallel modules are weighted and summed to obtain the coupled spatial-temporal features of the video samples. In the decoupled temporal-spatial modeling branch, features from the support set and query set are input in parallel to the dynamic sensitive difference aggregator and the diffusion heuristic long temporal adapter; the outputs of the two parallel modules are weighted and summed before being input to the progressive spatial optimization module; the output of the progressive spatial optimization module outputs the coupled temporal-spatial features of the video samples. The outputs of the two branches are summed element by element to output the coupling features of each video sample.
[0040] The formula for weighted summation is: ; in, This represents the final long-short time series fusion characteristics. This represents the learnable weights.
[0041] like Figure 3 As shown, the progressive spatial optimization module inputs the visual features of each sample into a kernel with a size of [missing information]. The depthwise separable convolution captures large-scale local spatial features and applies layer normalization in the channel dimension to eliminate distribution differences between channels; Next, use two consecutive Convolution and a GELU activation function to generate spatial features The first convolution expands the channel dimension by four times to enhance feature representation capabilities, while the second convolution maps features back to the original dimension, enabling the network to retain low-level semantic information. Then, spatial context information is further mined using a spatial channel attention module, which consists of a channel attention block and a spatial attention block. In the channel attention block, the input features are first subjected to global average pooling and global max pooling operations, respectively. The processed features are then input into a multilayer perceptron (MLP) with shared parameters to generate two different channel features. Finally, the fused channel-weighted features are obtained through element-wise addition.
[0042] The processing procedure for channel attention blocks can be described as follows: ; in, and These represent global average pooling and global max pooling, respectively. and Indicates learnable weights, This represents the sigmoid activation function.
[0043] In the spatial attention block, the fused channel features are first subjected to global averaging and global maximizing operations respectively, then a concatenation operation is performed to generate fused spatial features, and finally a kernel size of [missing value]. A convolutional layer and a sigmoid activation function generate the final spatial features.
[0044] The processing procedure for spatial attention blocks can be described as follows: ; in, and These represent the concatenation and convolution operations, respectively. and These represent the global average operation and the global maximum operation, respectively.
[0045] To preserve original spatial details, the output of spatial attention blocks... The final spatial features will be obtained through a residual operation. .
[0046] To enhance feature discrimination while preserving the original input information, this method uses stacked progressive spatial optimization modules to iteratively enhance spatial representation. This embodiment stacks three layers, with the output of the current layer serving as the input to the next layer.
[0047] like Figure 4 As shown, the Dynamic Sensitive Difference Aggregator inputs features into a fully connected layer to generate query, key, and value features; and then reconstructs the dimensions of the query and key features before inputting them into a kernel with a kernel size of [missing information]. Local spatial neighborhood relationships are explicitly modeled in the convolutional layers.
[0048] Next, feature-level action difference calculation is performed on two adjacent frames in the time dimension. The calculation formula is as follows: ; in, Indicates time step The motion characteristics, and These represent query features and key features, respectively. and This indicates a 2D convolution operation, setting the time step. Motion characteristics To maintain consistency with the original dimensions of the query features.
[0049] By concatenating all the obtained motion features along the time dimension, the final representation is obtained. ,in This represents a splicing operation, and a gating mechanism consisting of two linear layers, two activation functions, and a layer normalization is used to enhance salient regions in the motion features, resulting in enhanced motion features.
[0050] Calculate the Hadamard product between enhanced motion features and value features to generate hybrid gated features. The process can be described as follows: ; in, and Indicates learnable weights, Representation layer normalization, It represents the Hadamardi (or Hadama) stack.
[0051] right Channel attention is applied and channel visual features are generated through Hadamard product operations. The process can be described as follows: ; ; in, and Represents different 3D convolution operations. This indicates a 3D adaptive average pooling operation.
[0052] Finally, a 1D convolution operation is used in the time dimension to enhance the model's short-temporal modeling capability, generating the final short-temporal features. .
[0053] like Figure 5 As shown, the diffusion-heuristic long-time adapter inputs features into a holed temporal diffusion unit, which first uses a 1D depthwise separable convolution with progressively increasing hole factors to capture long-time patterns.
[0054] Then, batch normalization and the GELU activation function are used to enable the model to learn complex temporal features, and residual connections are used to ensure efficient gradient flow. This process can be described as follows: ; in, and These represent batch normalization and 1D convolution operations, respectively.
[0055] After passing through the void time-series diffusion unit The data is fed into a temporal Transformer block, which utilizes multi-head attention and a feedforward network to further mine long-term temporal features. This process can be described as follows: ; ; in, This indicates the attention of the bulls. This represents a feedforward network.
[0056] Finally, 1D convolution operations and layer normalization are used to preserve the original spatial features and generate the final long-term temporal features. .
[0057] The bidirectional cross-attention module includes: First, the input support set and query set are coupled with spatial-temporal features and coupled temporal-spatial features element by element, and then reconstructed into a tokens sequence (the input tensor is transformed into the shape required by the attention module, i.e. [B,L,C]). Next, each token is mapped to a scalar score through a linear layer, and the most discriminative one is selected in the time dimension. This design, which uses a number of tokens, can effectively reduce redundancy and emphasize key regions, thereby enabling the model to prioritize key features and suppress irrelevant information. Then, the supporting and query features are input into a bidirectional multi-head attention mechanism to exchange information. Specifically, the query tokens will focus on the filtered top... Each set of support tokens allows it to focus on discriminative contextual information from the support set. Finally, the output of bidirectional multi-head attention is processed by a layer normalization and a feedforward network to obtain the final support for the query features.
[0058] The cross-attention process supporting queries can be described as follows: ; ; in, and These represent the query features and the selected top ones, respectively. Each set of tokens has a support set. The same operation is performed on the cross-attention function to find supporting tokens. This yields the final fused features of the support set and the query set.
[0059] Supports metrics for sets and query sets, including: First, construct a tuple of all possible support sets of video frame sequences (for simplicity, we select...). Each tuple can be represented as , and Indicates the first and the Each frame is represented. Indicates the number of frames that support centralized sampling; Next, the distance between the query subsequence and the support class subsequence is calculated using the CrossTransformer-based metric module. This process can be described as follows: ; in, Indicates the query subsequence features. Indicates belonging to a category Support for subsequence features This represents all possible supporting video subsequence features. and These represent the query video and the support set, respectively; the entire training process uses standard cross-entropy loss to guide model optimization.
[0060] To address the fundamental problem of insufficient spatiotemporal information interaction and weak feature discrimination caused by the single sequential paradigm of "space first, time later" in existing methods, this solution designs an alternating decoupled dual-path modeling architecture. By processing the same visual feature along two paths, "space first, time later" and "time later, space later," the rigid pattern of a single processing order is broken. This allows the model to jointly explore the deep dependencies between key poses (spatial) and motion patterns (temporal) in the video from two complementary perspectives. The features output from the two paths are then fused, forcing the model to simultaneously focus on spatial details and the dynamic changes at multiple temporal scales (short-term motion and long-term patterns), thereby generating a more discriminative and comprehensive coupled spatiotemporal representation. This makes the feature representations learned by the model from a very small number of samples more resistant to noise such as viewpoint changes and background interference, improving the recognition accuracy of behaviors in small samples.
[0061] Traditional methods, relying on single-scale temporal modeling (such as 3D convolution), struggle to simultaneously capture subtle instantaneous motion changes and macroscopic motion rhythms. This approach combines two complementary modeling methods—"short-time" and "long-time"—when fusing temporal features. Specifically, it explicitly captures fine, localized motion differences between frames (short-time) through differential computation and gating mechanisms, while simultaneously using dilated convolution and a temporal Transformer to capture global temporal dependencies and evolution patterns spanning multiple time steps (long-time). These two feature sets are adaptively fused using a learnable weight, allowing the model to dynamically balance the importance of instantaneous changes and long-term patterns based on the characteristics of the action itself. This mechanism effectively addresses the temporal diversity of different action categories, enabling the model to achieve stable and reliable recognition even in complex scenarios with varying action execution speeds and durations.
[0062] In existing methods, prototypes are typically generated through simple averaging, which is susceptible to noise from individual samples in the support set and has limited interaction with query samples. This approach introduces a bidirectional cross-set attention interaction mechanism after obtaining the coupled spatiotemporal representation. This mechanism first selects the most discriminative key feature fragments from the support and query sets, and then performs bidirectional attention interaction between them (support → query, query → support). This process allows query samples to actively "refer" to the discriminative context of the support set to optimize their own representation, while also allowing the support set prototypes to be fine-tuned according to the characteristics of the query samples, thereby generating more targeted and robust enhanced representations. Finally, a fine-grained measurement strategy based on subsequences further enhances the accuracy of matching. This method effectively alleviates the prototype noise problem caused by a very small number of support samples (e.g., 1-shot), improves the reliability of the measurement process, and makes the final category determination more accurate.
[0063] Correspondingly, a few-sample behavior recognition system with progressive alternating decoupling includes: The data input module is configured to: acquire video samples to be identified and use them as a query set, and use a support set containing N categories, each category containing K labeled video samples, where N and K are positive integers; The feature acquisition and enhancement module is configured to: extract the visual features of all video samples in the support set and query set, and perform spatiotemporal feature decoupling and fusion along the first path and the second path respectively; In the first path, the visual features are first subjected to spatial context enhancement processing, and then short-time and long-time motion features are fused to obtain the first coupled feature; In the second path, visual features are first fused with short-time and long-time motion features, and then spatial context enhancement is performed to obtain the second coupled feature; The fusion output module is configured as follows: the first coupling feature and the second coupling feature are fused to obtain the coupled spatiotemporal representation corresponding to the corresponding video sample. Further, through bidirectional cross-set attention interaction, the enhanced representation corresponding to the support set and the query set is obtained. The enhanced representation is constructed by subsequence and distance metric to calculate the distance between the video sample of the query set and the video samples of each category in the support set. Based on the obtained distance, the category of each video sample to be identified in the query set is determined.
[0064] Correspondingly, a computer program product includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the aforementioned progressively alternating decoupled few-sample behavior recognition method.
[0065] Accordingly, an electronic device includes at least one processor and a memory connected to the processor, the memory being used to store a computer program; the processor is used to execute the computer program, enabling the electronic device to implement the aforementioned progressively alternating decoupled few-sample behavior recognition method.
[0066] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A progressively alternating decoupling method for identifying small sample behaviors, characterized in that, Includes the following steps: Obtain the video samples to be identified and use them as the query set; use the support set containing N categories, each containing K labeled video samples, where N and K are positive integers; Visual features of all video samples in the support set and query set are extracted, and spatiotemporal features are decoupled and fused along the first path and the second path, respectively. In the first path, the visual features are first subjected to spatial context enhancement processing, and then short-time and long-time motion features are fused to obtain the first coupled feature; In the second path, visual features are first fused with short-time and long-time motion features, and then spatial context enhancement is performed to obtain the second coupled feature; The first coupling feature and the second coupling feature are fused to obtain the coupled spatiotemporal representation of the corresponding video sample. Further, through bidirectional cross-set attention interaction, the enhanced representation corresponding to the support set and the query set is obtained. Enhanced representation is constructed by subsequence and distance metric, calculating the distance between video samples in the query set and video samples of each category in the support set, and determining the category of each video sample to be identified in the query set based on the obtained distance; The fusion of short-time and long-time motion features includes the following steps: Short-term motion features of visual features are extracted through differential computation and gating mechanisms; Long-term temporal motion features of visual features are extracted using dilated convolution and temporal attention mechanisms; The short-time motion features and long-time motion features are weighted and summed according to learnable weights to obtain the temporal fusion features; Specifically, short-term motion features are extracted through differential computation and gating mechanisms: The inter-frame motion feature sequences are passed through a gated network containing layer normalization and linear layers to generate a weight mask; The weight mask is combined with the original features by Hadamard product operation, and further enhanced by channel attention mechanism to obtain short temporal motion features; Specifically, long-term temporal motion features are extracted through dilated convolution and temporal attention mechanisms: Multi-scale temporal modeling of visual features is performed using depthwise separable convolutions with increasing dilation factors. The features output by the convolution are input into the temporal Transformer block, and the multi-head attention mechanism is used to capture long-distance temporal dependencies. The output of the Transformer block is subjected to one-dimensional convolution and layer normalization to generate long-term temporal motion features; The spatial context enhancement process includes the following steps: Large-scale local spatial features in visual features are extracted using depthwise separable convolution. Two consecutive 1×1 convolutions are used to expand and restore the channels of local spatial features to improve representation capabilities; Channel attention and spatial attention are applied sequentially to the restored features to extract spatial contextual information; The attention-processed features are residually connected to the input features to preserve the original spatial details, resulting in enhanced spatial features.
2. The small sample behavior recognition method with progressive alternating decoupling as described in claim 1, characterized in that, Spatial context enhancement processing is performed multiple times in a stacked manner to enhance spatial representation capabilities in an iterative way.
3. The small sample behavior recognition method with progressive alternating decoupling as described in claim 1, characterized in that, Bidirectional cross-set attention interaction includes the following steps: The spatiotemporal representations of the support set and query set are reconstructed into token sequences, and the most discriminative key tokens are selected based on attention scores. Using the filtered support set key tokens and query set key tokens as input, perform cross-attention calculations for "support to query" and "query to support" respectively; The results of cross-attention output are processed by layer normalization and feedforward network to generate enhanced representations corresponding to the support set and query set.
4. A progressively alternating decoupled few-sample behavior recognition system, used to implement the progressively alternating decoupled few-sample behavior recognition method as described in any one of claims 1-3, characterized in that, include: The data input module is configured to: acquire video samples to be identified and use them as a query set, and use a support set containing N categories, each category containing K labeled video samples, where N and K are positive integers; The feature acquisition and enhancement module is configured to: extract the visual features of all video samples in the support set and query set, and perform spatiotemporal feature decoupling and fusion along the first path and the second path respectively; In the first path, the visual features are first subjected to spatial context enhancement processing, and then short-time and long-time motion features are fused to obtain the first coupled feature; In the second path, visual features are first fused with short-time and long-time motion features, and then spatial context enhancement is performed to obtain the second coupled feature; The fusion output module is configured to: fuse the first coupling feature and the second coupling feature to obtain the coupled spatiotemporal representation corresponding to the corresponding video sample, and further obtain the enhanced representation corresponding to the support set and the query set through bidirectional cross-set attention interaction; Enhanced representation is constructed by subsequence and distance metric, calculating the distance between video samples in the query set and video samples of each category in the support set, and determining the category of each video sample to be identified in the query set based on the obtained distance.
5. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to perform the steps in the progressively alternating decoupled few-sample behavior recognition method as described in any one of claims 1-3.
6. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, the memory being used to store a computer program; the processor is used to execute the computer program, enabling the electronic device to perform the steps in the progressively alternating decoupled few-sample behavior recognition method as described in any one of claims 1-3.