Small sample behavior recognition method and system based on hierarchical representation learning network
By using multimodal feature fusion through hierarchical representation learning networks, the challenges of knowledge transfer and feature utilization between tasks in few-sample behavior recognition are addressed, improving recognition accuracy and robustness, achieving a balance between computational efficiency and feature richness, and providing a more practical solution.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QINGDAO UNIV OF SCI & TECH
- Filing Date
- 2026-02-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing few-sample behavior recognition methods face challenges in terms of knowledge transfer between tasks, utilization of spatiotemporal features, and data efficiency. In particular, under conditions of scarce data, the models struggle to learn robust visual representations and fail to effectively utilize semantic prior information.
A hierarchical representation learning network is adopted, which works collaboratively with spatial, temporal and textual branches to extract and fuse multimodal features, including spatial relationship enhancement, temporal aggregation alignment and semantic supervision, to generate fused features for few-sample behavior recognition.
It improves the model's recognition accuracy and robustness in data-scarce scenarios, achieves a balance between computational efficiency, feature richness, and generalization ability, overcomes the limitations of existing methods, and provides a more comprehensive and efficient solution for small-sample behavior recognition.
Smart Images
Figure CN122116477A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of few-shot recognition technology, specifically relating to a few-shot behavior recognition method and system based on hierarchical representation learning network. Background Technology
[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.
[0003] Few-shot recognition is a key task in machine learning, aiming to solve the recognition challenge in scenarios with scarce data. This task requires models to accurately identify unseen samples based on only a very small number of labeled samples for each class. Compared to traditional recognition tasks, its biggest advantage is a significantly reduced dependence on the amount of labeled data, making it particularly suitable for real-world scenarios where data acquisition is difficult or labeling costs are high.
[0004] In the field of action recognition, few-shot action recognition aims to enable models to accurately identify similar actions in unseen videos using a very small number of labeled video samples. Taking various sports actions (such as basketball, swimming, rock climbing, weightlifting, and fencing) as examples, even if the model only obtains a few examples (e.g., 3 videos of "figure skating jumps"), it still needs to correctly identify the action category from a large number of query videos. The core challenge of this task lies in how to learn a spatiotemporal motion representation with high discriminative power and good generalization ability from extremely limited supervision signals.
[0005] Current research on few-shot behavior recognition largely relies on metric-based meta-learning frameworks. These methods typically begin with meta-training on a base dataset containing a rich variety of categories to obtain a deep feature representation space with good transferability. During testing, the model is tasked with handling few-shot tasks consisting of N new categories, each containing only K labeled samples. In this process, the model performs classification by measuring the similarity between the query video representation and the category representations formed by the supporting samples of each category in the feature space. The category representations are generally obtained by statistically aggregating the features of supporting samples of the same category.
[0006] Existing few-shot behavior recognition methods still face numerous challenges in terms of inter-task knowledge transfer, spatiotemporal feature utilization, and data efficiency. On the one hand, these methods often neglect in-depth exploration of the intrinsic connections between meta-learning tasks, making it difficult to effectively transfer learned knowledge to new tasks. On the other hand, most of them focus on temporal dynamic information, especially relying on computationally expensive optical flow modeling, while failing to fully extract discriminative spatial structural features, thus limiting the model's ability to capture key behavioral information. This problem is particularly prominent under conditions of scarce data: models struggle to learn robust visual representations from a small number of samples, especially for certain difficult categories, where the learned features are often noisy and lack discriminative power. Furthermore, existing methods often fail to effectively utilize semantic prior information, while semantic knowledge can provide valuable contextual guidance for categories with insufficient visual information, compensating for the limitations of few-shot learning and helping to bridge the gap between heterogeneous visual and semantic feature spaces. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technologies and provide a few-shot behavior recognition method and system based on hierarchical representation learning networks. This invention effectively overcomes the problems of insufficient task generalization ability, imbalance in the utilization of spatiotemporal features, difficulty in representation learning under data scarcity, and failure to effectively coordinate multimodal information in existing few-shot behavior recognition methods. It reduces the dependence on a large amount of labeled data, improves the recognition accuracy and robustness of the model in data-scarce scenarios, and achieves an effective balance between computational efficiency, feature richness, and generalization ability. This invention provides a more comprehensive, efficient, and practical solution for few-shot behavior recognition.
[0008] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the technical solution of the present invention provides a few-shot behavior recognition method based on a hierarchical representation learning network, including: Obtain video samples to be identified to construct a query set, and obtain labeled video samples to construct a support set; Using video samples from the support set and query set as input, a pre-trained behavior recognition model is used to generate fusion features; based on the fusion features, video-level features are obtained, and query prototypes for the query set and support prototypes for each category in the support set are generated; the distance between the query prototype and the support prototype of each support category is calculated, and the final prediction result of the video sample to be identified is obtained and output.
[0009] In at least one embodiment, the behavior recognition model includes three parallel branches and a few-shot classifier. The three parallel branches are a spatial branch, a temporal branch, and a text branch, respectively; wherein the text branch is used only during the training phase.
[0010] In at least one embodiment, in the spatial branch, with video samples of the support set and query set as input, visual features are first extracted using a visual encoder, then spatial augmentation attention is introduced to reconstruct and enhance the visual features in the spatial dimension, then key information is aggregated through average pooling operation to generate average visual features of the video samples, and finally spatial augmentation is performed in the channel dimension to generate aggregated spatial features. In the temporal branch, using video samples from the support set and query set as input, visual features are first extracted using a visual encoder, then motion features are calculated, and finally temporal aggregation and alignment are performed to generate aggregated temporal features. The aggregated spatial features and aggregated temporal features are fused together by element-wise addition to generate fused features.
[0011] In at least one embodiment, in the text branch, taking the category labels of the annotated video samples in the support set as input, the CLIP text encoder is used to extract real semantic features, and the learning of fused features is supervised based on the real semantic features.
[0012] In at least one embodiment, a global average pooling operation on the temporal dimension is applied to the fused features to obtain video-level features; for the support set, the support prototype of each category is obtained by calculating the average vector of the support samples within the same category; for the query set, the aggregated vector of a single sample is used as its query prototype.
[0013] In at least one embodiment, spatial enhancement is specifically represented as:
[0014] In the formula, Indicates a space augmentation function; and These represent the average visual features of the supporting video samples and the query video samples, respectively. Indicates a splicing operation; This is the aggregated spatial feature output by the spatial enhancement function.
[0015] In at least one embodiment, motion feature calculation is specifically represented as follows:
[0016] In the formula, This represents a two-dimensional convolutional layer that performs feature transformations on each channel separately. Indicates the features of the input video frame; This indicates the number of video frames sampled for each video sample.
[0017] In at least one embodiment, time aggregation alignment is specifically represented as follows:
[0018] In the formula, This is the output of the time aggregation alignment module; Indicates attention operation; Indicates motion characteristics, This indicates the average motion characteristics.
[0019] In at least one embodiment, the distance between the query prototype and the supporting prototype of each supporting category is calculated, specifically expressed as:
[0020] In the formula, Represents the similarity function; Indicates a query sample in the query set. The query prototype; This indicates that support for concentration belongs to the action category. Support for prototypes of support samples; Indicates query sample Belongs to the action category The predicted probability; Indicates support for set number Support prototypes for each category For category indexing, .
[0021] Secondly, the technical solution of the present invention also provides a few-shot behavior recognition system based on a hierarchical representation learning network, comprising: The sample acquisition module is configured to: acquire video samples to be identified to construct a query set, and acquire labeled video samples to construct a support set. The behavior recognition module is configured to: take video samples from the support set and query set as input, generate fusion features using a pre-trained behavior recognition model; obtain video-level features based on the fusion features, and generate query prototypes for the query set and support prototypes for each category in the support set; calculate the distance between the query prototype and the support prototype of each support category, obtain the final prediction result for the video sample to be recognized, and output it.
[0022] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) This invention effectively overcomes the limitations of existing few-sample behavior recognition methods by constructing a multimodal recognition framework that coordinates spatial, temporal, and textual branches. While improving the model's recognition performance, it achieves an effective balance between computational efficiency, feature richness, and generalization ability, providing a more comprehensive, efficient, and practical solution for few-sample behavior recognition.
[0023] 2) The spatial relationship enhancement module introduced by the spatial branch of this invention can actively extract and fuse contextual information, thereby strengthening the category-specific spatial feature representation and making up for the shortcomings of existing works that neglect the importance of spatial structure due to excessive focus on temporal dynamics; the temporal branch models action dynamics by directly calculating the displacement between adjacent frames, avoiding the problems of high computational cost and low efficiency in small sample scenarios of traditional optical flow methods, and realizing efficient and lightweight capture of action temporal information; the text branch uses prior knowledge and contextual information learned from semantic information such as category labels to provide a powerful semantic supplement to the support set with insufficient visual information; by fusing semantic features, it not only alleviates the problem of high visual representation noise and weak discriminative power caused by data scarcity, but also bridges the heterogeneous gap between visual space and semantic space, especially when dealing with difficult categories, it can significantly improve the robustness of feature representation and discriminative quality. Attached Figure Description
[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0025] Fig. 1 This is a schematic diagram of the few-sample behavior recognition method based on hierarchical representation learning network disclosed in Embodiment 1 of the present invention; Fig. 2 This is a schematic diagram of the behavior recognition model disclosed in Embodiment 1 of the present invention; Fig. 3 This is a time branching diagram disclosed in Embodiment 1 of the present invention. Detailed Implementation
[0026] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0027] As described in the background section, the purpose of this invention is to overcome the shortcomings of the prior art and provide a few-shot behavior recognition method and system based on hierarchical representation learning networks. This effectively overcomes the limitations of existing few-shot behavior recognition methods, improves model recognition performance, and achieves an effective balance between computational efficiency, feature richness, and generalization ability, providing a more comprehensive, efficient, and practical solution for few-shot behavior recognition.
[0028] Example 1 In a typical embodiment of the present invention, such as Figs. 1 to 3As shown, this embodiment discloses a few-shot behavior recognition method based on a hierarchical representation learning network, which specifically includes the following steps: S1. Obtain video samples to be identified to construct a query set, and obtain labeled video samples to construct a support set; S2. Using video samples from the support set and query set as input, generate fusion features using a pre-trained behavior recognition model; obtain video-level features based on the fusion features, and generate query prototypes for the query set and support prototypes for each category in the support set; calculate the distance between the query prototype and the support prototype of each support category, obtain the final prediction result for the video sample to be identified, and output it.
[0029] The following describes in detail the few-sample behavior recognition method based on hierarchical representation learning network with specific implementation methods.
[0030] S1. Obtain video samples to be identified to construct a query set, and obtain labeled video samples to construct a support set.
[0031] In this step, a video sample to be identified is obtained and used as a query set, while simultaneously obtaining samples containing... Each category contains [number] categories, and each category contains [number] categories. The labeled video samples serve as the support set, among which... and A positive integer constitutes an episode task.
[0032] S2. Using video samples from the support set and query set as input, generate fusion features using a pre-trained behavior recognition model; obtain video-level features based on the fusion features, and generate query prototypes for the query set and support prototypes for each category in the support set; calculate the distance between the query prototype and the support prototype of each support category, obtain the final prediction result for the video sample to be identified, and output it.
[0033] S21. Feature extraction and aggregation are performed based on the behavior recognition model.
[0034] In this step, video samples from the query set and support set are used as input. The behavior recognition model includes three parallel branches and a few-sample classifier. The three parallel branches are the spatial branch, the temporal branch, and the text branch.
[0035] In the spatial branch, using video samples from the support set and query set as input, visual features are first extracted using a visual encoder. Then, spatial augmentation attention is introduced to reconstruct and enhance the visual features in the spatial dimension. Next, key information is aggregated through average pooling to generate the average visual features of the video samples. Finally, spatial augmentation is performed in the channel dimension to generate aggregated spatial features.
[0036] Specifically, in each episode task, a feature extractor (e.g., ResNet-50) is used to obtain the visual features of a video sequence. ,in, Indicates batch size, and These represent the time dimension and feature channels, respectively. and Corresponding to the spatial shape. Subsequently, average pooling is used to obtain the average visual features in the spatial dimension. To extract more discriminative spatial information, a spatial relationship enhancement module is used to aggregate video representations, specifically as follows:
[0037] In the formula, Indicates a space augmentation function; and These represent the average visual features of the supporting video samples and the query video samples, respectively. Indicates a splicing operation; This is the aggregated spatial features output by the spatial augmentation function. The spatial relationship augmentation module can mine the relationships between support samples and query samples using different methods.
[0038] In the temporal branch, using video samples from the support set and query set as input, visual features are first extracted using a visual encoder, then motion features are calculated, and finally temporal aggregation and alignment are performed to generate aggregated temporal features.
[0039] Specifically, in each episode task, a feature extractor is used to obtain the visual features of a video sequence. The temporal differences between adjacent frames can approximate action features. Specifically, given the features of input video frames... First, the transformed features are obtained through a two-dimensional convolutional transformation. Then, the motion representation is obtained by calculating the difference between the transformed features and the video frame features. Specifically, it is expressed as:
[0040] In the formula, This represents a two-dimensional convolutional layer that performs feature transformation on each channel; to transform motion features... With visual features Temporal dimension alignment, setting the motion representation at the end of the time series to zero, i.e. This yields the motion features of a video sequence. , This represents the number of video frames sampled for each video sample. Subsequently, the average action features are obtained through a global average pooling operation. .
[0041] To capture rich and discriminative temporal information in videos, this embodiment designs a temporal aggregation and alignment module. This module first uses three independent fully connected layers to average motion features. The data is projected into query, key, and value vectors, respectively. Then, a weighted feature is obtained through an attention operation: the pairwise similarity between different frames is measured by calculating the dot product of the query and key vectors. The similarity scores are then normalized using the Softmax function to obtain the final temporal attention weights. ; in the value vector and the final temporal attention weights Matrix multiplication is performed between the features to obtain weighted features; finally, the weighted features are concatenated with the residuals of the average action features to obtain aggregated time features. Specifically, it is expressed as:
[0042] In the formula, This is the output of the time aggregation alignment module; Attention operations representing temporal sequence; Indicates motion characteristics, This indicates the average motion characteristics.
[0043] The aggregated spatial features output from the spatial branch and the aggregated temporal features output from the temporal branch are fused together using element-wise addition to obtain the fused features. .
[0044] In the text branch, taking the category labels of the labeled video samples in the support set as input, we first use CLIP's text encoder to extract ground truth semantic features, and then use the ground truth semantic features to supervise the feature fusion. The learning.
[0045] Specifically, for the input category labels, CLIP's text encoder extracts true semantic features at the category level, and then averages and fuses these features over time. Obtain video features .
[0046] To improve the quality of the generated features, a semantic loss is introduced. The generated fused features are then regressed to near-true semantic features, specifically as follows:
[0047] In the formula, Indicates fusion characteristics; Represents true semantic features; This represents the square of the L2 norm. Using this loss function, the parameters of the visual encoder, spatial branch, and temporal branch can be optimized, enabling them to extract feature representations that more closely approximate the true semantics. It is worth noting that this embodiment only uses the text branch during the training phase; during the testing phase, only the spatial and temporal branches are used.
[0048] S22. Generate query prototypes and support prototypes for each supported category.
[0049] In this step, the fusion features are... A global average pooling operation is applied along the temporal dimension to obtain a video-level representation. For the support set, the support prototype for each class is obtained by computing the average vector of the support samples within the same class. For query sets, the aggregated vector of a single sample is directly used as its query prototype. .
[0050] S23. Calculate the distance between the query prototype and the supporting prototypes of each supporting category based on a few-sample classifier, specifically expressed as:
[0051] In the formula, Represents the similarity function; Indicates a query sample in the query set. The query prototype; This indicates that support for concentration belongs to the action category. Support for prototypes; Indicates query sample Belongs to the action category The predicted probability; Indicates support for set number Support prototypes for each supported category For category indexing, .
[0052] The category with the highest predicted probability is used as the final prediction result for the video sample to be identified and output.
[0053] The training process for an action recognition model includes: Construct a dataset consisting of video frame images with known category labels; The training set is input into the behavior recognition model to train the model. In each round of training, a few-sample learning task is constructed from the training set. The model predicts the label of the query set by learning the knowledge of the support set and updates the parameters according to the prediction loss. When the loss function of the model tends to converge, or when the number of iterations reaches the set number, training stops, and the trained few-sample behavior recognition model is obtained.
[0054] The training set is a video dataset, which includes several videos, each containing several video frames. The few-sample behavior recognition method in this scheme uses a sparse temporal sampling strategy to sample video frame sequences as output, and outputs the classification result of the video to be recognized.
[0055] The few-shot learning task refers to, given a dataset, sampling one sample from it. -way -shot task, which consists of a support set and a query set; Support set includes There are 10 categories, each with 10 categories. One video sample; The query set consists of several unlabeled video samples from the same category as the support set; The model predicts the labels for the query set by learning knowledge from the support set and updates its parameters based on the cross-entropy loss.
[0056] During the training phase, an episode is sampled from the training set. An episode refers to a small training unit that simulates a single test task. An episode is constructed according to the following process: (1) Randomly sample n categories from the training set; (2) Sample k samples from each category as support; (3) Sample Q queries for each category, and these queries do not overlap with the support queries.
[0057] Once the behavior recognition model is trained, determine whether user input is required for the support set based on the categories to be recognized. (1) When the categories are fixed: For example, if 20 action categories are identified online, the categories remain unchanged. In this case, the support set can be prepared and fixed in advance by the system: directly store several representative video features of each class, or store the "prototype" (class center vector) of each class, or store a small support library, from which the nearest neighbor is selected online. In this case, the user does not need to input the support set, but only needs to input the query; (2) When the category is not fixed: For example, if the user wants the system to temporarily recognize "a new action" that the system has not seen before, then the user needs to provide a support set.
[0058] The sparse temporal sampling strategy is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Randomly divided into The system uses 8 video clips (the default setting is 8 clips), and then randomly selects one frame from each video clip to construct a video frame sequence. .
[0059] Example 2 In a typical embodiment of the present invention, this embodiment discloses a few-shot behavior recognition system based on a hierarchical representation learning network, comprising: The sample acquisition module is configured to: acquire video samples to be identified to construct a query set, and acquire labeled video samples to construct a support set. The behavior recognition module is configured to: take video samples from the support set and query set as input, generate fusion features using a pre-trained behavior recognition model; obtain video-level features based on the fusion features, and generate query prototypes for the query set and support prototypes for each category in the support set; calculate the distance between the query prototype and the support prototype of each support category, obtain the final prediction result for the video sample to be recognized, and output it.
[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A few-shot behavior recognition method based on hierarchical representation learning networks, characterized in that, include: Obtain video samples to be identified to construct a query set, and obtain labeled video samples to construct a support set; Using video samples from the support set and query set as input, a pre-trained behavior recognition model is used to generate fused features; Video-level features are obtained based on fusion features, and query prototypes for query sets and support prototypes for each category in the support set are generated. Calculate the distance between the query prototype and the supporting prototypes of each supporting category to obtain the final prediction result of the video sample to be identified and output it.
2. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 1, characterized in that, The behavior recognition model consists of three parallel branches and a few-shot classifier. The three parallel branches are a spatial branch, a temporal branch, and a text branch; the text branch is used only during the training phase.
3. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 2, characterized in that, In the spatial branch, using video samples from the support set and query set as input, the visual encoder is first used to extract visual features, then spatial augmentation attention is introduced to reconstruct and enhance the visual features in the spatial dimension, then key information is aggregated through average pooling to generate the average visual features of the video samples, and finally spatial augmentation is performed in the channel dimension to generate aggregated spatial features. In the temporal branch, using video samples from the support set and query set as input, visual features are first extracted using a visual encoder, then motion features are calculated, and finally temporal aggregation and alignment are performed to generate aggregated temporal features. The aggregated spatial features and aggregated temporal features are fused together by element-wise addition to generate fused features.
4. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 3, characterized in that, In the text branch, the class labels of the annotated video samples in the support set are used as input. The CLIP text encoder is used to extract real semantic features, and the learning of fused features is supervised based on the real semantic features.
5. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 4, characterized in that, Global average pooling operation in the temporal dimension is applied to the fused features to obtain video-level features. For the support set, the support prototype of each category is obtained by calculating the average vector of the support samples within the same category. For a query set, the aggregated vector of a single sample is used as its query prototype.
6. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 3, characterized in that, Spatial augmentation is specifically represented as: In the formula, Indicates a space augmentation function; and These represent the average visual features of the supporting video samples and the query video samples, respectively. Indicates a splicing operation; This refers to the aggregated spatial features output by the spatial enhancement function.
7. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 3, characterized in that, The calculation of motion features is specifically represented as follows: In the formula, This represents a two-dimensional convolutional layer that performs feature transformations on each channel separately. Indicates the features of the input video frame; This indicates the number of video frames sampled for each video sample.
8. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 3, characterized in that, The specific representation of time aggregation alignment is as follows: In the formula, This is the output of the time aggregation alignment module; Indicates attention operation; Indicates motion characteristics, This indicates the average motion characteristics.
9. The few-shot behavior recognition method based on hierarchical representation learning network as described in claim 1, characterized in that, The distance between the query prototype and the supporting prototypes for each supporting category is calculated and represented as follows: In the formula, Represents the similarity function; Indicates a query sample in the query set. The query prototype; This indicates that support for concentration belongs to the action category. Support for prototypes of support samples; Indicates query sample Belongs to the action category The predicted probability; Indicates support for set number Support prototypes for each category For category indexing, .
10. A few-shot behavior recognition system based on a hierarchical representation learning network, characterized in that, include: The sample acquisition module is configured to: acquire video samples to be identified to construct a query set, and acquire labeled video samples to construct a support set. The behavior recognition module is configured to take video samples from the support set and query set as input and generate fused features using a pre-trained behavior recognition model. Video-level features are obtained based on fusion features, and query prototypes for query sets and support prototypes for each category in the support set are generated. Calculate the distance between the query prototype and the supporting prototypes of each supporting category to obtain the final prediction result of the video sample to be identified and output it.