Few-shot action recognition network, method, device, and storage medium based on meta-learning and time alignment.

Abstract

This invention provides a few-shot action recognition network, method, device, and storage medium based on meta-learning and time alignment. It includes a meta-training unit and a meta-testing unit. The meta-training unit trains an embedding model with good generalization performance to extract temporal and spatial information from video frames. After training, all network headers, including the classifier, are removed. η The embedded model / feature extractor, whose parameters are frozen or fine-tuned as a meta-test unit, is then evaluated by a linear classifier. This invention applies the novel meta-learning algorithm NIL to few-sample action recognition, achieving accurate and efficient classification of human actions with limited sample data.

Description

Technical Field

[0001] This invention belongs to the field of computer vision (few-shot action recognition) technology. Specifically, it relates to a few-shot action recognition network, method, device, and storage medium based on meta-learning and temporal alignment. Background Technology

[0002] Existing few-shot action recognition still follows the standard Model-Independent Meta-Learning Algorithm (MAML), aiming to acquire a good set of model initialization parameters (i.e., allowing the model to learn to initialize itself). It employs contextual training, comprising meta-training and meta-testing phases, aiming to learn a model or optimizer from a set of visible tasks during meta-training so that it can generalize well to new tasks with a small number of labeled training instances during meta-testing. While MAML has gained widespread acceptance, a fundamental open question remains: is the algorithm's success due to the prior knowledge learned during meta-initialization, which prepares it for rapid learning, or is it due to feature reuse (where the meta-initialization already possesses high-quality features)? This question has also attracted the attention of Aniruddh et al., who, through ablation experiments and latent representation analysis on image classification tasks, found that feature reuse is the primary factor. This led to the development of the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML by removing all inner loops except for the task-specific head of the underlying neural network. Further research into the precise contributions of the network head and body showed that the performance on the test task depends entirely on the quality of the learned features, even to the point that the network head can be removed (NIL algorithm). A detailed comparison of the MAML and NIL algorithms is as follows: Figure 1 As shown ( Embedded model parameters, η: network header parameters).

[0003] Few-shot Action Recognition (FSAR) methods typically follow the standard meta-learning training paradigm MAML, which can be traced back to Compound Memory Networks (CMN). CMN proposed a novel concept of compound memory networks, improving feature representation and offering stronger representational capabilities, enabling action classification through matching and ranking. Furthermore, recent FSAR work focuses on metric-based classification methods, learning a generalizable embedding function to measure instance distances between actions. Zhang et al. proposed an Action Relation Network (ARN), which utilizes a C3D network to extract features and constructs a self-supervised spatiotemporal video module to capture the inherent temporal distribution using attention. Videos are aggregated into fixed-length representations, and the matching degree between two videos is calculated through the relation network. Further, Cao et al. proposed a Temporal Alignment Module (TAM), using the dynamic temporal warping (DTW) optimization algorithm to find the optimal matching path. Temporal alignment effectively utilizes the temporal ranking information in video data, significantly improving the data efficiency of few-shot learning. Around the same time as TAM, Mina et al. proposed a Temporal Attention Relation Network (TARN), which also uses sequence matching to match the query set with the support set videos. The recently state-of-the-art (SOTA) performance of TRX involves constructing a class prototype representation to compare the query subsequence (a combination of multiple frames) with all support set subsequences, aiming to find the closest matching combination to the query sample in the support set.

[0004] In short, the above works all follow the standard meta-learning paradigm MAML. Many works (e.g., TRX, ARN) focus too much on distance metrics between action subsequences, neglecting the long-term temporal relationships of actions. Other works (e.g., TARN) perform strict temporal alignment along the time dimension, ignoring the unique temporal distributions of different actions.

[0005] Furthermore, intra-class temporal offsets, where two frame sequences sampled from two samples of the same class may have some frame offset, can lead to over-aligned embeddings of the same class. For example... Figure 2As shown, both actions involve drinking water. One action places the cup to the lips in the last frame of the video, while the other action puts the cup down after drinking in the last frame. These two issues exist in a large number of action instances and severely degrade the performance of time alignment. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a few-sample action recognition network based on meta-learning and time alignment, which can accurately and efficiently classify human actions with only a small amount of sample data.

[0007] The technical solution adopted by this invention to solve the technical problem is:

[0008] A first aspect of the present invention is to provide a few-shot action recognition network based on meta-learning and time alignment, comprising: a meta-training unit and a meta-testing unit.

[0009] The meta-training unit employs the NIL algorithm and includes:

[0010] The first feature extraction network is used to extract the temporal and spatial information of video frames;

[0011] The average pooling module yields 2N video-level feature sequences. Indicates the features of the original or enhanced video;

[0012] Two fully connected (FC) layers are used to classify and obtain the tags of the original video. and enhance video tags The model is then trained using cross-entropy loss (CELoss), which is the cross-entropy between the original video and the data-augmented video. and The average value is taken as the final cross-entropy loss;

[0013] Frame Cyclic Shift Module (FCSM), Feature V i Simultaneously, the input frame cyclic shift module FCSM generates a video sequence in which frames are cyclically shifted to the right or left, which, together with the original sequence, is mapped to a low-dimensional space through a mapping head;

[0014] Attention-based temporal alignment module (ATA) is used to perform frame-to-frame temporal alignment, obtain frame-level similarity between videos, and aggregate them to obtain the overall similarity of the videos for temporal comparison learning.

[0015] The meta-test unit includes:

[0016] The second feature extraction network freezes the parameters of the embedded model trained in the meta-training stage or fine-tunes them on the support set as a feature extractor.

[0017] A linear classifier is then applied after the second feature extraction network.

[0018] Furthermore, the meta-training unit trains an embedding model with good generalization performance. After extracting the temporal and spatial information of the video frames and completing the training, all network headers, including the classifier, are removed. η ),right The embedded model / feature extractor, which performs parameter freezing or fine-tuning as a meta-test unit, is followed by a linear classifier to evaluate the model.

[0019] Furthermore, the Frame Cyclic Shifting (FCSM) module processes the two input video sequences. And data augmentation Simultaneously, shift the T frames of the video cyclically left or right by T sequences to obtain two video sequences. and in The video sequences representing the shifts are U1 and U2, each containing 2N videos. First, we construct the temporal alignment between the original video sequence and the cyclically shifted video sequence to calculate the inter-frame similarity. At the same time, we retain the main temporal alignment between the original S1 video and the enhanced S2 video as a regularization to correct the alignment between videos without temporal shifts.

[0020] Furthermore, the attention-based temporal alignment module (ATA) utilizes cross-attention to calculate temporal attention weights to obtain... Its alignment-specific feature representation is used with Perform similarity calculations to achieve time alignment.

[0021] Furthermore, the calculation method for the attention-based temporal alignment module (ATA) is as follows:

[0022] Given a pair of features and Map it to a lower-dimensional space using a mapping head. Then By attention weight Mapping to query q i key k i ,value v i ,and By attention weight Mapping to query q j key k j , value v j In order to obtain the alignment-specific T-frame features of action i relative to j First use q i and k j Calculate the attention weights, and then apply them to v. j Interactions

[0023]

[0024] Use q j and k i Calculate the alignment-specific T-frame features of action j relative to i. use The feature vector of any frame is represented by L2 normalization. Then, frame-level temporal alignment between actions is performed, and the temporal features corresponding to actions i and j are calculated. cosine distance

[0025]

[0026] Where cos represents the cosine distance, the alignment score between frames is calculated, and then the scores are aggregated to obtain the global similarity between two actions.

[0027] Furthermore, the network uses a combination of cross-entropy loss and temporal contrast loss for training the network model.

[0028] Furthermore, the temporally aligned network is trained using supervised contrastive learning loss, specifically temporal contrastive loss. The definition is as follows:

[0029]

[0030] Where τ represents the temperature coefficient, and I∈(0,1) equals 1 to indicate that the condition is satisfied. Indicates that they have the same class tag The number of videos is used to train the entire model end-to-end. The overall loss of the model is defined as:

[0031] L total =λ1L ce +λ2L tc

[0032] Where λ1 and λ2 represent the weights of the cross-entropy loss and the global contrastive loss, respectively, except for the embedding model. Other network parameters, denoted as η, are obtained after meta-training. and η:

[0033]

[0034] During the meta-testing phase, the network parameter η is discarded, and only the model is embedded. It was retained and reused for model evaluation on the meta-test set.

[0035] A second aspect of the present invention is to provide an action recognition method based on a few-shot action recognition network with meta-learning and temporal alignment, comprising: a meta-training phase and a meta-testing phase.

[0036] In the meta-training phase, the NIL algorithm is used to perform all meta-learning tasks. Merged into a complete meta-training set First, from the merged meta-training set By sampling N pairs of videos and tags from a batch, a video sequence can be obtained. Furthermore, data augmentation is performed on each video to obtain an enhanced video sequence. For an input batch containing 2N videos, following the TSN sampling strategy, each video is divided into T segments, and then one frame is uniformly sampled from each segment. Therefore, each video is represented by a sequence of T frames. i Then embedding the model to V i The T frames are mapped into T frame-level feature sequences respectively. T and n C Let T represent the number of video frames and the channel dimension of each frame, respectively. Then, we perform average pooling on the T-frame features of each video to obtain 2N video-level feature sequences. The features of the original or enhanced video are represented; then, the original video is labeled by classification using two fully connected (FC) layers. and enhance video tags Finally, the model is trained using cross-entropy loss CELoss; the cross-entropy loss is obtained from the original video and the data-augmented video. and The average value is taken as the final cross-entropy loss; meanwhile, feature V i The input frame cyclic shift module FCSM generates an additional video sequence in which frames are cyclically shifted to the right or left. This sequence, along with the original sequence, is mapped to a low-dimensional space through a mapping head. Then, the attention-based temporal alignment module ATA performs frame-to-frame temporal alignment to obtain frame-level similarity between videos. Finally, the sequences are aggregated to obtain the overall similarity of the videos for temporal comparison learning.

[0037] During the meta-testing phase, Dividing the model into a support set and a query set, the parameters of the embedded model trained during the meta-training phase are frozen or stored in the support set. The above is fine-tuned as a feature extractor, followed by a linear classifier for classification; on the query set... Evaluate the network model; depending on the settings, for each task, the query set N*K' samples and the support set N*K samples come from the same N classes.

[0038] A third aspect of the present invention is to provide a few-shot action recognition device based on meta-learning and time alignment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0039] A fourth aspect of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0040] The advantages and positive effects of this invention are:

[0041] 1. This invention applies the novel meta-learning algorithm NIL to few-shot action recognition, demonstrating the effectiveness of its training method for few-shot classification.

[0042] 2. This invention proposes a Circular Shift Module (FCSM) to alleviate the problem of intra-class time offset of actions.

[0043] 3. This invention proposes an attention-based temporal alignment module (ATA) that preserves long-term temporal information of actions, encodes relevant temporal information into the learned visual representation, and uses contrastive learning for supervision to improve the robustness of the embedding model. Attached Figure Description

[0044] Figure 1 A comparison chart of the MAML and NIL algorithms;

[0045] Figure 2 Image comparison for intra-class time offset issues;

[0046] Figure 3 This is a diagram of the overall network structure.

[0047] Figure 4 Here is a structural diagram of the time-cycle shift module;

[0048] Figure 5 A schematic diagram of the Kinetics and SSv2 datasets;

[0049] Figure 6 A visual comparison chart of the visual features of each module. Detailed Implementation

[0050] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0051] To address the problem that previous work, in the process of unordered matching between subsequences, overemphasized the distance metric between actions, neglected the long-term temporal relationship of actions, and lacked rich temporal context information of actions, which may ultimately lead to a suboptimal measurement of action distance, we propose a novel frame-to-frame temporal alignment network for FSAR.

[0052] Compared to all previous few-shot action recognition methods, our proposed method follows the novel meta-learning training paradigm NIL, performing frame-to-frame temporal alignment along the temporal dimension. Simultaneously, we design an additional module to address the temporal offset problem of actions, providing additionally refined feature representations. Furthermore, since using only cross-entropy loss may introduce embeddings tailored for solving the seen-class classification task, the learned visual embeddings may lack generally discriminative visual features. Therefore, considering the good performance of supervised contrastive learning in image classification, we use contrastive learning as an auxiliary pre-training objective to learn, enabling the model to learn to capture useful discriminative features beyond the meta-training set.

[0053] The overall structure of the network is as follows: Figure 3 As shown, the network consists of a meta-training unit and a meta-testing unit. The main purpose of the meta-training unit is to train an embedding model with good generalization performance. After extracting the temporal and spatial information of the video frames and completing the training, all network headers, including the classifier, are removed. η ),right The embedded model / feature extractor undergoes parameter freezing or fine-tuning as part of the meta-testing phase, followed by a linear classifier to evaluate the model. Specifically, the overall overview of the method proposed in this invention is as follows:

[0054] In the meta-training unit, unlike the MAML algorithm, we follow the NIL algorithm, such as... Figure 1 The right figure shows all meta-learning tasks Merged into a complete meta-training set We first start with the merged meta-training set By sampling N pairs of videos and tags from a batch, a video sequence can be obtained. Furthermore, data augmentation is performed on each video to obtain an enhanced video sequence. For each input batch, we obtain 2N videos. We follow the TSN sampling strategy, dividing each video into T segments, and then uniformly sampling one frame from each segment. Therefore, each video is represented by a sequence of T frames, V. i Then the embedded model will V i The T frames are mapped into T frame-level feature sequences respectively. T and n CThese represent the number of video frames and the channel dimension of each frame, respectively. Then, we perform average pooling on the T-frame features of each video to obtain 2N video-level feature sequences. This represents the features of the original or enhanced video. Then, the original video is labeled using two fully connected (FC) layers for classification. and enhance video tags Finally, the model is trained using cross-entropy loss (CELoss). The cross-entropy is calculated by comparing the original video and the data-augmented video. and The average value is taken as the final cross-entropy loss. Meanwhile, feature V... i The Frames Cyclic Shift Module (FCSM) generates an additional video sequence with frames cyclically shifted to the right or left. This sequence, along with the original sequence, is mapped to a low-dimensional space using a mapping head. Then, an Attention-based Temporal Alignment (ATA) module performs frame-to-frame temporal alignment, ultimately obtaining frame-level similarity between videos. This is then aggregated to obtain the overall similarity of the videos for temporal contrastive learning. Therefore, our network ultimately employs a combination of cross-entropy loss and temporal contrastive loss for training the network model.

[0055] In the meta-test unit, similar to MAML, Dividing the model into a support set and a query set, we freeze the parameters of the embedded model trained during the meta-training phase or place them in the support set. The feature extractor is fine-tuned on the query set, followed by a linear classifier for classification. Evaluate the network model. As configured, for each task, the query set of N*K' samples and the support set of N*K samples come from the same N classes.

[0056] To reduce the compactness of embedding features caused by intra-class temporal shifts, we propose frame cyclic shifting to further improve the generalization performance of the embedding model. Specifically, for two input video sequences... And data augmentation We simultaneously shift the T frames of the video cyclically left or right by T sequences, such as... Figure 4 As shown, we obtained two video sequences. and in The video sequences represent the shifted sequences, so video sequences U1 and U2 each contain 2N videos. However, accurately determining which videos are affected by the temporal shift can be difficult. Therefore, we first construct the temporal alignment between the original video sequence and the cyclically shifted video sequence to calculate inter-frame similarity. Simultaneously, we preserve the primary temporal alignment between the original S1 video and the enhanced S2 video as a regularization to correct the alignment between videos without temporal shifts. Thus, further alignment on the shifted video sequences calibrates the distance metric between actions, allowing the embedding model to provide additional refined representations.

[0057] Attention-based temporal alignment (ATA) modules encode relevant temporal information into the learned visual representation, playing a crucial role in improving the robustness of the embedding model and reducing its sensitivity to domain variations. This is important for locally comparing a pair of action features. and We propose an attention-based temporal alignment (ATA) module that performs frame-level temporal alignment along the temporal dimension of the video, ultimately obtaining aggregated video-level similarity for temporal comparison learning, thus preserving long-term temporal relationships of actions. Figure 3 In the TemporalShift Alignment Net, we utilize cross-attention to compute temporal attention weights to obtain... Its alignment-specific feature representation is used with Perform similarity calculations to achieve time alignment.

[0058] Specifically, given a pair of features and We use a mapping head to map it to a lower-dimensional space to obtain Then By attention weight Mapping to query q i ,keyk i ,value v i ,and By attention weight Mapping to query q j key k j , value v j To obtain the alignment-specific T-frame features of action i relative to j. We first use q i and k j Calculate the attention weights, and then apply them to v. j Interactions

[0059]

[0060] To make frame-level alignment between the two videos more stringent, we similarly use q. j and k i Calculate the alignment-specific T-frame features of action j relative to i. Then we use The feature vector of any frame is represented by L2 normalization. Then, frame-level temporal alignment between actions is performed, and the temporal features corresponding to actions i and j are calculated. cosine distance

[0061]

[0062] Here, cosine distance is used to calculate the alignment score between frames. Then, aggregation is performed to obtain the global similarity between two actions.

[0063] In the meta-training phase, we first explored an extremely simple transfer learning baseline, in which the embedding model... First, use the meta-training set. We pre-train the code using the standard cross-entropy (CE) loss. Specifically, for the two input video sequences S1 and S2, we perform average pooling on multiple frames of each video to obtain video-level features. We then use a fully connected layer to perform the subsequent classification, utilizing Cross-Entropy loss. Conduct network training:

[0064]

[0065] Since we are still using the NIL algorithm discussed in the previous chapter, similarly in the meta-training phase, for the two input video sequences S1 and S2, we first perform average pooling on T frames of each video to obtain video-level features. We then use a fully connected layer to perform the subsequent classification, using cross-entropy loss. Conduct network training:

[0066]

[0067] Furthermore, to enable the embedding model to learn more temporal information and better adapt to new tasks, we propose temporal contrastive learning. We further utilize supervised contrastive learning loss for training the temporal alignment network. Specifically, the temporal contrastive learning loss... The definition is as follows:

[0068]

[0069] Where τ represents the temperature coefficient, and I∈(0,1) equals 1 to indicate that the condition is satisfied. Indicates that they have the same class tag The number of videos. We train the entire model end-to-end, and the overall loss of the model is defined as:

[0070] L total =λ1L ce +λ2L tc

[0071] Where λ1 and λ2 represent the weights of the cross-entropy loss and the global contrastive loss, respectively. (Besides the embedding model) Other network parameters, denoted as η, can be obtained after meta-training. and η:

[0072]

[0073] During the meta-testing phase, the network parameter η is discarded, and only the model is embedded. It was retained and reused for model evaluation on the meta-test set.

[0074] Experimental parameter settings

[0075] All experiments in this invention were designed and implemented under the Ubuntu 18.04 operating system, and the programming language used was the widely used high-level programming language Python. The verification implementation of the framework proposed in this invention is based on the deep learning framework PyTorch.

[0076] The experimental parameters are shown in Table 1. In the experiment, for the input video, each video segment was sampled uniformly over time, with the number of frames T set to 8. Each frame was cropped to 224 pixels × 224 pixels, and the designed embedding model was used. Extract video features and output the feature dimension n of each frame. C =d=2048, whose embedding model weights are derived from a ResNet-50 pre-trained on the ImageNet dataset, and finally a fully connected layer is used for classification. For the projection head, we use an MLP with one hidden layer and a ReLU nonlinearity similar to SimCLR, outputting 1152-dimensional features (i.e., d'=1152). For the weight settings of the loss function, λ1=λ2=1. In addition, for the N augmented samples in each batch, we use contrast augmentation (random cropping, color jittering, random horizontal flipping, and random grayscale conversion).

[0077] Table 1 Experimental parameter settings

[0078]

[0079] Experiments were conducted on two GeForce RTX 3090 GPUs. The model was trained for 40 epochs on the HMDB51 and UCF101 datasets, 90 epochs on the SSv2 dataset, and 60 epochs on the Kinetics dataset, with a batch size of 32. We used the SGD optimizer to train all networks, with a momentum of 0.9 and a weight decay of 5e-4. The learning rate was initially set to 0.0001, decaying by a factor of 0.5 every 25 epochs. We employed uniform sampling using TSN, sampling 8 frames evenly from each video. Experimental results show the average accuracy (%) of 5-way 1-shot and 5-way 5-shot over 10,000 tasks.

[0080] 1. Evaluation Indicators

[0081] In the meta-testing phase, we evaluated our network model on standard 5-way 1-shot and 5-way 5-shot settings, reporting the average accuracy across 10,000 tasks randomly selected from the test set. Specifically, for each task / episode, we divided the test set into a support set and a quary set. For the support set, we randomly sampled five action categories and their corresponding class labels, with each category containing either one or five (1-shot / 5-shot) action samples. For the quary set (test samples), we randomly sampled K' videos from the same five categories (different values ​​of K' produce different results). The classifier then outputs which of the five action categories each of the 5*K' test samples belongs to. Furthermore, for each support set video in the meta-testing phase, we created 10 enhanced versions to overcome data insufficiency and used centercropping on the query set video frames.

[0082] 2. Objective assessment

[0083] To objectively and fairly evaluate the effectiveness of the proposed method, comparative experiments were conducted with several advanced high-performance algorithms for few-shot action recognition on four datasets: HMDB51, UCF101, SSv2, and Kinetics. The comparison results are shown in Table 2. The experimental results present the average accuracy (%) under the 5-way 1-shot (5-1) and 5-way 5-shot (5-5) few-shot settings on the four datasets.

[0084] Overall, on four datasets widely used in few-shot action recognition, the proposed method outperforms existing state-of-the-art recognition algorithms. Specifically, in the 5-way 1-shot experimental setting, our proposed method generally outperforms the recently proposed TRX method on all four datasets, achieving improvements of 5.4 / 9.1 / 1.0 / 9.1 percentage points, respectively. In the 5-way 5-shot experimental setting, our method also achieves similar performance on the other three datasets besides SSv2, demonstrating that the NIL-based temporal alignment network is effectively applicable to few-shot action recognition. This invention uses a temporal alignment network to align frames along the temporal dimension, optimizing the distance metric between actions while preserving complete long-term information, further improving performance by 2-3 percentage points compared to our baseline. Furthermore, we utilize cyclic shifting to alleviate intra-class temporal misalignment issues, further improving the performance of the embedding model to obtain more accurate visual representations.

[0085] Table 2 compares the validation results on four datasets with those of state-of-the-art methods.

[0086]

[0087] For the Kinetics-100 dataset used for few-shot action recognition, such as Figure 5 As shown in (i), it focuses more on the overall appearance of the action and does not need to consider the timing misalignment between actions as much, so frame-level timing alignment may be sufficient. For SSv2, as... Figure 5 As shown in (ii), recognizing actions within its dataset requires explicitly capturing changes in the shape or attributes of objects during the action. However, in the meta-testing stage, only the embedding model (ResNet-50) was retained, and its extracted features lacked sensitivity to changes in objects, even though we compensated for this deficiency using a temporal attention network. Therefore, we only performed frame cyclic shifting on HMDB and UCF.

[0088] 3. Ablation test

[0089] 3.1 The impact of network header retention / removal on experimental results

[0090] In the definition of meta-learning problems, we mentioned that Aniruddh et al. proposed NIL, and they first mentioned the ANIL (almost no inner loop) algorithm. They studied the precise contributions of the network head and body, showing that the performance of the test task depends entirely on the quality of the learned features, and that the network head could even be removed, which led to the development of the NIL algorithm. In other words, the difference between ANIL and NIL lies in the retention or removal of the network head. To more accurately apply meta-learning algorithms to few-shot action recognition, we conducted comparative experiments on network head removal and retention, verifying that the network head can indeed be removed in action recognition. Specifically, for the NIL algorithm, we removed the last fully connected (FC) layer from the ResNet-50 model obtained in the meta-training stage, then froze its parameters, and then added a new randomly initialized FC classifier for evaluation in the meta-testing stage. For the ANIL algorithm, we similarly trained the visual embedding model ResNet-50 in the meta-training stage, then retained its last network head (fully connected layer) and froze all network parameters, and then used the frozen network body head (FC) for classification in the meta-testing stage. We also conducted experiments with the MAML algorithm to compare and highlight the effectiveness of the NIL algorithm in small sample action recognition.

[0091] Table 3 compares the accuracy of different meta-learning algorithms on the HMDB51 and UCF101 datasets.

[0092]

[0093] As shown in Table 3, the experiments show that retaining the network head has little impact on our experimental results. During testing, the NIL algorithm without an inner loop performs slightly better than MAML and ANIL, indicating that it learns strong features. Furthermore, the importance of the head is relatively low during testing. The features learned by the main body of the network (and reused during testing) are the key to the meta-learning algorithm.

[0094] 3.2 Ablation experiments of different modules

[0095] To verify that the proposed attention-based temporal alignment module (ATA) and frame circular shift module (FCSM) can effectively improve the model's capture of temporal information and alleviate the problem of unique temporal distribution of different actions, we conducted ablation studies on the proposed modules on the HMDB51 and UCF101 datasets, as shown in Table 4. The network trained with cross-entropy (CE) loss was used as our baseline, and each module was compared with it.

[0096] Table 4 Ablation experiments of different modules

[0097]

[0098] As shown in Table 4, the model performance was significantly improved after adding our volume-based temporal alignment network branch to the baseline. Specifically, integrating our attention-based temporal alignment module, ATA, into the baseline to perform frame-to-frame temporal alignment achieved gains of approximately 1.7% and 1.1% on the HMDB51 and UCF101 datasets, respectively, demonstrating the effectiveness of the ATA module in aggregating frame-level similarity for measuring inter-action distance. Before performing frame-to-frame temporal alignment, we added a frame cyclic shift module, FCSM, to perform temporal cyclic shifting of frame-level features, achieving further improvements of 1.4% and 0.9% on the -way1-shot and 5-way5-shot settings of UCF101, alleviating the intra-class temporal misalignment problem. Overall, our designed temporal alignment module achieved a 2%–3% improvement over the baseline. Furthermore, integrating FCSM into the ATA module produces better feature embeddings than ATA alone, solving the intra-class temporal shift problem caused by the unique temporal distribution of actions, and achieving an effective measure of inter-action distance.

[0099] 4. Subjective assessment

[0100] To more intuitively demonstrate the effectiveness of the proposed network, this invention provides a visual representation of the clustering distribution of visual features in the semantic space under three different conditions. For example... Figure 6 As shown, the methods are: training without a temporal alignment network (using only the baseline (cross-entropy loss)); frame-to-frame temporal alignment using an attention-based temporal alignment network (ATA) on top of the baseline; and finally, FCSM (Fully Connected Simulator) for the complete network. The effectiveness of the proposed method is further analyzed through qualitative visualization of the visual feature distribution with different modules. We use the t-SNE visualization method to visualize the visual feature distribution of some unseen classes in two datasets, extracting 10 unseen classes from the HMDB51 dataset. Figure 6 It can be observed that adding the ATA module during training enables frame-to-frame temporal alignment, achieving an effective distance measurement and widening the inter-class gap. The addition of the Frame Cyclic Shifting (FCSM) module alleviates the problems of local similarity between classes and temporal misalignment within classes, further improving the distance measurement between actions. Figure 6 The visualization results show that the proposed method can effectively utilize time information for action distance measurement, improve the generalization performance of the embedding model, and obtain a more transferable visual embedding representation.

[0101] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the inventive concept, and these all fall within the protection scope of the present invention.

Claims

1. A small sample action recognition network system based on meta-learning and time alignment, characterized in that, include: Meta-training unit and meta-test unit, The meta-training unit employs the NIL algorithm and includes: The first feature extraction network is used to extract the temporal and spatial information of video frames; an average pooling module, obtaining 2N video-level feature sequences , denotes the feature of the original video, denotes the feature of the enhanced video; Two FC full connection layers, classification to get the label of the original video 1 and the label of the enhanced video 2, and the model is trained by cross-entropy loss CE Loss, and the cross-entropy of the original video and the video after data enhancement and Take the average as the final cross-entropy loss frame cyclic shift module FCSM, input frame cyclic shift module FCSM, generating a video sequence which is cyclically shifted right or left, and which is mapped together with the original sequence to the low-dimensional space by the mapping head; Attention-based temporal alignment module (ATA) is used to perform frame-to-frame temporal alignment, obtain frame-level similarity between videos, and aggregate them to obtain the overall similarity of the videos for temporal comparison learning. The meta-test unit includes: The second feature extraction network freezes the parameters of the embedded model trained in the meta-training stage or fine-tunes them on the support set as a feature extractor. A linear classifier is then applied after the second feature extraction network.

2. The identification network system according to claim 1, characterized in that, The meta-training unit trains an embedding model with good generalization performance. After completing the extraction of temporal and spatial information from video frames and training, all network heads, including the classifier, are removed. ,right The embedded model / feature extractor, which performs parameter freezing or fine-tuning as a meta-test unit, is followed by a linear classifier to evaluate the model.

3. The identification network system according to claim 2, characterized in that, The Frame Cyclic Shifter (FCSM) module processes the two input video sequences. And data augmentation Simultaneously, shift the T frames of the video to the left or right by T sequences to obtain two video sequences. and ,in This represents a shifted video sequence, frequency sequence. and Each sequence contains 2N videos. First, temporal alignment is constructed between the original video sequence and the cyclically shifted video sequence to calculate inter-frame similarity. Simultaneously, the original... Video and Enhanced The primary time alignment method between videos is used as a regularization to correct alignment issues between videos that do not have a time offset.

4. The identification network system according to claim 3, characterized in that, The attention-based temporal alignment module (ATA) utilizes cross-attention to calculate temporal attention weights to obtain... Its alignment-specific feature representation is used with Perform similarity calculations to achieve time alignment.

5. The identification network system according to claim 4, characterized in that, The calculation method for the attention-based temporal alignment module (ATA) is as follows: Given a pair of features and Using a mapping head, it is mapped to a lower-dimensional space to obtain ,Then By attention weight Mapping to query key value ,and By attention weight Mapping to query key value In order to obtain action i Compared to j Its alignment-specific T-frame features First use and Calculate the attention weights, and then apply them to... Interactions , use and Calculate the alignment-specific T-frame features of action j relative to i. ,use Represents the feature vector of any frame, and performs the following on each frame: l 2 Normalization is performed, followed by frame-level temporal alignment between actions, and action calculation is then performed. i and j Corresponding time features cosine distance , ， Where cos represents the cosine distance, the alignment score between frames is calculated, and then the scores are aggregated to obtain the global similarity between two actions.

6. The identification network system according to claim 5, characterized in that, The network model is trained using a combination of cross-entropy loss and temporal contrast loss.

7. The identification network system according to claim 6, characterized in that, The temporally aligned network is trained using supervised contrastive learning loss, specifically the temporal contrastive loss. The definition is as follows: in Indicates temperature coefficient ， An equal value of 1 indicates that the condition is met. ， Indicates that they have the same class tag The number of videos is used to train the entire model end-to-end. The overall loss of the model is defined as: in, These represent the weights of the cross-entropy loss and the global contrastive loss, respectively, excluding the embedding model. Other network parameters are represented as After the meta-training is completed, the parameters are obtained. : During the meta-testing phase, network parameters Discarded, only embedded models It was retained and reused for model evaluation on the meta-test set.

8. The identification method of the identification network according to any one of claims 1-7, characterized in that, include: Meta-training phase and meta-testing phase In the meta-training phase, the NIL algorithm is used to perform all meta-learning tasks. Merged into a complete meta-training set First, from the merged meta-training set By sampling N pairs of videos and labels from a batch, a video sequence can be obtained. And data augmentation is performed on each video to obtain an enhanced video sequence. For an input batch containing 2N videos, following the TSN sampling strategy, each video is divided into T segments, and then one frame is uniformly sampled from each segment. Therefore, each video is represented by a sequence of T frames. V i Then the embedded model will V i The T frames are mapped into T frame-level feature sequences respectively. , T and n C Let T represent the number of video frames and the channel dimension of each frame, respectively. Then, average pooling is performed on the T-frame features of each video to obtain 2N video-level feature sequences. , Indicates the characteristics of the original video, This involves enhancing the video's features; then, two fully connected (FC) layers are used for classification to obtain the original video's labels. 1 and enhanced video tags 2. Finally, the model is trained using Cross-Entropy Loss (CE Loss); the cross-entropy between the original video and the data-augmented video is used. and The average value is taken as the final cross-entropy loss; simultaneously, the features The input frame cyclic shift module FCSM generates an additional video sequence in which frames are cyclically shifted to the right or left. This sequence, along with the original sequence, is mapped to a low-dimensional space through a mapping head. Then, the attention-based temporal alignment module ATA performs frame-to-frame temporal alignment to obtain frame-level similarity between videos. Finally, the sequences are aggregated to obtain the overall similarity of the videos for temporal comparison learning.

9. The method according to claim 8, characterized in that, During the meta-testing phase, Dividing the model into a support set and a query set, the parameters of the embedded model trained during the meta-training phase are frozen or stored in the support set. The above is fine-tuned as a feature extractor, followed by a linear classifier for classification; on the query set... Evaluate the network model; based on the settings, the query set for each task. Each sample and the support set The samples come from the same N classes.

10. A few-shot action recognition device based on meta-learning and time alignment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 8-9.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 8-9.

Images