A long sequence action recognition method for small sample learning

Abstract

The application provides a long sequence action recognition method for small sample learning, aiming to solve the high computational complexity problem faced by traditional Transformer-based methods when processing long sequences, improve the ability of local feature modeling and time alignment, and alleviate the negative impact of intra-class difference accumulation on model performance. The application mainly includes three core parts: a feature extraction module, a Mamba branch module and a contrast learning module. In view of the limitations of the traditional Mamba method in small sample action recognition, the application proposes a multi-scale Matryoshka Mamba model. The model enhances the expression ability of local features through the inner module, and realizes the accurate alignment of local features through the outer module. In addition, the application also proposes a hybrid contrast learning method to effectively alleviate the negative impact of intra-class difference accumulation of long sequences of the same class on small sample action recognition performance. The application improves the efficiency of small sample learning while significantly improving the accuracy and robustness of long sequence action recognition.

Description

[0001] Field of study

[0002] This invention relates to a method for recognizing long sequence actions using few-shot learning, belonging to the field of artificial intelligence technology. Background Technology

[0003] With the rapid development of computer vision and deep learning technologies, action recognition technology has been widely applied in practical applications such as intelligent surveillance, video understanding, and health monitoring. However, traditional data-driven training methods typically rely on a large number of labeled samples, which is not only costly but also time-consuming. To address this issue, Few-Shot Learning (FSL) has emerged, its core idea being to learn and recognize unseen action categories using very few samples. Few-Shot Action Recognition (FSAR) aims to reduce the dependence on large-scale labeled samples and has become an important research direction in this field.

[0004] Currently, Transformer-based models are one of the mainstream methods in FSAR. These methods utilize self-attention mechanisms to capture global features in video sequences, achieving significant results when processing short video clips. However, due to computational complexity limitations, these methods typically can only process short video subsequences of fewer than 16 frames, making it difficult to fully exploit the rich spatiotemporal contextual information contained in long sequences. Although longer subsequences can provide more contextual information, existing research in the FSAR field still has limited utilization of long sequences.

[0005] Recently, the Mamba model has been proposed and has demonstrated superior long sequence modeling capabilities on multiple tasks while avoiding high computational burden. Mamba employs State Space Models (SSMs) to replace the complex attention mechanism in the Transformer, enabling flexible propagation or discarding of contextual information. However, Mamba primarily focuses on global feature modeling, neglecting the alignment capability of local features between samples, which contradicts the requirement of FSAR to fully utilize local features with a very small number of samples.

[0006] While applying Mamba to long sequence modeling in FSAR shows promise, it faces two main challenges. First, Mamba lags behind in local feature modeling and temporal alignment. Some local features, though subtle, are crucial. Mamba's emphasis on global features inevitably overlooks these important local features, leading to decreased action classification accuracy. Furthermore, Mamba lacks an effective temporal alignment mechanism, failing to effectively match key local features across different samples, further impacting FSAR performance. Second, as subsequence length increases, the accumulation of intra-class discrepancies becomes more pronounced, negatively affecting model performance.

[0007] In existing FSAR methods, metric-based meta-learning has become the mainstream paradigm. Through feature extraction, these methods embed support set samples into class prototypes to calculate distances between query samples and perform classification. However, existing methods often neglect the capture and alignment of local features when dealing with long sequences, leading to poor model performance. Furthermore, the problem of intra-class variance accumulation is not significant when processing short sequences, but it becomes significant when using Mamba with long sequences.

[0008] To address the above observations, we propose a few-shot learning method for long sequence action recognition, effectively solving the related problems. However, in practical applications, the problem of increased computational complexity may still arise. Although the Mamba model exhibits good performance when processing long sequences, the computational complexity increases accordingly with extreme increases in sequence length, becoming an unavoidable challenge. Furthermore, for some applications requiring real-time processing, such as video surveillance or online health monitoring, this model may fail to meet real-time requirements due to computational complexity and processing time constraints. In such cases, a balance needs to be found between model performance and real-time performance, or the model needs to be optimized to reduce latency. Summary of the Invention

[0009] This invention provides a long-sequence action recognition method for few-shot learning, aiming to solve the high computational complexity problem faced by traditional Transformer-based methods when processing long sequences. It also improves the ability of local feature modeling and temporal alignment, and mitigates the negative impact of intra-class variance accumulation on model performance. This invention mainly comprises three core parts: a feature extraction module, a Mamba branch module, and a contrastive learning module. Addressing the limitations of traditional Mamba methods in few-shot action recognition, this invention proposes a multi-scale Matryoshka Mamba model. This model enhances the expressive power of local features through inner modules and achieves accurate alignment of local features through outer modules. Furthermore, this invention proposes a hybrid contrastive learning method to effectively mitigate the negative impact of intra-class variance accumulation in long sequences of the same class on few-shot action recognition performance. Compared with existing technologies, this invention significantly improves the accuracy and robustness of long-sequence action recognition while enhancing few-shot learning efficiency.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows: a long sequence action recognition method based on few-sample learning, characterized in that the method includes the following steps:

[0011] Step 1: Use a deep neural network as a feature extractor, built using the PyTorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips from the support set and query set into the feature extractor respectively, extract and generate feature vectors with a shape of 2048 × frame number, as preparation for steps 2 and 3.

[0012] Step 2: Input the feature vectors generated in Step 1 into the Matryoshka Mamba model. This model employs a multi-scale design, enhancing local features through inner modules and aligning temporal features using outer modules to construct class prototypes. Then, based on vector inversion and cross-distance calculation, the final category of the query set is determined.

[0013] Step 3: Based on the results of Step 1, supervised contrastive learning is performed on the videos in the support set, unsupervised contrastive learning is performed on the videos in the query set, and unsupervised contrastive learning is performed on all videos in both the support set and the query set. Three loss function results are obtained through these three methods, and their sum is the loss result of the contrastive learning.

[0014] Step 4: Multiply the cross-entropy loss of the Mamba branch model by the weights and sum the result of the contrastive learning loss to obtain the total loss result for the final training objective.

[0015] Step one is described in detail as follows:

[0016] Step 1-1: Construct training, validation, and test sets. These sets are further divided into support and query sets. A unit of the support set is an N-way K-shot task with N classifications and K labeled samples per class. The query set contains N unlabeled samples across all classes. The purpose of classification is to assign unlabeled query set samples to a class within the support set.

[0017] Steps 1-2: Extract features from the support set S and query set Q using a feature extractor. The feature extractor can be instantiated as ResNet50 and implemented using the PyTorch framework. and These are the features of the k-th video in category c of the support set and the query set Q, respectively, where f represents the feature, f θ For feature extractor, s and q represent the support set and query set videos respectively, r indicates that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

[0018]

[0019] Step two is described in detail below:

[0020] The MatryoshkaMamba module employs a multi-scale architecture. At each scale, multiple inner modules are used for local feature enhancement. The outputs of the inner modules are concatenated and then input into the outer module for temporal alignment. Simultaneously, this input is convolved with the feature vector itself through a two-dimensional convolution operation. After summing, the scale weights are calculated using the sigmoid function. We define the set O as the multi-scale parameter, |O| represents the number of scales, and element o represents the number of input frames per intramodule for each scale. For simplicity, we use the following table to represent any scale o.

[0021] Step 2-1: In the internal module, using the state-space model (SSM) of a linear time-invariant system, the input feature x(t)∈R is transformed. L Replace with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L Its basic working principle is as follows:

[0022] h′(t)=Ah(t)+Bx(t),

[0023] y(t) = Ch(t).

[0024] Where A∈R H×H Let B, C ∈ R be the state transition matrix. H These are projection parameters. Local features. It is a tensor with hidden states, where L = D and H = 0. Due to the variability of local features, the inner module includes forward and backward sub-branches IM. Fw (·) and IM Bw (·), the two do not share parameters. The final output is:

[0025]

[0026] Where Linear represents a linear transformation. This represents the addition of vectors.

[0027] The local features are concatenated and added to the input features to calculate the features for temporal alignment by the external module. and They are also tensors with implicit states, where L = F and H = D. The calculation process is as follows:

[0028]

[0029] Step 2-2: In the external module, considering the symmetry of timing information, the external module performs a forward scan of the OM (Original Metrics and Memory). Fw (·) and backward scan OM Bw (·) Bidirectional feature capture is performed to perceive the dependencies between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is represented as:

[0030]

[0031] Learnable weights and This is used to perform a weighted average of features at different scales. The weights are derived from the input features of the outer module. and its own characteristics The learnable weights are calculated from the above. Specifically, the calculation steps are as follows:

[0032]

[0033] Here, Conv2D Block is defined as CB(·).

[0034] Next, by combining the learnable weights with the output of the outer module, an output representation of arbitrary scale can be obtained. and As shown below:

[0035]

[0036] Finally, the output features of the Matryoshka Mamba model and The following results were obtained by averaging the outputs at each scale:

[0037]

[0038] Steps 2-3: Support centralized class prototypes It is constructed using the following formula:

[0039]

[0040] Steps 2-4: To further enhance the alignment of temporal features, this invention employs a cross-distance calculation method. Specifically, firstly, the output features... and Reverse the image to obtain the reversed features. and Considering the principle of symmetry alignment, tensors sharing the same superscript are more likely to belong to the same category if the distance between them is small, or tensors with different superscripts are more likely to belong to the same category. Therefore, the reciprocal of the distance between tensors with different superscripts is taken.

[0041] The specific distance calculation steps are as follows:

[0042]

[0043]

[0044] Then, the distance between the query set and the c-th category of the support set is defined as the average of the above four distances, and the category with the smallest distance is the final prediction result of the model. The prediction result is:

[0045]

[0046] Based on the actual category of the query set Y Q It is the set of all categories, and the cross-entropy loss result is calculated as follows:

[0047]

[0048] Step three is described in detail below:

[0049] By introducing a hybrid contrastive learning approach, the negative impact of intra-class variance accumulation on model performance is mitigated. The calculated loss function is used to optimize the model, and its expression is as follows:

[0050]

[0051] Where sim(·,·) represents cosine similarity, and τ is the hyperparameter temperature. In the supervised learning paradigm, for support set samples, features z with the same label...p As a positive sample, the R features z with different labels are... n These are then treated as negative samples. In the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L... hc It consists of the following three parts: supervised contrast loss of the support set. Unsupervised contrastive loss of query set and the contrast loss of all samples The final loss function is:

[0052]

[0053] Step four is described in detail below:

[0054] The final loss function is obtained by calculating the loss functions of the Mamba branch module and the contrastive learning module, and is used to train the model.

[0055] L total =λL ce +L hc

[0056] Where λ represents the weight parameter of the cross-entropy loss.

[0057] A long sequence action recognition system with few-shot learning, the system comprising the following modules: a feature extraction module, a multi-scale Matryoshka-Mamba model, and a contrastive learning module.

[0058] As the front-end module of the system, the main function of the feature extraction module is to transform the original video clips into feature vectors suitable for machine learning processing, providing sufficient data support for subsequent multi-scale processing and comparative learning.

[0059] The multi-scale MatryoshkaMamba model and the contrastive learning module are implemented in parallel.

[0060] The multi-scale MatryoshkaMamba model performs local feature enhancement and global temporal alignment on the feature vectors, generates class prototypes for classification, and calculates the cross-entropy loss. This multi-scale processing ensures that the system can capture complex spatiotemporal dynamic features, improving the accuracy of action recognition.

[0061] To reduce the impact of accumulated intra-class differences on model performance, a contrastive learning module was introduced. This module improves the system's recognition accuracy by reducing intra-class differences and increasing inter-class differences, while also calculating the contrastive learning loss.

[0062] The final loss result is obtained by multiplying the cross-entropy loss of the MatryoshkaMamba module by a weight and summing it with the contrastive learning loss. This result is then used to train the model.

[0063] Compared with the prior art, the advantages of the present invention are as follows:

[0064] 1. Highly efficient long sequence modeling capability: This invention employs a multi-scale Matryoshka-Mamba model, which effectively improves the ability to capture temporal information in long video sequences by decomposing them into multiple sub-segments and processing them separately, while significantly reducing computational complexity. Compared with existing Transformer-based methods, this invention can handle even longer sequences while maintaining lower computational resource consumption.

[0065] 2. Precise Alignment of Local Features: By introducing a multi-scale design of inner and outer modules, this invention can accurately align important local features between samples while maintaining global feature modeling capabilities. Compared with existing methods, this invention significantly improves the accuracy of similar action recognition, especially in application scenarios sensitive to local features. In actual testing, as the training task increases, the DTW (Dynamic Time Warping) metric gradually decreases, indicating a significant improvement in the model's temporal alignment capability. DTW is a technique for temporal alignment specifically designed to address the temporal dimension misalignment problem in long sequence action recognition. A lower DTW metric indicates better alignment performance between different time series.

[0066] 3. Effective Intra-Class Discrepancy Handling: This invention combines supervised and unsupervised contrastive learning strategies. By designing a hybrid contrastive learning paradigm, it effectively mitigates the problem of intra-class discrepancy accumulation caused by shooting conditions or post-processing in long sequences. In practical applications, this method can significantly improve the classification accuracy and robustness of the model when processing long video sequences under different shooting angles or lighting conditions. Attached image description:

[0067] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0068] Figure 2 This is a schematic diagram of the MatryoshkaMamba module of the present invention;

[0069] Figure 3 This is a schematic diagram of the internal modules of the MatryoshkaMamba module of the present invention;

[0070] Figure 4 This is a schematic diagram of the external module of the MatryoshkaMamba module of the present invention. Detailed Implementation

[0071] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0072] Example 1: See Figure 1 A method for recognizing long sequence actions using few-shot learning, comprising the following steps:

[0073] Step 1: Use a deep neural network as a feature extractor, built using the PyTorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips from the support set and query set into the feature extractor respectively, extract and generate feature vectors with a shape of 2048 × frame number, as preparation for steps 2 and 3.

[0074] Step 2: Input the feature vectors generated in Step 1 into the Matryoshka Mamba model. This model employs a multi-scale design, enhancing local features through inner modules and aligning temporal features using outer modules to construct class prototypes. Then, based on vector inversion and cross-distance calculation, the final category of the query set is determined.

[0075] Step 3: Based on the results of Step 1, supervised contrastive learning is performed on the videos in the support set, unsupervised contrastive learning is performed on the videos in the query set, and unsupervised contrastive learning is performed on all videos in both the support set and the query set. Three loss function results are obtained through these three methods, and their sum is the loss result of the contrastive learning.

[0076] Step 4: Multiply the cross-entropy loss of the Mamba branch model by the weights and sum the result of the contrastive learning loss to obtain the total loss result for the final training objective.

[0077] Step one is described in detail as follows:

[0078] Step 1-1: Construct training, validation, and test sets. These sets are further divided into support and query sets. A unit of the support set is an N-way K-shot task with N classifications and K labeled samples per class. The query set contains N unlabeled samples across all classes. The purpose of classification is to assign unlabeled query set samples to a class within the support set.

[0079] Steps 1-2: Extract features from the support set S and query set Q using a feature extractor. The feature extractor can be instantiated as ResNet50 and implemented using the PyTorch framework. and These are the features of the k-th video in category c of the support set and the query set Q, respectively, where f represents the feature, f θFor feature extractor, s and q represent the support set and query set videos respectively, r indicates that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

[0080]

[0081] Step two is described in detail below:

[0082] See Figure 2 The MatryoshkaMamba module employs a multi-scale architecture. At each scale, multiple inner modules are used for local feature enhancement. The outputs of the inner modules are concatenated and then input into the outer module for temporal alignment. Simultaneously, this input is convolved with the feature vector itself through a two-dimensional convolution operation. After summing, the scale weights are calculated using the sigmoid function. We define the set O as the multi-scale parameter, |O| represents the number of scales, and element o represents the number of input frames per intramodule for each scale. For simplicity, we use the following table to represent any scale o.

[0083] Step 2-1: See Figure 3 Within the internal module, the state-space model (SSM) of a linear time-invariant system is used to represent the input feature x(t)∈R. L Replace with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L Its basic working principle is as follows:

[0084] h′(t)=Ah(t)+Bx(t),

[0085] y(t) = Ch(t).

[0086] Where A∈R H×H Let B, C ∈ R be the state transition matrix. H These are projection parameters. Local features. It is a tensor with hidden states, where L = D and H = 0. Due to the variability of local features, the inner module includes forward and backward sub-branches IM. Fw (·) and IM Bw (·), the two do not share parameters. The final output is:

[0087]

[0088] Where Linear represents a linear transformation. This represents the addition of vectors.

[0089] The local features are concatenated and added to the input features to calculate the features for temporal alignment by the external module. and They are also tensors with implicit states, where L = F and H = D. The calculation process is as follows:

[0090]

[0091] Step 2-2: See Figure 4 In the external module, considering the symmetry of timing information, the external module uses a forward scan OM. Fw (·) and backward scan OM Bw (·) Bidirectional feature capture is performed to perceive the dependencies between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is represented as:

[0092]

[0093] Learnable weights and This is used to perform a weighted average of features at different scales. The weights are derived from the input features of the outer module. and its own characteristics The learnable weights are calculated from the above. Specifically, the calculation steps are as follows:

[0094]

[0095] Here, Conv2D Block is defined as CB(·).

[0096] Next, by combining the learnable weights with the output of the outer module, an output representation of arbitrary scale can be obtained. and As shown below:

[0097]

[0098] in, This represents vector multiplication.

[0099] Finally, the output features of the Matryoshka Mamba model and The following results were obtained by averaging the outputs at each scale:

[0100]

[0101]

[0102] Steps 2-3: Support centralized class prototypes It is constructed using the following formula:

[0103]

[0104] Steps 2-4: To further enhance the alignment of temporal features, this invention employs a cross-distance calculation method. Specifically, firstly, the output features... and Reverse the image to obtain the reversed features. and Considering the principle of symmetry alignment, tensors sharing the same superscript are more likely to belong to the same category if the distance between them is small, or tensors with different superscripts are more likely to belong to the same category. Therefore, the reciprocal of the distance between tensors with different superscripts is taken.

[0105] The specific distance calculation steps are as follows:

[0106]

[0107] Then, the distance between the query set and the c-th category of the support set is defined as the average of the above four distances, and the category with the smallest distance is the final prediction result of the model. The prediction result is:

[0108]

[0109] Based on the actual category of the query set Y Q It is the set of all categories, and the cross-entropy loss result is calculated as follows:

[0110]

[0111] Step three is described in detail below:

[0112] By introducing a hybrid contrastive learning approach, the negative impact of intra-class variance accumulation on model performance is mitigated. The calculated loss function is used to optimize the model, and its expression is as follows:

[0113]

[0114] Where sim(·,·) represents cosine similarity, and τ is the hyperparameter temperature. In the supervised learning paradigm, for support set samples, features z with the same label... p As a positive sample, the R features z with different labels are... n These are then treated as negative samples. In the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L... hc It consists of the following three parts: supervised contrast loss of the support set. Unsupervised contrastive loss of query set and the contrast loss of all samples The final loss function is:

[0115]

[0116] Step four is described in detail below:

[0117] The final loss function is obtained by calculating the loss functions of the Mamba branch module and the contrastive learning module, and is used to train the model.

[0118] L total =λ Le +L hc

[0119] Where λ represents the weight parameter of the cross-entropy loss.

[0120] Specific implementation: This invention is mainly trained on GPU servers and tested on edge devices.

[0121] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0122] The system of this invention consists of a server and multiple edge devices. The training and validation sets are stored on the server side, primarily used for model training and validation; the test set is stored on the edge devices, eliminating the need to upload to the server to maintain data privacy and real-time performance. This invention provides a long-sequence action recognition method for few-shot learning, mainly addressing the problems of high computational complexity, insufficient local feature modeling, temporal alignment discrepancies, and intra-class difference accumulation in traditional methods when processing long sequences. The core of this invention includes three main modules: feature extraction, a Matryoshka-Mamba module, and a contrastive learning module. First, addressing the problem of few-shot action recognition in long video sequences, this invention analyzes the reasons for low efficiency in temporal information extraction and improves upon the shortcomings of traditional methods. Specifically, this invention designs a Matryoshka-Mamba model, introducing multiple internal modules to enhance local features and external modules to align temporal features. Simultaneously, a contrastive learning paradigm is introduced to reduce the impact of inter-class difference accumulation on model performance. When applying this method, the server first needs to perform preprocessing preparation work, including the following steps:

[0123] Step 1: Use a deep neural network as a feature extractor, built using the PyTorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips from the support set and query set into the feature extractor respectively, extract and generate feature vectors with a shape of 2048 × frame number, as preparation for steps 2 and 3.

[0124] Step 2: Input the feature vectors generated in Step 1 into the Matryoshka Mamba model. This model employs a multi-scale design, enhancing local features through inner modules and aligning temporal features using outer modules to construct class prototypes. Then, based on vector inversion and cross-distance calculation, the final category of the query set is determined.

[0125] Step 3: Based on the results of Step 1, supervised contrastive learning is performed on the videos in the support set, unsupervised contrastive learning is performed on the videos in the query set, and unsupervised contrastive learning is performed on all videos in both the support set and the query set. Three loss function results are obtained through these three methods, and their sum is the loss result of the contrastive learning.

[0126] Step 4: Multiply the cross-entropy loss of the Mamba branch model by the weights and sum the result of the contrastive learning loss to obtain the total loss result for the final training objective.

[0127] Step one is described in detail as follows:

[0128] Step 1-1: Construct training, validation, and test sets. These sets are further divided into support and query sets. A unit of the support set is an N-way K-shot task with N classifications and K labeled samples per class. The query set contains N unlabeled samples across all classes. The purpose of classification is to assign unlabeled query set samples to a class within the support set.

[0129] Steps 1-2: Extract features from the support set S and query set Q using a feature extractor. The feature extractor can be instantiated as ResNet50 and implemented using the PyTorch framework. and These are the features of the k-th video in category c of the support set and the query set Q, respectively, where f represents the feature, f θ For feature extractor, s and q represent the support set and query set videos respectively, r indicates that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

[0130]

[0131] Step two is described in detail below:

[0132] The MatryoshkaMamba module employs a multi-scale architecture. At each scale, multiple inner modules are used for local feature enhancement. The outputs of the inner modules are concatenated and then input into the outer module for temporal alignment. Simultaneously, this input is convolved with the feature vector itself through a two-dimensional convolution operation. After summing, the scale weights are calculated using the sigmoid function. We define the set O as the multi-scale parameter, |O| represents the number of scales, and element o represents the number of input frames per intramodule for each scale. For simplicity, we use the following table to represent any scale o.

[0133] Step 2-1: In the internal module, using the state-space model (SSM) of a linear time-invariant system, the input feature x(t)∈R is transformed. L Replace with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L Its basic working principle is as follows:

[0134] h′(t)=Ah(t)+Bx(t),

[0135] y(t) = Ch(t).

[0136] Where A∈R H×H Let B, C ∈ R be the state transition matrix. H These are projection parameters. Local features. It is a tensor with hidden states, where L = D and H = 0. Due to the variability of local features, the inner module includes forward and backward sub-branches IM. Fw (·) and IM Bw (·), the two do not share parameters. The final output is:

[0137]

[0138] Where Linear represents a linear transformation. This represents the addition of vectors.

[0139] The local features are concatenated and added to the input features to calculate the features for temporal alignment by the external module. and They are also tensors with implicit states, where L = F and H = D. The calculation process is as follows:

[0140]

[0141] Step 2-2: In the external module, considering the symmetry of timing information, the external module performs a forward scan of the OM (Original Metrics and Memory). Fw (·) and backward scan OM Bw (·) Bidirectional feature capture is performed to perceive the dependencies between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is represented as:

[0142]

[0143] Learnable weights and This is used to perform a weighted average of features at different scales. The weights are derived from the input features of the outer module. and its own characteristics The learnable weights are calculated from the above. Specifically, the calculation steps are as follows:

[0144]

[0145] Here, Conv2D Block is defined as CB(·).

[0146] Next, by combining the learnable weights with the output of the outer module, an output representation of arbitrary scale can be obtained. and As shown below:

[0147]

[0148] Finally, the output features of the Matryoshka Mamba model and The following results were obtained by averaging the outputs at each scale:

[0149]

[0150] Steps 2-3: Support centralized class prototypes It is constructed using the following formula:

[0151]

[0152] Steps 2-4: To further enhance the alignment of temporal features, this invention employs a cross-distance calculation method. Specifically, firstly, the output features... and Reverse the image to obtain the reversed features. and Considering the principle of symmetry alignment, tensors sharing the same superscript are more likely to belong to the same category if the distance between them is small, or tensors with different superscripts are more likely to belong to the same category. Therefore, the reciprocal of the distance between tensors with different superscripts is taken.

[0153] The specific distance calculation steps are as follows:

[0154]

[0155]

[0156] Then, the distance between the query set and the c-th category of the support set is defined as the average of the above four distances, and the category with the smallest distance is the final prediction result of the model. The prediction result is:

[0157]

[0158] Based on the actual category of the query set The cross-entropy loss result was calculated as follows:

[0159]

[0160] Step three is described in detail below:

[0161] By introducing a hybrid contrastive learning approach, the negative impact of intra-class variance accumulation on model performance is mitigated. The calculated loss function is used to optimize the model, and its expression is as follows:

[0162]

[0163] Where sim(·,·) represents cosine similarity, and τ is the hyperparameter temperature. In the supervised learning paradigm, for support set samples, features z with the same label... p As a positive sample, the R features z with different labels are... n These are then treated as negative samples. In the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L... hc It consists of the following three parts: supervised contrast loss of the support set. Unsupervised contrastive loss of query set and the contrast loss of all samples The final loss function is:

[0164]

[0165] Step four is described in detail below:

[0166] The final loss function is obtained by calculating the loss functions of the Mamba branch module and the contrastive learning module, and is used to train the model.

[0167] L total =λL ce +L hc

[0168] Where λ represents the weight parameter of the cross-entropy loss.

[0169] This invention may also have other various embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, and these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for recognizing long sequence actions using few-shot learning, characterized in that, The method includes the following steps: Step 1: Use a deep neural network as a feature extractor, built using the PyTorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips from the support set and query set into the feature extractor respectively, extract and generate feature vectors of shape 2048 × frame number, as preparation for steps 2 and 3. Step 2: Input the feature vectors generated in Step 1 into the Matryoshka Mamba model. This model adopts a multi-scale design, enhancing local features through inner modules and aligning temporal features using outer modules to construct class prototypes. Then, based on vector inversion and cross distance calculation, the final category of the query set is determined. Step 3: Based on the results of Step 1, perform supervised contrastive learning on the support set videos, unsupervised contrastive learning on the query set videos, and unsupervised contrastive learning on all videos in both the support and query sets. These three methods yield three loss function results, the sum of which is the loss result for contrastive learning. Step 4: Multiply the cross-entropy loss of the Mamba branch model by the weights and sum the results with the loss from contrastive learning to obtain the total loss result for the final training objective; Step two is described in detail below: The MatryoshkaMamba module employs a multi-scale architecture. At each scale, multiple inner modules are used for local feature enhancement. The outputs of the inner modules are concatenated and then input into the outer module for temporal alignment. Simultaneously, this input is subjected to a two-dimensional convolution operation and then processed by the feature vector itself. After summing, the scale weights are calculated using the sigmoid function. The set O is defined as the multi-scale parameter, |O| represents the number of scales, and element o represents the number of input frames per intramodule at each scale. The following table uses o to represent any scale. Step 2-1: Within the internal module, utilize the state-space model (SSM) of a linear time-invariant system to convert the input features... Replace with output features Through intermediate hidden states Its basic working principle is as follows: in, Here is the state transition matrix. For projection parameters, local features It is a tensor with hidden states, where L = D, H = o. Due to the variability of local features, the inner module includes forward and backward sub-branches. The two do not share parameters, and the final output is: Where Linear represents a linear transformation. This represents vector addition. The local features are concatenated and added to the input features to calculate the features for temporal alignment by the external module. and ∈ They are also tensors with hidden states, where L = F and H = D, and their calculation process is as follows: ， Step 2-2: In the external module, considering the symmetry of timing information, the external module performs a forward scan. and backward scan Bidirectional feature capture is performed to perceive the dependencies between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is represented as follows: Learnable weights and This is used to perform a weighted average of features at different scales, with the weights derived from the input features of the outer module. , and its own characteristics Specifically, the calculation steps for learnable weights are as follows: Wherein, Conv2D Block is defined as CB(·), Next, the learnable weights are combined with the output of the outer module to obtain the output representation at any scale. and As shown below: Finally, the output features of the Matryoshka Mamba model and ∈ The following results were obtained by averaging the outputs at each scale: Steps 2-3: Support centralized class prototypes It is constructed using the following formula: Steps 2-4: To further enhance the alignment of temporal features, this invention employs a cross-distance calculation method. Specifically, firstly, the output features... and Reverse the image to obtain the reversed features. and Considering the principle of symmetry alignment, if tensors sharing the same superscript are close in distance, or tensors with different superscripts are far apart, it indicates that they are more likely to belong to the same category. Therefore, the reciprocal of the distance between tensors with different superscripts is taken. The specific distance calculation steps are as follows: Then, the distance between the query set and the c-th category of the support set is defined as the average of the above four distances, and the category with the smallest distance is the final prediction result of the model. The prediction result is: Based on the actual category of the query set , It is the set of all categories, and the cross-entropy loss result is calculated as follows: 。 2. The method for long sequence action recognition based on few-shot learning according to claim 1, characterized in that, Step one is described in detail as follows: Step 1-1: Construct training, validation, and test sets. These sets are further divided into support and query sets. A unit of the support set is an N-way K-shot task with K labeled samples per class. The query set contains N unlabeled samples across all classes. The purpose of the classification is to assign the unlabeled query set samples to a class within the support set. Step 1-2: Use a feature extractor to process the support set and query set Feature extraction is performed; the feature extractor can be instantiated as ResNet50, implemented using the PyTorch framework. and These are the features of the k-th video in category c of the support set and the query set Q, respectively, where f represents the feature. Let F be the feature extractor, where s and q represent the support set and query set videos, respectively, r indicates that the video frames are randomly selected, and F represents the number of frames. The support set video frames are uniformly sampled, while the query set video frames are randomly sampled. 。 3. The method for long sequence action recognition using few-shot learning according to claim 1, characterized in that, Step three is described in detail below: By introducing a hybrid contrastive learning method, the negative impact of intra-class variance accumulation on model performance is mitigated. The calculated loss function is used to optimize the model, and its expression is as follows: Where sim(⋅,⋅) represents cosine similarity, and τ is the hyperparameter temperature. In the supervised learning paradigm, for support set samples, features with the same label are considered similar. As a positive sample, and R features with different labels As a negative sample, in the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final mixture contrastive learning loss function... It consists of the following three parts: supervised contrast loss of the support set. Unsupervised contrastive loss of query sets and the contrast loss of all samples The final loss function is: 。 4. The method for long sequence action recognition using few-shot learning according to claim 1, characterized in that, Step four is described in detail below: The final loss function is obtained by calculating the loss functions of the Mamba branch module and the contrastive learning module, and is used to train the model. in The weight parameters represent the cross-entropy loss.

5. A long sequence action recognition system based on few-shot learning, characterized in that, To implement the recognition method according to any one of claims 1-4, the system comprises the following modules: a feature extraction module, a multi-scale Matryoshka-Mamba model, and a contrastive learning module. As the front-end module of the system, the main function of the feature extraction module is to transform the raw video clips into feature vectors suitable for machine learning processing, providing sufficient data support for subsequent multi-scale processing and comparative learning. The multi-scale MatryoshkaMamba model and the contrastive learning module are implemented in parallel. The multi-scale Matryoshka-Mamba model performs local feature enhancement and global temporal alignment of the feature vectors, generates class prototypes for classification, and calculates the cross-entropy loss. The contrastive learning module aims to reduce the impact of accumulated intra-class variance on model performance. It improves the system's recognition accuracy by reducing intra-class variance and increasing inter-class variance, while simultaneously calculating the contrastive learning loss. The final loss result is obtained by multiplying the cross-entropy loss of the MatryoshkaMamba model by a weight and summing it with the contrastive learning loss. This result is then used to train the model.

Images