Long sequence action recognition method for small sample learning

Through the multi-scale Matryoshka Mamba model and hybrid contrast learning method, the problems of high computational complexity and insufficient local feature alignment in long-sequence action recognition are solved, and efficient and accurate action recognition is achieved, meeting the requirements of real-time.

CN118968385AActive Publication Date: 2024-11-15SOUTHEAST UNIV

Patent Information

Application Number
CN202411138995.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-15
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

The prior art has high computational complexity in long-sequence action recognition, making it difficult to effectively capture and align local features, resulting in reduced accuracy of action classification and accumulation of intra-class differences, which cannot meet the real-time processing requirements.

Method used

A multi-scale Matryoshka Mamba model is proposed, combining internal and external modules to enhance local feature expression and timing alignment capabilities, and a hybrid comparison learning method is used to alleviate the accumulation of differences within the class, and small samples are improved through feature extraction module, Matryoshka Mamba module and contrast learning module. The efficiency and accuracy of learning.

Benefits of technology

It significantly reduces the computational complexity, improves the accuracy and robustness of long-sequence action recognition, improves local feature alignment capabilities, reduces the negative impact of intra-class differences accumulation on model performance, and meets the real-time processing needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968385A_ABST
    Figure CN118968385A_ABST
Patent Text Reader

Abstract

The invention provides a long sequence action recognition method for small sample learning, and aims to solve the problem of high calculation complexity of a traditional Transform-based method during long sequence processing, improve the capabilities of local feature modeling and time sequence alignment and relieve the negative influence of intra-class difference accumulation on model performance. The method mainly comprises three core parts: a feature extraction module, a Mamba branch module and a comparative learning module. The invention provides a multi-scale Matryoshka Mama model, aiming at the limitation of a traditional Mama method in small sample action recognition. According to the model, the expression ability of local features is enhanced through the inner module, and accurate alignment of the local features is realized through the outer module. In addition, the invention further provides a mixed contrast learning method which is used for effectively relieving the negative influence of intra-class difference accumulation of long sequences of the same class on small sample action recognition performance. According to the method, the accuracy and robustness of long sequence action recognition are remarkably improved while the small sample learning efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Field

[0002] The invention relates to a long sequence action recognition method for small sample learning, belonging to the technical field of artificial intelligence. Background Art

[0003] With the rapid development of computer vision and deep learning technologies, action recognition technology has been widely used in practical applications such as intelligent monitoring, video understanding, and health monitoring. However, traditional data-driven training methods usually rely on a large number of labeled samples, which is not only costly but also time-consuming to collect. To solve this problem, Few-Shot Learning (FSL) came into being. Its core idea is to learn and recognize unseen action categories with very few samples. Few-Shot Action Recognition (FSAR) aims to reduce the dependence on large-scale labeled samples and has become an important research direction in this field.

[0004] Currently, Transformer-based models are one of the mainstream methods in FSAR. This type of method uses the self-attention mechanism to capture global features in video sequences and has achieved remarkable results in processing short video clips. However, due to the limitation of computational complexity, these methods can usually only process short video subsequences of less than 16 frames, and it is difficult to fully mine the rich spatiotemporal contextual information contained in long sequences. Although longer subsequences can provide more contextual information, existing research on the use of long sequences in the field of FSAR is still limited.

[0005] Recently, the Mamba model was proposed and demonstrated superior long sequence modeling capabilities in multiple tasks while avoiding high computational burdens. Mamba uses state space models (SSMs) to replace the complex attention mechanism in Transformer, which can flexibly propagate or discard contextual information. However, Mamba mainly focuses on global feature modeling and ignores the ability to align local features between samples, which is contrary to the requirement of FSAR to fully utilize local features with very few samples.

[0006] Although applying Mamba to long sequence modeling for FSAR has potential, it still faces two major challenges. First, Mamba has shortcomings in local feature modeling and temporal alignment. Some local features are not obvious but crucial. Mamba's emphasis on global features inevitably ignores these important local features, resulting in a decrease in action classification accuracy. In addition, Mamba lacks an effective temporal alignment mechanism and cannot effectively match key local features in different samples, which further affects the performance of FSAR. Second, as the length of subsequences increases, the accumulation of intra-class differences becomes more obvious, which has a negative impact on model performance.

[0007] Among the existing FSAR methods, metric-based meta-learning has become the mainstream paradigm. Through feature extraction, these methods embed support set samples into class prototypes to calculate the distance between query samples and perform classification. However, when processing long sequences, existing methods often ignore the capture and alignment of local features, resulting in poor model performance. In addition, the problem of intra-class difference accumulation is not significant when processing short sequences, but this problem cannot be ignored when using Mamba in long sequences.

[0008] In response to the above observations, we proposed a long sequence action recognition method with small sample learning, which effectively solved the related problems. However, in practical applications, we may still face the problem of increased computational complexity. Although the Mamba model shows good performance when processing long sequences, as the sequence length increases extremely, the computational complexity will also increase accordingly, which becomes an inevitable challenge. In addition, for some application scenarios that require real-time processing, such as video surveillance or online health monitoring, the model may not meet the real-time requirements due to computational complexity and processing time. In this case, it is necessary to find a balance between model performance and real-time performance, or optimize the model to reduce latency. Summary of the invention

[0009] The present invention provides a long sequence action recognition method for small sample learning, aiming to solve the high computational complexity problem faced by traditional Transformer-based methods when processing long sequences, while improving the ability of local feature modeling and time series alignment, and alleviating the negative impact of intra-class difference accumulation on model performance. The present invention mainly includes three core parts: a feature extraction module, a Mamba branch module and a contrastive learning module. In view of the limitations of the traditional Mamba method in small sample action recognition, the present invention proposes a multi-scale Matryoshka Mamba model. The model enhances the expression ability of local features through an internal module and realizes the precise alignment of local features through an external module. In addition, the present invention also proposes a hybrid contrastive learning method to effectively alleviate the negative impact of intra-class difference accumulation of long sequences of the same category on the performance of small sample action recognition. Compared with the prior art, the present invention significantly improves the accuracy and robustness of long sequence action recognition while improving the efficiency of small sample learning.

[0010] In order to achieve the above object, the technical solution of the present invention is as follows: a long sequence action recognition method for small sample learning, characterized in that the method comprises the following steps:

[0011] Step 1: Use a deep neural network as a feature extractor, built using the pytorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips of the support set and the query set into the feature extractor, extract and generate feature vectors with a shape of 2048 × the number of frames, as a preparation for steps 2 and 3.

[0012] Step 2: Input the feature vector generated in step 1 into the Matryoshka Mamba model. The model adopts a multi-scale design as a whole, enhances local features through the inner module, and uses the outer module to align the temporal features to build a class prototype. Then, the final category of the query set is determined based on vector inversion and cross distance calculation.

[0013] Step 3: Based on the results of step 1, supervised contrastive learning is performed on the support set videos, unsupervised contrastive learning is performed on the query set videos, and unsupervised contrastive learning is performed on all videos in the support set and query set. Three loss function results are obtained in these three ways, and their sum is the loss result of contrastive learning.

[0014] Step 4: Multiply the cross entropy loss of the Mamba branch model by the weight and add the loss result of contrastive learning to get the total loss result for the final training target.

[0015] The step 1 is specifically as follows:

[0016] Step 1-1: Construct training set, validation set and test set, which are further divided into support set and query set. One unit of support set is N-wayK-shot task with K labeled samples in each class. The query set has N classified samples without labels. The purpose of classification is to classify the unlabeled query set samples into one class in the support set.

[0017] Step 1-2: Extract features from the support set S and query set Q through the feature extractor. The feature extractor can be instantiated as ResNet50 and implemented by the Pytorch framework. and are the features of the kth video of category c in the support set and the query set Q, respectively. f represents the feature, and f θ is the feature extractor, s and q represent the support set and query set videos respectively, r means that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

[0018]

[0019] The step 2 is specifically as follows:

[0020] The MatryoshkaMamba module adopts a multi-scale architecture. At each scale, multiple inner modules are used to enhance local features. The outputs of the inner modules are connected and then input to the outer module for time alignment. At the same time, the input is combined with the feature vector itself through a two-dimensional convolution operation. After adding, the scale weight is calculated by the sigmoid function. We define the set O as the multi-scale parameter, |O| represents the number of scales, and the element o represents the number of input frames of each internal module at each scale. For simplicity, we use the following table o to represent any scale.

[0021] Step 2-1: In the internal module, use the state space model (SSM) of the linear time-invariant system to transform the input feature x(t)∈R L Replaced with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L The basic working principle is:

[0022] h′(t)=Ah(t)+Bx(t),

[0023] y(t)=Ch(t).

[0024] Among them, A∈R H×H is the state transfer matrix, B,C∈R H is the projection parameter. Local features is a tensor with hidden states, where L = D and H = o. Due to the variability of local features, the inner module includes forward and backward sub-branches IM Fw (·) and IM Bw (·), the two do not share parameters. The final output is:

[0025]

[0026] Where Linear represents linear transformation, Represents vector addition.

[0027] Each local feature is connected in series and added to the input feature to calculate the feature for timing alignment of the external module. and They are also tensors with implicit states, where L = F and H = D. The calculation process is:

[0028]

[0029] Step 2-2: In the external module, considering the symmetry of the timing information, the external module scans OM forward. Fw (·) and backward scanning OM Bw (·) Perform bidirectional feature capture to perceive the dependency between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is expressed as:

[0030]

[0031] Learnable weights and It is used to perform weighted averaging of features of different scales. The weight is derived from the input features of the external module. and its own characteristics Specifically, the calculation steps of the learnable weights are as follows:

[0032]

[0033] Among them, Conv2D Block is defined as CB(·).

[0034] Next, the above learnable weights are combined with the output of the external module to obtain an output representation of any scale and As shown below:

[0035]

[0036] Finally, the output features of the Matryoshka Mamba model and The average of the outputs at each scale is calculated as:

[0037]

[0038] Step 2-3: Support centralized class prototype It is constructed using the following formula:

[0039]

[0040] Step 2-4: In order to further enhance the alignment of time series features, the present invention adopts a cross distance calculation method. Specifically, firstly, the output features and Reverse and get the reversed features and Considering the principle of symmetry alignment, if the distance between tensors sharing the same superscript is small, or the distance between tensors with different superscripts is large, it indicates that they are more likely to belong to the same category. Therefore, the inverse of the distance between tensors with different superscripts is taken.

[0041] The specific distance calculation steps are as follows:

[0042]

[0043]

[0044] Then, the distance between the query set and the cth category of the support set is defined as the average of the above four distances. The type with the smallest distance is the final prediction result of the model, and the prediction result is:

[0045]

[0046] According to the actual category of the query set Y Q is the set of all categories, and the cross entropy loss result is calculated:

[0047]

[0048] The step three is as follows:

[0049] By introducing a hybrid contrastive learning method, the negative impact of intra-class difference accumulation on model performance is reduced. The calculated loss function is used to optimize the model. The specific loss function expression is as follows:

[0050]

[0051] Among them, sim(·,·) represents the cosine similarity and τ is the hyperparameter temperature. In the supervised learning paradigm, for the support set samples, the features z with the same labelp As a positive sample, and R features z with different labels n is regarded as a negative sample. In the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L hc It consists of the following three parts: supervised contrast loss of the support set Unsupervised contrastive loss on the query set And the contrast loss of all samples The final loss function is:

[0052]

[0053] The step 4 is specifically as follows:

[0054] By calculating the loss functions of the Mamba branch module and the contrastive learning module, the final loss function is obtained to train the model:

[0055] L total =λL ce +L hc

[0056] Where λ represents the weight parameter of the cross entropy loss.

[0057] A small sample learning long sequence action recognition system comprises the following modules: a feature extraction module, a multi-scale Matryoshka Mamba model, and a contrast learning module.

[0058] As the front-end module of the system, the feature extraction module mainly converts the original video clips into feature vectors suitable for machine learning processing, providing sufficient data support for subsequent multi-scale processing and comparative learning.

[0059] The multi-scale MatryoshkaMamba model and the contrastive learning module are implemented in parallel.

[0060] The multi-scale MatryoshkaMamba model completes local feature enhancement and global temporal alignment of feature vectors, generates class prototypes for classification, and calculates cross entropy loss. This multi-scale processing method ensures that the system can capture complex spatiotemporal dynamic features and improve the accuracy of action recognition.

[0061] In order to reduce the impact of the accumulation of intra-class differences on model performance, a contrastive learning module is introduced. It further improves the recognition accuracy of the system by reducing intra-class differences and increasing inter-class differences, and at the same time calculates the contrastive learning loss results.

[0062] The final loss result is obtained by multiplying the cross entropy loss of the MatryoshkaMamba module by a weight and adding it to the contrastive learning loss, and the model is trained using this result.

[0063] Compared with the prior art, the advantages of the present invention are as follows:

[0064] 1. Efficient long sequence modeling capability: This paper adopts a multi-scale Matryoshka Mamba model, which effectively improves the ability to capture temporal information in long sequences by decomposing long video sequences into multiple sub-segments and processing them separately, and significantly reduces the computational complexity. Compared with existing Transformer-based methods, this paper can process longer sequences while maintaining lower computing resource consumption.

[0065] 2. Accurate alignment of local features: By introducing a multi-scale design of inner and outer modules, the present invention can accurately align important local features between samples while maintaining the global feature modeling capability. Compared with existing methods, the present invention has significantly improved the recognition accuracy of similar actions, especially in application scenarios that are sensitive to local features. In actual tests, as the training tasks increase, the DTW (Dynamic Time Warping) index gradually decreases, which indicates that the model's time alignment capability has been significantly improved. DTW is a technology used for time series alignment that specifically solves the problem of misalignment of the time dimension in long sequence action recognition. The lower the DTW index, the better the alignment performance of the model between different time series.

[0066] 3. Effective intra-class difference processing: This paper combines supervised and unsupervised contrastive learning strategies and designs a hybrid contrastive learning paradigm to effectively reduce the problem of intra-class difference accumulation caused by shooting conditions or post-processing in long sequences. This method can significantly improve the classification accuracy and robustness of the model when processing long video sequences under different shooting angles or lighting conditions in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0068] Figure 2 It is a schematic diagram of the MatryoshkaMamba module of the present invention;

[0069] Figure 3 This is a schematic diagram of the internal modules of the MatryoshkaMamba module of the present invention;

[0070] Figure 4 It is a schematic diagram of the external module of the MatryoshkaMamba module of the present invention. DETAILED DESCRIPTION

[0071] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0072] Example 1: See Figure 1 , a method for long sequence action recognition with small sample learning, the method comprising the following steps:

[0073] Step 1: Use a deep neural network as a feature extractor, built using the pytorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips of the support set and the query set into the feature extractor, extract and generate feature vectors with a shape of 2048 × the number of frames, as a preparation for steps 2 and 3.

[0074] Step 2: Input the feature vector generated in step 1 into the Matryoshka Mamba model. The model adopts a multi-scale design as a whole, enhances local features through the inner module, and uses the outer module to align the temporal features to build a class prototype. Then, the final category of the query set is determined based on vector inversion and cross distance calculation.

[0075] Step 3: Based on the results of step 1, supervised contrastive learning is performed on the support set videos, unsupervised contrastive learning is performed on the query set videos, and unsupervised contrastive learning is performed on all videos in the support set and query set. Three loss function results are obtained in these three ways, and their sum is the loss result of contrastive learning.

[0076] Step 4: Multiply the cross entropy loss of the Mamba branch model by the weight and add the loss result of contrastive learning to get the total loss result for the final training target.

[0077] The step 1 is specifically as follows:

[0078] Step 1-1: Construct training set, validation set and test set, which are further divided into support set and query set. One unit of support set is N-wayK-shot task with K labeled samples in each class. The query set has N classified samples without labels. The purpose of classification is to classify the unlabeled query set samples into one class in the support set.

[0079] Step 1-2: Extract features from the support set S and query set Q through the feature extractor. The feature extractor can be instantiated as ResNet50 and implemented by the Pytorch framework. and are the features of the kth video of category c in the support set and the query set Q, respectively. f represents the feature, and f θis the feature extractor, s and q represent the support set and query set videos respectively, r means that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

[0080]

[0081] The step 2 is specifically as follows:

[0082] See also Figure 2 The MatryoshkaMamba module adopts a multi-scale architecture. At each scale, multiple inner modules are used to enhance local features. The outputs of the inner modules are connected and then input to the outer module for time alignment. At the same time, the input is combined with the feature vector itself through a two-dimensional convolution operation. After adding, the scale weight is calculated by the sigmoid function. We define the set O as the multi-scale parameter, |O| represents the number of scales, and the element o represents the number of input frames of each internal module at each scale. For simplicity, we use the following table o to represent any scale.

[0083] Step 2-1: See Figure 3 In the internal module, the state space model (SSM) of the linear time-invariant system is used to transform the input feature x(t)∈R L Replaced with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L The basic working principle is:

[0084] h′(t)=Ah(t)+Bx(t),

[0085] y(t)=Ch(t).

[0086] Among them, A∈R H×H is the state transfer matrix, B,C∈R H is the projection parameter. Local features is a tensor with hidden states, where L = D and H = o. Due to the variability of local features, the inner module includes forward and backward sub-branches IM Fw (·) and IM Bw (·), the two do not share parameters. The final output is:

[0087]

[0088] Where Linear represents linear transformation, Represents vector addition.

[0089] Each local feature is connected in series and added to the input feature to calculate the feature for timing alignment of the external module. and They are also tensors with implicit states, where L = F and H = D. The calculation process is:

[0090]

[0091] Step 2-2: See Figure 4 In the external module, considering the symmetry of timing information, the external module scans OM Fw (·) and backward scanning OM Bw (·) Perform bidirectional feature capture to perceive the dependency between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is expressed as:

[0092]

[0093] Learnable weights and It is used to perform weighted averaging of features of different scales. The weight is derived from the input features of the external module. and its own characteristics Specifically, the calculation steps of the learnable weights are as follows:

[0094]

[0095] Among them, Conv2D Block is defined as CB(·).

[0096] Next, the above learnable weights are combined with the output of the external module to obtain an output representation of any scale and As shown below:

[0097]

[0098] in, Represents vector multiplication.

[0099] Finally, the output features of the Matryoshka Mamba model and The average of the outputs at each scale is calculated as:

[0100]

[0101]

[0102] Step 2-3: Support centralized class prototype It is constructed using the following formula:

[0103]

[0104] Step 2-4: In order to further enhance the alignment of time series features, the present invention adopts a cross distance calculation method. Specifically, firstly, the output features and Reverse and get the reversed features and Considering the principle of symmetry alignment, if the distance between tensors sharing the same superscript is small, or the distance between tensors with different superscripts is large, it indicates that they are more likely to belong to the same category. Therefore, the inverse of the distance between tensors with different superscripts is taken.

[0105] The specific distance calculation steps are as follows:

[0106]

[0107] Then, the distance between the query set and the cth category of the support set is defined as the average of the above four distances. The type with the smallest distance is the final prediction result of the model, and the prediction result is:

[0108]

[0109] According to the actual category of the query set Y Q is the set of all categories, and the cross entropy loss result is calculated:

[0110]

[0111] The step three is as follows:

[0112] By introducing a hybrid contrastive learning method, the negative impact of intra-class difference accumulation on model performance is reduced. The calculated loss function is used to optimize the model. The specific loss function expression is as follows:

[0113]

[0114] Among them, sim(· ,·) represents the cosine similarity and τ is the hyperparameter temperature. In the supervised learning paradigm, for the support set samples, the features z with the same label p As a positive sample, and R features z with different labels n is regarded as a negative sample. In the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L hc It consists of the following three parts: supervised contrast loss of the support set Unsupervised contrastive loss on the query set And the contrast loss of all samples The final loss function is:

[0115]

[0116] The step 4 is specifically as follows:

[0117] By calculating the loss functions of the Mamba branch module and the contrastive learning module, the final loss function is obtained to train the model:

[0118] L total =λ Le +L hc

[0119] Where λ represents the weight parameter of the cross entropy loss.

[0120] Specific embodiment: The present invention is mainly trained on a GPU server and tested on an edge device.

[0121] The technical solutions provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementations are only used to illustrate the present invention, but not to limit the scope of the present invention.

[0122] The system of the present invention consists of a server and multiple edge devices. The training set and the validation set are stored on the server side, mainly used for model training and validation; the test set is stored on the edge device and does not need to be uploaded to the server to maintain the privacy and real-time nature of the data. The present invention provides a long sequence action recognition method for small sample learning, which mainly solves the problems of high computational complexity, insufficient local feature modeling, poor temporal alignment, and intra-class difference accumulation in traditional methods when processing long sequences. The core part of the present invention includes three main modules: feature extraction, Matryoshka Mamba module, and contrastive learning module. First, for the problem of small sample action recognition in long video sequences, the present invention analyzes the reasons for the low efficiency of temporal information extraction and improves the shortcomings of traditional methods. Specifically, the present invention introduces multiple internal modules to enhance local features by designing the Matryoshka Mamba model, and realizes the alignment of temporal features through external modules. At the same time, a contrastive learning paradigm is introduced to reduce the impact of inter-class difference accumulation on model performance. When this method is specifically applied, the server first needs to perform preprocessing preparation, which includes the following steps:

[0123] Step 1: Use a deep neural network as a feature extractor, built using the pytorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips of the support set and the query set into the feature extractor, extract and generate feature vectors with a shape of 2048 × the number of frames, as a preparation for steps 2 and 3.

[0124] Step 2: Input the feature vector generated in step 1 into the Matryoshka Mamba model. The model adopts a multi-scale design as a whole, enhances local features through the inner module, and uses the outer module to align the temporal features to build a class prototype. Then, the final category of the query set is determined based on vector inversion and cross distance calculation.

[0125] Step 3: Based on the results of step 1, supervised contrastive learning is performed on the support set videos, unsupervised contrastive learning is performed on the query set videos, and unsupervised contrastive learning is performed on all videos in the support set and query set. Three loss function results are obtained in these three ways, and their sum is the loss result of contrastive learning.

[0126] Step 4: Multiply the cross entropy loss of the Mamba branch model by the weight and add the loss result of contrastive learning to get the total loss result for the final training target.

[0127] The step 1 is specifically as follows:

[0128] Step 1-1: Construct training set, validation set and test set, which are further divided into support set and query set. One unit of support set is N-wayK-shot task with K labeled samples in each class. The query set has N classified samples without labels. The purpose of classification is to classify the unlabeled query set samples into one class in the support set.

[0129] Step 1-2: Extract features from the support set S and query set Q through the feature extractor. The feature extractor can be instantiated as ResNet50 and implemented by the Pytorch framework. and are the features of the kth video of category c in the support set and the query set Q, respectively. f represents the feature, and f θ is the feature extractor, s and q represent the support set and query set videos respectively, r means that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

[0130]

[0131] The step 2 is specifically as follows:

[0132] The MatryoshkaMamba module adopts a multi-scale architecture. At each scale, multiple inner modules are used to enhance local features. The outputs of the inner modules are connected and then input to the outer module for time alignment. At the same time, the input is combined with the feature vector itself through a two-dimensional convolution operation. After adding, the scale weight is calculated by the sigmoid function. We define the set O as the multi-scale parameter, |O| represents the number of scales, and the element o represents the number of input frames of each internal module at each scale. For simplicity, we use the following table o to represent any scale.

[0133] Step 2-1: In the internal module, use the state space model (SSM) of the linear time-invariant system to transform the input feature x(t)∈R L Replaced with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L The basic working principle is:

[0134] h′(t)=Ah(t)+Bx(t),

[0135] y(t)=Ch(t).

[0136] Among them, A∈R H×H is the state transfer matrix, B,C∈R H is the projection parameter. Local features is a tensor with hidden states, where L = D and H = o. Due to the variability of local features, the inner module includes forward and backward sub-branches IM Fw (·) and IM Bw (·), the two do not share parameters. The final output is:

[0137]

[0138] Where Linear represents linear transformation, Represents vector addition.

[0139] Each local feature is connected in series and added to the input feature to calculate the feature for timing alignment of the external module. and They are also tensors with implicit states, where L = F and H = D. The calculation process is:

[0140]

[0141] Step 2-2: In the external module, considering the symmetry of the timing information, the external module scans OM forward. Fw (·) and backward scanning OM Bw (·) Perform bidirectional feature capture to perceive the dependency between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is expressed as:

[0142]

[0143] Learnable weights and It is used to perform weighted averaging of features of different scales. The weight is derived from the input features of the external module. and its own characteristics Specifically, the calculation steps of the learnable weights are as follows:

[0144]

[0145] Among them, Conv2D Block is defined as CB(·).

[0146] Next, the above learnable weights are combined with the output of the external module to obtain an output representation of any scale and As shown below:

[0147]

[0148] Finally, the output features of the Matryoshka Mamba model and The average of the outputs at each scale is calculated as:

[0149]

[0150] Step 2-3: Support centralized class prototype It is constructed using the following formula:

[0151]

[0152] Step 2-4: In order to further enhance the alignment of time series features, the present invention adopts a cross distance calculation method. Specifically, firstly, the output features and Reverse and get the reversed features and Considering the principle of symmetry alignment, if the distance between tensors sharing the same superscript is small, or the distance between tensors with different superscripts is large, it indicates that they are more likely to belong to the same category. Therefore, the inverse of the distance between tensors with different superscripts is taken.

[0153] The specific distance calculation steps are as follows:

[0154]

[0155]

[0156] Then, the distance between the query set and the cth category of the support set is defined as the average of the above four distances. The type with the smallest distance is the final prediction result of the model, and the prediction result is:

[0157]

[0158] According to the actual category of the query set The cross entropy loss is calculated as:

[0159]

[0160] The step three is as follows:

[0161] By introducing a hybrid contrastive learning method, the negative impact of intra-class difference accumulation on model performance is reduced. The calculated loss function is used to optimize the model. The specific loss function expression is as follows:

[0162]

[0163] Among them, sim(·,·) represents the cosine similarity and τ is the hyperparameter temperature. In the supervised learning paradigm, for the support set samples, the features z with the same label p As a positive sample, and R features z with different labels n is regarded as a negative sample. In the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L hc It consists of the following three parts: supervised contrast loss of the support set Unsupervised contrastive loss on the query set And the contrast loss of all samples The final loss function is:

[0164]

[0165] The step 4 is specifically as follows:

[0166] By calculating the loss functions of the Mamba branch module and the contrastive learning module, the final loss function is obtained to train the model:

[0167] L total =λL ce +L hc

[0168] Where λ represents the weight parameter of the cross entropy loss.

[0169] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention. These corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for long sequence action recognition based on small sample learning, characterized in that: The method comprises the following steps: Step 1: Use a deep neural network as a feature extractor, built using the pytorch framework. The feature extractor can be an instantiated ResNet50 network. Input the video clips of the support set and the query set into the feature extractor respectively, extract and generate feature vectors with a shape of 2048×number of frames, as a preparation for steps 2 and 3. Step 2: Input the feature vector generated in step 1 into the Matryoshka Mamba model. The model adopts a multi-scale design as a whole. It enhances local features through internal modules and uses external modules to align temporal features to build a class prototype. Then, based on vector inversion and cross distance calculation, the final category of the query set is determined. Step 3: Based on the results of step 1, supervised contrastive learning is performed on the support set videos, unsupervised contrastive learning is performed on the query set videos, and unsupervised contrastive learning is performed on all videos in the support set and query set. Three loss function results are obtained in three ways, and their sum is the loss result of contrastive learning. Step 4: Multiply the cross entropy loss of the Mamba branch model by the weight and add the loss result of contrastive learning to get the total loss result for the final training target.

2. The method for long sequence action recognition based on small sample learning according to claim 1, characterized in that: The step 1 is specifically as follows: Step 1-1: Construct training set, validation set and test set. The training set, validation set and test set are further divided into support set and query set. A unit of the support set is a task with N classifications and K labeled samples per class (N-wayK-shot). The query set has N classified samples without labels. The purpose of classification is to classify the unlabeled query set samples into a class in the support set. Step 1-2: Extract features from the support set S and query set Q through the feature extractor. The feature extractor can be instantiated as ResNet50 and implemented by the Pytorch framework. and are the features of the kth video of category c in the support set and the query set Q, respectively. f represents the feature, and f θ is the feature extractor, s and q represent the support set and query set videos respectively, r indicates that the video frames are randomly selected, and F represents the number of frames, where the support set video frames are uniformly sampled and the query set video frames are randomly sampled.

3. The method for long sequence action recognition based on small sample learning according to claim 1, characterized in that: The step 2 is specifically as follows: The MatryoshkaMamba module adopts a multi-scale architecture. At each scale, multiple inner modules are used to enhance local features. The outputs of the inner modules are connected and input to the outer module for time alignment. At the same time, the input is combined with the feature vector itself through a two-dimensional convolution operation. After addition, the scale weight is calculated by the sigmoid function. The set O is specified as a multi-scale parameter, |O| represents the number of scales, and the element o represents the number of input frames of each internal module at each scale. The following table o is used to represent any scale. Step 2-1: In the internal module, the input feature x(t)∈R is transformed using the state space model (SSM) of the linear time-invariant system L Replaced with output feature y(t)∈R L Through the intermediate hidden state h(t)∈R L , its basic working principle is: h ′ (t)=Ah(t)+Bx(t), y(t)=Ch(t), Among them, A∈R H×H is the state transfer matrix, B,C∈R H is the projection parameter, local features is a tensor with hidden states, where L = D, H = o. Due to the variability of local features, the inner module includes forward and backward sub-branches IM Fw (·) and IM Bw (·), the two do not share parameters, and the final output is: Where Linear represents linear transformation, represents vector addition, Each local feature is connected in series and added to the input feature to calculate the feature for timing alignment of the external module. and They are also tensors with implicit states, where L = F, H = D, and the calculation process is: Step 2-2: In the external module, considering the symmetry of the timing information, the external module scans OM forward. Fw (·) and backward scanning OM Bw (·) Perform bidirectional feature capture to perceive the dependency between local features and achieve implicit temporal alignment. After bidirectional scanning, the fused output is expressed as: Learnable weights and It is used to perform weighted averaging of features of different scales, where the weights are derived from the input features of the external module. and its own characteristics Specifically, the calculation steps of the learnable weights are as follows: Among them, Conv2D Block is defined as CB(·), Next, the above learnable weights are combined with the output of the external module to obtain an output representation of any scale and As shown below: Finally, the output features of the Matryoshka Mamba model and The average of the outputs at each scale is calculated as: Step 2-3: Support centralized class prototype It is constructed using the following formula: Step 2-4: In order to further enhance the alignment of time series features, the present invention adopts a cross distance calculation method. Specifically, firstly, the output features and Reverse and get the reversed features and Considering the principle of symmetry alignment, if the distance between tensors sharing the same superscript is small, or the distance between tensors with different superscripts is large, it indicates that they are more likely to belong to the same category. Therefore, the inverse of the distance between tensors with different superscripts is taken. The specific distance calculation steps are as follows: Then, the distance between the query set and the cth category of the support set is defined as the average of the above four distances. The type with the smallest distance is the final prediction result of the model, and the prediction result is: According to the actual category of the query set Y Q is the set of all categories, and the cross entropy loss result is calculated:

4. The method for long sequence action recognition based on small sample learning according to claim 1, characterized in that: The step three is as follows: By introducing a hybrid contrastive learning method, the negative impact of intra-class difference accumulation on model performance is reduced. The calculated loss function is used to optimize the model. The specific loss function expression is as follows: Among them, sim(·,·) represents the cosine similarity, τ is the hyperparameter temperature, and in the supervised learning paradigm, for the support set samples, the features z with the same label p As a positive sample, and R features z with different labels n As negative samples, in the unsupervised learning paradigm, no label guidance is provided for the query set and all samples. The final hybrid contrastive learning loss function L hc It consists of the following three parts: supervised contrast loss of the support set Unsupervised contrastive loss on the query set And the contrast loss of all samples The final loss function is:

5. The method for long sequence action recognition based on small sample learning according to claim 1, characterized in that: The step 4 is specifically as follows: By calculating the loss functions of the Mamba branch module and the contrastive learning module, the final loss function is obtained to train the model: THE total =λL ce +L hc Where λ represents the weight parameter of the cross entropy loss.

6. A long sequence action recognition system with small sample learning, characterized in that: The system includes the following modules: feature extraction module, multi-scale Matryoshka Mamba model and contrastive learning module. As the front-end module of the system, the feature extraction module mainly converts the original video clips into feature vectors suitable for machine learning processing, providing sufficient data support for subsequent multi-scale processing and comparative learning. The multi-scale MatryoshkaMamba model and contrastive learning module are parallelized. The multi-scale MatryoshkaMamba model completes the local feature enhancement and global temporal alignment of the feature vector, generates class prototypes for classification, and calculates the cross entropy loss. The contrastive learning module is designed to reduce the impact of the accumulation of intra-class differences on model performance. It further improves the recognition accuracy of the system by reducing intra-class differences and increasing inter-class differences. At the same time, it calculates the contrastive learning loss result. The final loss result is obtained by multiplying the cross entropy loss of the MatryoshkaMamba model by a weight and adding it to the contrastive learning loss, and the model training is completed using this result.

Citation Information

Patent Citations

  • Human body posture image intelligent identification method and system

    CN115527269A

  • Small sample action recognition network, method and equipment based on meta learning and time alignment and storage medium

    CN116486475A

  • Small sample action recognition method based on long and short time sequence feature enhancement and key frame feature extraction

    CN117894079A

  • Few-shot gesture recognition method

    US20230326249A1

  • Small sample remote sensing image scene classification method based on embedding smoothing graph neural network

    WO2023087558A1

Cited By

  • Long-term time sequence prediction method of double-path Mama for data processing

    CN119557605A

  • Spatial and temporal feature enhancement method for infrared video denoising

    CN120655531A

  • Method and device for recommending POI at specific time and storage medium

    CN121278190A

  • Sewage treatment blower fault prediction method based on multi-scale Mangbar model

    CN121278505A

  • Gas transient signal rapid detection method based on Mama and ordinal number metric learning

    CN121561352A