Small sample action recognition method based on space and long short-term time sequence feature modeling

Abstract

The application relates to the field of computer vision, and discloses a small sample action recognition method based on space and long and short term time sequence feature modeling, constructs a space feature fusion architecture search unit to automatically search the best fusion mode of shallow and deep space features, so as to improve and enhance the space representation; a long and short term time sequence feature modeling unit is constructed to encode complementary global and local time sequence representations, so as to enhance the expression ability of the time sequence features. The application uses a cross attention mechanism to complete class prototype establishment and matching, inputs the integrated rich space-time enhanced features into a frame level class prototype establishment and matching unit, so that the system can accurately and quickly recognize target actions in a small amount of labeled sample tasks.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a few-sample action recognition method based on spatial and long- and short-term temporal feature modeling. Background Technology

[0002] With the development of deep learning, a wealth of excellent work has emerged in the field of action recognition. Most studies use large amounts of labeled data to perform video understanding or classification tasks to learn video representations. However, this approach is unsatisfactory in industrial applications due to the large amount of time-consuming and labor-intensive data annotation. Conversely, the core assumption of few-shot learning is to use only a small number of labeled training samples from numerous similar tasks as a substitute for a large number of labeled training samples. Therefore, there is increasing interest in few-shot learning methods. The task of few-shot action recognition is to classify unlabeled query videos into one action category from a support set (typically five categories), where the training samples for each action category are limited.

[0003] Existing mainstream few-shot image classification methods can be broadly categorized into two types: adaptation-based and metric-based. Adaptation-based methods aim to find a network initialization that can be fine-tuned using a small amount of data for an unknown task; this is known as gradient-by-gradient. The application of adaptation-based methods can be clearly seen in the MAML and Reptile examples. Metric-based methods aim to find a fixed feature representation into which the target task can be embedded for classification. The effectiveness of this method has been demonstrated in Prototypical Networks and Matching Networks. Furthermore, CrossTransformer combines metric-based features with task-specific adaptability to match queries with support sets.

[0004] Inspired by few-shot image classification, existing few-shot video action recognition methods mainly rely on metric-based approaches to compare the similarity of different videos in the feature space for identification. However, adaptation-based methods require significant computational resources and lengthy experimental time. Compared to images, videos have an additional temporal dimension, making it insufficient to represent the entire video as a single feature vector. Therefore, modeling spatiotemporal features has become a core problem in few-shot action recognition. Specifically, spatial features aim to express the spatial semantic information of each single frame. In some cases, a video can be identified using only one frame, and current methods such as STRM and TRX typically extract spatial features through TSN models. However, they often only use deep spatial features, neglecting equally crucial shallow spatial features, which contain rich texture information. Fusing shallow and deep spatial features can compensate for or even highlight shallow semantic features. For temporal features, we categorize them into two types: long-term and short-term. Long-term temporal features reveal the relationships between spatial appearance features at different timestamps, a topic that has been a focus of previous work. Existing methods such as OTAM and CMN primarily acquire long-term temporal features through manually designed temporal alignment algorithms during class prototype construction, aiming to obtain better global features for comparison. On the other hand, short-term temporal features represent the motion characteristics of adjacent frames, focusing on modeling local temporal relationships. Short-term temporal modeling is also quite important, but we observe that relevant research on short-term temporal models in the field of few-sample action recognition has not yet been explored. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a few-sample action recognition method based on spatial and long- and short-term temporal feature modeling, which enables accurate and rapid action recognition during the inference stage in real-world scenarios with limited labeled training data.

[0006] To achieve the above objectives, this invention provides a few-sample action recognition method based on spatial and long- and short-term temporal feature modeling, comprising the following steps:

[0007] S1. Construct a spatial feature fusion architecture search unit; by utilizing three different feature fusion operations and automatically searching for the relative weights of different operations, the output features of different layers of the feature extractor are fused together, enhancing the spatial representation of single-frame features.

[0008] S2. Construct long and short-term temporal feature enhancement units; by using long and short-term temporal attention mechanisms, the global and local temporal representation capabilities of video features are enhanced.

[0009] S3. Construct class prototype establishment and matching units using cross-attention mechanism; use spatiotemporal enhancement features to establish class prototypes of query videos and support set videos based on cross-attention mechanism, and match the class prototypes of query videos and support set videos.

[0010] S4. Train the network model. Use the server to train the network model on multiple similar small sample tasks. Optimize the network parameters by reducing the network loss function until the network converges, and obtain a small sample action recognition method based on spatial and long-term and short-term temporal features.

[0011] S5. Use the few-sample action recognition method based on spatial and long-term / short-term temporal features to identify target actions in new few-sample tasks.

[0012] Preferably, step S1 specifically includes the following steps:

[0013] S11. To facilitate feature fusion between shallow and deep layers and achieve spatial feature alignment of each layer of the feature extractor; specifically: align the features of layer L output from the feature extractor... Spatial alignment is performed and the data is stored in the feature library, where... These are the batch size, time, space, height, and width, respectively; the spatial and channel dimensions of each layer's features are aligned with the features of the last layer, specifically:

[0014]

[0015] in It is Convolutional layer;

[0016] S12. The features from different layers output by the feature extractor are fused with the automatic search and fusion module, enabling it to find the optimal combination of shallow and deep spatial features in different scenarios. Specifically:

[0017]

[0018] in It is the weighted sum of the features of layers i and j after all optional fusion operations; the set of fusion schemes is represented as... Three parameter-free fusion schemes are provided. , and Assign a weight to each operation. and carry out Operations make the search space continuous; simplify the search task to learning weights. ,and It can be calculated as follows:

[0019] ;

[0020] S13. Enhance the spatial representation capability of single-frame features; specifically: extract the last layer feature updated in the spatial feature fusion architecture search unit as... ; Characteristics of fusion The final features output by the feature extractor The weighted summation can be performed, and the learnable parameters are: Specifically:

[0021] .

[0022] Preferably, the three parameter-free fusion schemes in step S12 , and Specifically:

[0023]

[0024]

[0025]

[0026] in The features are the i-th and j-th layers in the feature library, and , For Max_Pooling, max pooling operation, This is the sigmoid function.

[0027] Preferably, step S2 specifically includes the following steps:

[0028] S21. Construct a long-term temporal modeling module; model the global temporal relationship based on the extracted spatial appearance features; represent the spatially enhanced video feature map as follows: ,Will Remodel into a sequence ,Will Self-attention is correlated over time, specifically as follows:

[0029]

[0030] in This is represented by an L-layer multi-head attention mechanism; then, a residual feedforward network is used to refine the obtained features in a targeted manner to obtain long-term temporal features. Specifically:

[0031]

[0032] in This represents hierarchical normalization, and this represents a multilayer perceptron; [the following is a list of characters and their meanings, which are not translated as they are not part of the main text] It is reshaped back to the original input shape (i.e., [N,T,C,H,W]).

[0033] S22. Construct a short-term temporal modeling module; encode motion information between adjacent frames at the feature level; given a spatially augmented video feature map. Use learnable weights The three key elements of a query key-value pair are:

[0034]

[0035] Remodeling The number of channels was reduced by a factor of r to alleviate computational costs, and two [channels] were utilized. Channel convolution and Effect on Above, specifically:

[0036]

[0037] in Two spatial dimensions representing channels and feature maps; and express Filter, subscript Represents the space coefficients of the kernel; in and Interleaved subtraction is performed along the time dimension to obtain motion information at the feature level, that is... and Between, specifically:

[0038]

[0039] Among them, for Frame video, Motion representation The time dimension is Using 0 to represent the motion information of the last time step helps Maintain time size compatible with input feature maps; Reshape to the shape of the original input features With the number of recovery channels as C, a feedforward network (FFN) is applied to motion attention. The final output is obtained and calculated as follows:

[0040] ;

[0041] S23. Perform short-term time series characteristics. and long-term time series characteristics The weighted summation yields enhanced time-series characteristics. The learnable parameters are Specifically:

[0042] .

[0043] Preferably, step S3 specifically includes the following steps:

[0044] S31. Construct a frame-level feature representation of the video. represent Frame feature representation, the sequence of frame i and frame j Represented as ,in And so on; for any tuple Gather a support video for action-related content. All possible subsequences are used to compute a class prototype for a specific query, where the aggregation weight is based on the cross attention of the query subsequence and the supporting class subsequence;

[0045] S32, The query video features are represented as follows: Specific query class features are represented as Calculate the query video With support set The distance D of one category across multiple candidate combinations is specifically:

[0046] .

[0047] Preferably, step S4 specifically includes the following steps:

[0048] S41. Utilize the server to execute the training video segment generation unit, generating a video segment with a length of [length missing]. The training video clips, in which ;

[0049] S42. Utilize the server to execute the spatial feature fusion architecture search unit to query the video. and class support video set The feature extractor extracts the output features of each layer and stores them in a feature library. The features from the feature library are then input into the spatial feature fusion architecture search unit to obtain spatial fusion features. and ; using learnable parameters The enhanced spatial features are obtained by weighted summation of the fused features and the original features from the last layer of the feature library.

[0050] S43. The server executes the long-term and short-term time series feature enhancement unit. After obtaining spatial features, the temporal relationship is modeled to obtain better spatiotemporal integration features. The enhanced spatial features are then processed by the long-term time series modeling module and the short-term time series modeling module to enhance the long-term and short-term time series features. Simultaneous modeling, using learnable parameters This parameter can adaptively fuse two types of time series features, combining long-term and short-term time series features.

[0051] S44. Utilize the server to execute the class prototype creation and matching unit to obtain the category prediction of the query video Q. and loss function ;

[0052] S45. Utilize a server for end-to-end network training; loss function for few-shot action recognition task. Distance over multiple candidate combinations ,Will This is minimized by a standard cross-entropy loss, which is the loss from the predicted category of the query video to its true category.

[0053] S46. Use the server to optimize the objective function to obtain the local optimal network parameters as the network weights for modeling the network's spatial features and long and short-term temporal features.

[0054] Preferably, step S41 specifically involves: processing 8 frames of a video... Uniform sampling was performed on the input image, and during training, the input image was randomly horizontally flipped and... Random cropping.

[0055] Preferably, the learnable parameters The initial value is set to 0.9, which is the learnable parameter. The initial value is set to 0.5;

[0056] Preferably, step S5 specifically includes the following steps:

[0057] S51. Utilize the server to execute the test video clip generation unit, generating 8 frames from a video. Uniform sampling was performed as input, and the input image was center-cropped during testing;

[0058] S52. Using the few-sample action recognition method based on spatial and short-term temporal features, the query video is assigned to the one closest to the query. The category, that is .

[0059] Compared with the prior art, the beneficial effects of the present invention are:

[0060] This invention provides a few-sample action recognition method based on spatial and long- and short-term temporal feature modeling. It integrates spatial feature modeling and long- and short-term temporal feature modeling into a unified framework: a spatial feature fusion architecture search unit is proposed to automatically search for the optimal combination of low-level and high-level spatial features to enhance the expressive power of spatial features; a long-term temporal modeling unit is introduced to model global temporal relationships based on extracted spatial appearance features; a short-term temporal modeling module is proposed to encode motion features between adjacent frames; and the integrated rich spatiotemporal features are input to a frame-level class prototype establishment and matching unit, enabling the system to accurately and quickly identify target actions in a small number of labeled sample tasks. In the 5-way 5-shot task, the recognition accuracy reaches 87.0% on the Kinetics test set, 68.3% on the SSV2 test set, 77.5% on the HMDB51 test set, and 97.1% on the UCF101 test set. Attached Figure Description

[0061] Figure 1 A spatial feature fusion architecture search unit diagram constructed for this invention;

[0062] Figure 2 This is a unit diagram of the long-term time series modeling module constructed for this invention;

[0063] Figure 3 A unit diagram of the short-term time series modeling module constructed for this invention;

[0064] Figure 4 This is an algorithm framework diagram of the few-sample action recognition method based on spatial and long- and short-term temporal features of the present invention;

[0065] Figure 5 This is a visualization of the attention results for target action recognition during the inference phase of the model in this invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] To address the problems and shortcomings of existing technologies, this invention proposes a small-sample action recognition method based on a comprehensive network modeled with spatial features and long- and short-term temporal features. The method mainly includes five stages: constructing a spatial feature fusion architecture search unit, constructing a long- and short-term temporal feature enhancement unit, constructing a class prototype establishment and matching unit, model training, and model inference.

[0068] The few-sample action recognition method proposed in this invention, based on spatial and long- and short-term temporal feature modeling, includes the following steps:

[0069] S1. Construct a spatial feature fusion architecture search unit;

[0070] S2. Construct long and short-term time series feature enhancement units;

[0071] S3. Construct class prototype creation and matching units using cross-attention mechanism;

[0072] S4. Train the network model and optimize the network parameters by reducing the network loss function until the network converges to obtain a small sample action recognition method based on spatial and long- and short-term temporal features.

[0073] S5. Use the few-sample action recognition method based on spatial and long-term / short-term temporal features to identify target actions in new few-sample tasks.

[0074] The following is a detailed explanation of each step.

[0075] Step S1: Construct a spatial feature fusion architecture search unit. For example... Figure 1 The diagram shows the spatial feature fusion architecture search unit constructed in this invention. To utilize shallow spatial features, a feature fusion structure search unit was designed to automatically search for the optimal fusion structure of shallow and deep spatial features in different scenarios. The search process is as follows: Figure 1 As shown in (b). In addition, three parameterless feature fusion methods are provided as follows: Figure 1 As shown in (a). The specific implementation process is as follows:

[0076] S11. To facilitate feature fusion between shallow and deep layers and achieve spatial feature alignment of each layer of the feature extractor; the features of layer L output from the feature extractor are... Spatial alignment is performed and the data is stored in the feature library, where... These are the batch size, time, space, height, and width, respectively; the spatial and channel dimensions of each layer's features are aligned with the features of the last layer, specifically:

[0077]

[0078] in It is Convolutional layer;

[0079] S12. The features from different layers output by the feature extractor are fused with the automatic search and fusion module, enabling it to find the optimal combination of shallow and deep spatial features in different scenarios. Specifically:

[0080]

[0081] in It is the weighted sum of the features of layers i and j after all optional fusion operations; the set of fusion schemes is represented as... Three parameter-free fusion schemes are provided. , and Assign a weight to each operation. and carry out Operations make the search space continuous; simplify the search task to learning weights. ,and It can be calculated as follows:

[0082] ;

[0083] S13. Enhance the spatial representation capability of single-frame features; specifically: extract the last layer feature updated in the spatial feature fusion architecture search unit as... ; Characteristics of fusion The final features output by the feature extractor The weighted summation can be performed, and the learnable parameters are: Specifically:

[0084] .

[0085] Among them, the three parameter-free fusion schemes in step S12 , and Specifically:

[0086]

[0087]

[0088]

[0089] in The features are the i-th and j-th layers in the feature library, and , For Max_Pooling, max pooling operation, This is the sigmoid function.

[0090] This invention constructs a spatial feature fusion architecture search unit to automatically search for the optimal fusion method between shallow and deep spatial features. Shallow features focus more on structural and textural information, while deep features focus on semantic information; their fusion helps improve spatial representation. This invention provides three parameter-free feature fusion methods, such as... Figure 1 As shown in (a), the feature search process of the present invention is as follows: Figure 1 As shown in (b), the weights of each fusion operation are initialized on average during the search process, and the optimal weights of each fusion operation are obtained after the search is completed.

[0091] S2. Construct long- and short-term time series feature enhancement units; such as... Figure 2 The diagram shown is a unit diagram of the long-term time series modeling module constructed in this invention. Figure 3 The diagram shown is a unit diagram of the short-term time series modeling module constructed in this invention. The specific implementation process is as follows:

[0092] S21. Construct a long-term temporal modeling module; model the global temporal relationship based on the extracted spatial appearance features; represent the spatially enhanced video feature map as follows: ,Will Remodel into a sequence ,Will Self-attention is correlated over time, specifically as follows:

[0093]

[0094] in This is represented by an L-layer multi-head attention mechanism; then, a residual feedforward network is used to refine the obtained features in a targeted manner to obtain long-term temporal features. Specifically:

[0095]

[0096] in This represents hierarchical normalization, and this represents a multilayer perceptron; [the following is a list of characters and their meanings, which are not translated as they are not part of the main text] It is reshaped back to the original input shape (i.e., [N,T,C,H,W]).

[0097] S22. Construct a short-term temporal modeling module; encode motion information between adjacent frames at the feature level; given a spatially augmented video feature map. Use learnable weights The three key elements of a query key-value pair are:

[0098]

[0099] Remodeling The number of channels was reduced by a factor of r to alleviate computational costs, and two [channels] were utilized. Channel convolution and Effect on Above, specifically:

[0100]

[0101] in Two spatial dimensions representing channels and feature maps; and express Filter, subscript Represents the space coefficients of the kernel; in and Interleaved subtraction is performed along the time dimension to obtain motion information at the feature level, that is... and Between, specifically:

[0102]

[0103] Among them, for Frame video, Motion representation The time dimension is Using 0 to represent the motion information of the last time step helps Maintain time size compatible with input feature maps; Reshape to the shape of the original input features With the number of recovery channels as C, a feedforward network (FFN) is applied to motion attention. The final output is obtained and calculated as follows:

[0104] ;

[0105] S23. Perform short-term time series characteristics. and long-term time series characteristics The weighted summation yields enhanced time-series characteristics. The learnable parameters are Specifically:

[0106] .

[0107] This invention constructs a long short-term temporal modeling unit based on an attention mechanism to encode complementary global and local temporal representations. In the field of few-shot action recognition, many objects move over time, thus many actions can be classified based on their global temporal context information. Therefore, we employ a long-term temporal modeling module to model global temporal relationships based on extracted spatial appearance features, such as... Figure 2As shown. Furthermore, the classification of many action categories requires short-term temporal information, representing the motion features of adjacent frames, which is beneficial for identifying many time-related actions. Therefore, we propose a novel short-term temporal modeling module that encodes the motion information between adjacent frame representations at the feature level, such as... Figure 3 As shown. Furthermore, to fuse long-term and short-term time-series features, a learnable parameter is used to adaptively fuse the two time-series features.

[0108] Step S3: Construct a class prototype creation and matching unit using a cross-attention mechanism. The specific implementation process is as follows:

[0109] S31. Construct a frame-level feature representation of the video. represent Frame feature representation, the sequence of frame i and frame j Represented as ,in And so on; for any tuple Gather a support video for action-related content. All possible subsequences are used to compute a class prototype for a specific query, where the aggregation weight is based on the cross attention of the query subsequence and the supporting class subsequence;

[0110] S32, The query video features are represented as follows: Specific query class features are represented as Calculate the query video With support set The distance D of one category across multiple candidate combinations is specifically:

[0111] .

[0112] This invention explores this method by constructing and matching class prototypes of query videos and supporting video sets. Spatiotemporal augmentation features are input into this class prototype matcher to obtain the class prediction of the query video Q. and loss function .

[0113] Step S4: Train the network model by optimizing the network parameters by reducing the network loss function until the network converges, thus obtaining a few-sample action recognition method based on spatial and long- and short-term temporal features. Figure 4 The diagram shows the algorithm framework of the few-sample action recognition method based on spatial and long- and short-term temporal feature modeling of the present invention. The specific implementation process is as follows:

[0114] S41. Utilize the server to execute the training video segment generation unit, generating a video segment with a length of [length missing]. The training video clips, in which ;

[0115] S42. Utilize the server to execute the spatial feature fusion architecture search unit to query the video. and class support video set The feature extractor extracts the output features of each layer and stores them in a feature library. The features from the feature library are then input into the spatial feature fusion architecture search unit to obtain spatial fusion features. and ; using learnable parameters The enhanced spatial features are obtained by weighted summation of the fused features and the original features from the last layer of the feature library.

[0116] S43. The server executes the long-term and short-term time series feature enhancement unit. After obtaining spatial features, the temporal relationship is modeled to obtain better spatiotemporal integration features. The enhanced spatial features are then processed by the long-term time series modeling module and the short-term time series modeling module to enhance the long-term and short-term time series features. Simultaneous modeling, using learnable parameters This parameter can adaptively fuse two types of time series features, combining long-term and short-term time series features.

[0117] S44. Utilize the server to execute the class prototype creation and matching unit to obtain the category prediction of the query video Q. and loss function ;

[0118] S45. Utilize a server for end-to-end network training; loss function for few-shot action recognition task. Distance over multiple candidate combinations ,Will This is minimized by a standard cross-entropy loss, which is the loss from the predicted category of the query video to its true category.

[0119] S46. Optimize the objective function using the server, wherein the objective function adopts the loss function from step S45. The local optimal network parameters are obtained as the network weights for modeling the integrated network based on spatial features and long-term and short-term temporal features.

[0120] Specifically, step S41 involves taking 8 frames from a video... Uniform sampling was performed on the input image, and during training, the input image was randomly horizontally flipped and... Random cropping.

[0121] Among them, learnable parameters The initial value is set to 0.9, which is the learnable parameter. The initial value is set to 0.5.

[0122] Step S5: Identify the target action in a new few-shot task using the few-shot action recognition method based on spatial and long- and short-term temporal features. The specific implementation process is as follows:

[0123] S51. Utilize the server to execute the test video clip generation unit, generating 8 frames from a video. Uniform sampling was performed as input, and the input image was center-cropped during testing;

[0124] S52. Using the few-sample action recognition method based on spatial and short-term temporal features, the query video is assigned to the one closest to the query. The category, that is .

[0125] This invention visualizes the attention results of each unit during the target action recognition phase of the model inference stage. For example... Figure 5 The image shows the attention visualization results of the target action recognition during the inference stage of the model in this invention, which progressively integrates the influence of each unit on the action recognition results. After integrating the spatial feature fusion architecture search unit (third row), the framework enhances the feature space representation, which helps to focus attention on relevant objects in a single video frame, for example, Figure 5 The frames in (a) and (b) reduce attention to the background and irrelevant objects. Furthermore, after integrating the long and short-term temporal feature modeling units (fourth row), the frame enhances the temporal relationships of the features, which makes our synthetic network highly relevant to the action subject. For example, the fourth and eighth frames from the left in (a) have better attention on the snowman, and the frames in (b) have more detailed attention extended to the marker pen.

[0126] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A small sample action recognition method based on spatial and long short-term temporal feature modeling, characterized in that, The method comprises the following steps: S1, constructing a spatial feature fusion architecture search unit for automatically searching for an optimal combination of low-level and high-level spatial features; S2, constructing a long-short-term time sequence feature enhancement unit; S3, constructing a prototype-like establishment and matching unit by using a cross-attention mechanism; S4, training the network model, optimizing the network parameters by reducing the network loss function, until the network converges, and obtaining a small sample action recognition method based on spatial and long-short-term time sequence feature modeling; S5, using the small sample action recognition method based on spatial and long-short-term time sequence feature modeling to identify target actions in new small sample tasks; The step S2 specifically comprises the following steps: S21, construct long-term time modeling module; model global time relationship according to extracted spatial appearance features; represent spatial enhanced video feature map as , and , reshape into a sequence , and , do self-attention correlation in time dimension, specifically: ； Wherein Multi-head attention mechanism represented as L layer; then the obtained features are refined by a residual feedforward network for directionality refinement to obtain long-term timing features Specifically: ； wherein denotes a hierarchical normalization, denotes a multi-layer perceptron; will be reshaped to the original input shape, i.e. [N, T, C, H, W]; S22, constructing a short-term time series modeling module; encoding motion information between adjacent frames at a feature level; given a spatially enhanced video feature map using learnable weights The query-key-value three-element is specifically obtained as follows: ； remodeling reducing the number of channels by a factor of r and convolving the two channel convolutions and acting on in particular ； wherein represent two spatial dimensions of a channel and a feature map; and denote a filter, subscript denote spatial coefficients of a kernel; in and interleaved subtraction is done over the temporal dimension between and , specifically: ； Among them, for Frame video, Motion representation The time dimension is Using 0 to represent the motion information of the last time step helps Maintain time size compatible with input feature maps; Reshape to the shape of the original input features With the number of recovery channels as C, a feedforward network (FFN) is applied to motion attention. The final output is obtained and calculated as follows: ； S23, a weighted sum of short-term timing features and long-term timing features , to obtain reinforced timing features , and the learnable parameters are , specifically: ； The step S3 specifically comprises the following steps: S31, constructing a frame-level feature representation of a video, representing frame feature representation, sequence of i-th frame and j-th frame denoted as wherein, and so on; for any tuple , aggregating support videos of an action class all possible sub-sequences in the support class to compute a class prototype for a particular query, wherein the aggregation weights are based on cross-attention between the query sub-sequence and the support class sub-sequence; S32, query video feature representation is , specific query class feature representation is , calculate the distance D of the query video with one category in the support set on multiple candidate combinations, specifically: ； The step S5 specifically comprises the following steps: S51, using the server to execute a test video segment generation unit, 8 frames of a video, Uniform sampling is performed as input, and center cropping is performed on the input picture in the test; S52, using the small sample action recognition method based on spatial and long short-term temporal feature modeling, the query video is assigned to the class closest to the query, that is . .

2. The small sample action recognition method based on space and long short-term timing feature modeling according to claim 1, characterized in that, The step S1 specifically comprises the following steps: S11, aligning the features of the Lth layer output from the feature extractor aligning and storing in the feature library, wherein respectively, batch size, time, space, height, and width; aligning the space and channel dimensions of the features of each layer with the features of the last layer, specifically: ； wherein is convolutional layer; S12, fusing the features of different layers output by the feature extractor with the automatic search fusion module, specifically: ； where is the weighted sum of the features of layers i and j after all optional fusion operations; the set of fusion schemes is denoted as , three parameter-free fusion schemes are provided , and ; a weight is assigned to each operation and the operation is performed to make the search space continuous; the search task is simplified to learning the weights , while is calculated as: ； S13, the last layer of features updated in the spatial feature fusion architecture search unit is taken out as ; the fused features and the final features output by the feature extractor are weighted and summed, and the learnable parameters are , specifically: 。 3. The small sample action recognition method based on space and long short-term timing feature modeling according to claim 2, characterized in that, The three parameter-free fusion schemes in the step S12 , and Specifically: ； ； ； wherein is the i-th layer and j-th layer feature in the feature library, and , is a Max_Pooling max pooling operation, is a sigmoid function. 4.The small sample action recognition method based on space and long short term timing feature modeling according to claim 2, characterized in that, The step S4 specifically comprises the following steps: S41, using the server to execute a training video segment generation unit to generate a training video segment with a length of , wherein ; S42, using the server to execute the spatial feature fusion architecture search unit to search for a query video and class support set videos through the feature extractor, and store the output features of each layer in a feature library; input the features in the feature library to the spatial feature fusion architecture search unit to obtain spatial fusion features and ; use learnable parameters to perform weighted summation on the fused features and the original features of the last layer in the feature library to obtain enhanced spatial features; S43, performing the long short-term temporal feature enhancement unit with the server, passing the enhanced spatial features through the long-term temporal modeling module and the short-term temporal modeling module to long-term and short-term temporal features Simultaneously modeling with learnable parameters Fusing long short-term temporal features; S44, using the server to perform the class prototype establishing and matching unit to obtain the category prediction of the query video Q and loss function ; S45, network training in an end-to-end manner using a server; small sample action recognition task loss function for distances on multiple candidate combinations , the is minimized by a standard cross-entropy loss; S46, using the server to optimize the objective function to obtain local optimal network parameters as network weights of the spatial feature and long-short-term time sequence feature modeling network.

5. The small sample action recognition method based on space and long short-term timing feature modeling according to claim 4, characterized in that, The step S41 specifically includes: uniformly sampling 8 frames of a video as input, and randomly horizontally flipping and randomly cropping the input picture in training. ​ 6. The small sample action recognition method based on space and long short-term timing feature modeling according to claim 4, characterized in that, the learnable parameters the initialization value of the learnable parameters is set to 0.5.

Images