A brain-like small sample behavior recognition algorithm based on motion guided attention mechanism

Abstract

A kind of brain-like small sample behavior recognition algorithm based on motion guided attention mechanism, it is related to the technical field of small sample behavior recognition of deep learning task.Brain-like inductive reasoning mechanism algorithm module based on attention mechanism imitates brain recognition mechanism, mainly includes self-attention motion feature guiding module, video-level global perception module and cross-attention motion feature guiding module, self-attention motion feature guiding module is used to capture the key features of each video frame and the motion-related area;Video-level perception module continuously focuses on the key motion-related area features of the entire video;Cross-attention motion feature guiding module compares the key features of the motion-related area in different videos, establishes the correlation between the motion-related areas;Compared with some existing classical methods, the constructed model can significantly improve the detection accuracy of small sample behavior recognition, and achieve the optimal result.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of small sample behavior recognition of deep learning tasks, in particular to a brain-like small sample behavior recognition algorithm based on a motion-guided attention mechanism. BACKGROUND

[0002] In recent years, the rapid development of deep learning has significantly improved the performance of behavior recognition models. Although existing behavior recognition methods have achieved good results, in the case of limited samples, they often cannot effectively distinguish different categories of videos, resulting in low accuracy. This is because these methods rely on a large amount of labeled data to learn the subtle differences between different behaviors. When the data sample is insufficient, the model is difficult to capture these differences, thereby affecting its recognition ability and generalization performance.

[0003] Therefore, for such a small sample behavior recognition environment, some related methods use advanced technologies such as meta-learning, transfer learning, and data augmentation to effectively learn and distinguish different categories of behaviors even with only a small amount of labeled data. The typical model structure includes a feature extractor, a spatiotemporal enhancement module, and a metric classification function. However, there is currently no method that considers how to use attention mechanisms to design spatiotemporal enhancement modules from the perspective of brain-like inductive reasoning mechanisms. Therefore, there is an urgent need to design a reasonable small sample behavior recognition network framework to address these issues and improve the accuracy of small sample behavior recognition. SUMMARY

[0004] The purpose of the present application is to solve the above-mentioned defects of the prior art and provide a brain-like inductive reasoning mechanism algorithm based on an attention mechanism, which is suitable for small sample behavior recognition. This method draws on the inductive reasoning mechanism of the human brain when recognizing behavior. First, the regions closely related to the action in a single video are determined, and the attention of short-range continuous actions is gradually extended to the long-range continuous action attention of the entire video. Finally, the differences or connections between the motion-related regions of different videos are compared at the task level. This is not only a completely new method, but also a novel small sample behavior recognition framework - "Video to Task".

[0005] The purpose of the present application can be achieved by the following technical means:

[0006] A brain-like small sample behavior recognition algorithm based on a motion-guided attention mechanism, comprising the following steps:

[0007] 1) Preprocessing the input data, the input data including support set videos and query set videos, the preprocessing including random cropping, horizontal flipping, and color jittering;

[0008] 2) The preprocessed input data is subjected to a feature extractor to extract spatial features;

[0009] 3) the feature map with spatial scale extracted by the feature extractor is input into the self-attention motion feature guiding module (S-MGA) to capture the correlation between the key features of the action-related regions in each video frame;

[0010] 4) the video-level perception module receives the output from the self-attention motion feature guiding module (S-MGA) to continuously focus on and correlate the key action-related region features of the entire video, providing a more comprehensive understanding for subsequent comparison of the differences and connections between different videos;

[0011] 5) the cross-attention motion feature guiding module (C-MGA) operates on the entire task video features output by the video perception module to compare the motion-related regions in different videos in the task, and to explicitly optimize the correlation between the motion-related regions in the given task, potentially increasing the inter-class difference while reducing the intra-class difference;

[0012] 6) the output distinctive task-specific features are combined with the existing metric function for small sample behavior recognition classification to achieve higher detection accuracy.

[0013] In step 2), the feature extractor uses ResNet.

[0014] In step 3), the self-attention motion feature guiding module (S-MGA) includes: using bidirectional and multi-scale motion to correlate the motion-related regions between adjacent frames in a video by using a time difference operation; S-MGA uses bidirectional and multi-scale motion features to learn a motion-related self-correlation scoring matrix, which explicitly represents the correlation degree of adjacent frames at the patch level, thereby guiding the correlation of the motion-related region features in the input video.

[0015] In step 3), the feature map with spatial scale extracted by the feature extractor is input into the self-attention motion feature guiding module (S-MGA) to capture the correlation between the key features of the action-related regions in each video frame, including the following steps:

[0016] 3.1) a convolutional dimension reduction operation is used to reduce the dimension of the input video features to reduce the computational load and parameter quantity of subsequent operations;

[0017] 3.2) a time difference feature extraction operation is used to operate on the reduced features to extract corresponding time difference features, which are in the form of simulating optical flow and contain rich motion features;

[0018] 3.3) a motion feature extraction operation is used to operate on the time difference features to extract bidirectional and multi-scale motion features;

[0019] 3.4) Self-association score matrix operation, used for operating on motion features, the calculation of the self-association score matrix is as follows:

[0020]

[0021] where μ represents the softmax function, D is the dimension of the frame-level feature, M(F i ,F i+1 ) is the motion feature extracted by F i and F i+1 frame difference, SA i of the frame-level feature F i is used to explicitly express the degree of association of the motion-related region in F i with the adjacent frame at the patch level; each video frame produces a corresponding self-association score matrix, which explicitly indicates the degree of association of the adjacent frame at the patch level, thereby guiding the association of the motion-related region features in the video;

[0022] 3.5) Output single association video operation, used for operating on the self-association score matrix and the deep convolution enhanced video feature, the calculation of the self-association score matrix SA i guiding the association of the motion-related region features in the video is as follows:

[0023] AF i = λ·SA i Φ3(F) i +F i , i = 1K L

[0024] where λ is a learnable parameter, L is the number of sampled frames in the video, and Φ3 is a 3D deep convolution using a 3x1x1 convolution kernel to enhance the modeling ability of the motion-related region with less spatial change in the time dimension, to explicitly highlight the motion-related region features in the original frame-level feature that have a higher correlation with the motion-related region in the adjacent frame at the block level.

[0025] In step 4), the video-level perception module includes: using the idea of multi-layer perception, efficiently fusing the temporal dynamic features and channel features in the video, which pays more attention to the interaction of dynamic features in the time dimension and the channel dimension, thereby completing the continuous attention to the action-related region in the entire video.

[0026] In step 4), the video-level perception module receives the output from the self-attention motion feature guiding module (S-MGA) to continuously pay attention to and associate the key action-related region features of the entire video, which includes the following steps:

[0027] 4.1) Time MLP operation, which is used to operate the video-level features output by S-MGA, and pays more attention to the interaction of dynamic features in the time dimension, and adopts the following calculation method:

[0028] G i,*,*,* = W2·σ(W1·F i,*,*,* )+ F i,*,*,* i=1K L

[0029] where σ() is the ReLU nonlinear function, and W1 and W2 are LxL matrices;

[0030] 4.2) Channel MLP operation, which is used to operate the features output by the time MLP, and pays more attention to the interaction of dynamic features in the channel dimension, and adopts the following calculation method:

[0031] H *,j,*,* = W4·σ(W3·G *,j,*,* )+ G *,j,*,* i=1K D

[0032] where W3 and W4 are DxD matrices;

[0033] The video-level perception module effectively expands the receptive field along the time dimension in the feature space, enhances the global context perception in time, and thus realizes continuous attention and correlation of the key action-related region features of the entire video, and provides a more comprehensive understanding of the video data.

[0034] In step 5), the cross-attention motion feature guiding module (C-MGA) includes: correlating motion-related regions in different videos in the same task, and C-MGA uses the same method as S-MGA to extract bidirectional and multi-scale motion features of the entire task; for a query video containing L frames, C-MGA calculates L motion-related cross-correlation score matrices to represent the correlation degree of the query video with all support videos in the entire task at the patch level, thereby guiding the correlation of motion-related region features between different videos. Through such correlation, motion-related region relationships within the task can be directly learned, and similar motion-related regions can obtain higher scores while different motion-related regions obtain lower scores.

[0035] In step 5), the cross-attention motion feature guiding module (C-MGA) operates the video features of the entire task output by the video perception module, and the specific steps include:

[0036] 5.1) Convolution dimension reduction operation, which is used to reduce the dimension of the input support set video features and query set video features, so as to reduce the calculation amount and parameter amount of subsequent operations;

[0037] 5.2) Temporal difference feature extraction operation, which is used to operate on the reduced dimension support set video features and query set video features, and extract the corresponding temporal difference features, which are designed to simulate the form of optical flow, containing rich motion features, it should be noted that the S-MGA extracts the temporal difference features of a single video, while the C-MGA extracts the temporal difference features of all support set and query set videos in a task;

[0038] 5.3) Motion feature extraction operation, which is used to operate on the support set video temporal difference features and query set temporal difference features, and extract bidirectional multi-scale motion features of the entire task;

[0039] 5.4) Calculate cross-association attention score matrix, which is used to operate on the motion features of the entire task, and the calculation method is as follows:

[0040]

[0041] Where μ represents the softmax function, MQ,m represents the motion features of the mth video in the query set Q, and MS represents the motion features of the entire support set S;

[0042] 5.5) Output associated support set video and query set video features operation, which is used to operate on the cross-association score matrix and the deep convolution enhanced video features, and the calculation method of cross-association score matrix guiding the calculation of different video motion related region features is as follows:

[0043] A Q,m,i = λ1·CA Q,m,i Φ3(F Q,m ) i + λ2· Φ3(F Q,m ) i + F Q,m,i ,

[0044] A S,n,i = λ2· Φ3(F S,n ) i + F S,n,i

[0045] Where the variable n ranges from 1 to N×K, representing the nth video in the support set, Φ3 is a 3D deep convolution, F x,x,x represents frame-level features, this method maximizes the difference between motion-related regions of different categories and minimizes the difference between the same categories, ensuring a more accurate task-specific attention mechanism, in the selected several categories with similar sub-actions, the C-MGA module accurately focuses on the motion-related regions within the same category, resulting in more discriminative task-specific features.

[0046] Compared with the prior art, the present application has the following advantages and effects:

[0047] 1. The present application proposes a brain-like inductive reasoning mechanism algorithm based on an attention mechanism, which is similar to the process of human small sample recognition, first determines the region closely related to the action in a single video, gradually extends the attention of short-range continuous action to the long-range continuous action attention of the whole video, and finally compares the differences or connections between the motion-related regions of different videos at the task level, which is not only a brand-new method, but also a novel small sample behavior recognition framework - "Video to Task".

[0048] 2. The method proposed by the present application includes three newly designed modules: a self-attention motion feature guide module (S-MGA), a video-level perception module and a cross-attention motion feature guide module (C-MGA) to solve two limitations of previous small sample behavior recognition methods: 1) they mainly use frame-level representations that do not contain spatial information in spatiotemporal relationship modeling, resulting in the neglect of important temporal and spatial information related to motion features within the frame; 2) in the process of generating class prototypes using task-specific methods, the temporal dynamics and spatial relationships in the task cannot be fully utilized at the same time, so the association of the motion-related regions in the video frame in the task is not considered, resulting in unsatisfactory results.

[0049] 3. The present application proposes a novel task-level spatiotemporal enhancement method that can be combined with any existing metric function to realize small sample behavior recognition classification, and compared with previous methods, the method realizes the best detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the overall framework diagram of the model designed by the brain-like inductive reasoning mechanism algorithm based on the attention mechanism disclosed by the present application and the schematic diagram of each module;

[0051] Figure 2 is the specific network structure diagram of the self-attention motion feature guide module designed by the brain-like inductive reasoning mechanism algorithm based on the attention mechanism disclosed by the present application;

[0052] Figure 3 is the specific network structure diagram of the video-level perception module designed by the brain-like inductive reasoning mechanism algorithm based on the attention mechanism disclosed by the present application;

[0053] Figure 4 is the specific network structure diagram of the cross-attention motion feature guide module designed by the brain-like inductive reasoning mechanism algorithm based on the attention mechanism disclosed by the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application more clear, the following examples will be further described in combination with the drawings. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application. On the contrary, the present application covers any substitution, modification, equivalent method and solution defined by the claims within the essence and scope of the present application. Further, in order to make the public have a better understanding of the present application, some specific details are described in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0055] The brain-like inductive reasoning mechanism algorithm based on attention mechanism disclosed in the present application is suitable for small sample behavior recognition, and the small sample behavior recognition is defined as follows:

[0056] The task of small sample behavior recognition aims to identify new classes using only a small number of labeled samples, and the task usually contains a training set D train ={(vi,yi),yi∈C train} for training small sample model, and a test set D test ={(vi,yi),yi∈C test} for testing, wherein Adopting a scenario training mode, wherein each scenario contains a support set video S composed of K videos randomly extracted from each of the N action categories in D train Subsequently, Q samples of query set video are selected from the remaining videos of the N action categories in D train In each scenario, it is expected to classify the videos in the query set into one of the N categories with the help of the support set. For the inference process, the average accuracy rate of the recognition performance of all scenarios in the test set D test

[0057] Figure 1 The overall framework diagram of the model designed by the brain-like inductive reasoning mechanism algorithm based on attention mechanism and the schematic diagram of each module are given, mainly including:

[0058] The self-attention motion feature guidance module (S-MGA) is used to capture the association between the key features of the action-related regions between each video frame;

[0059] The video-level perception module is used to continuously focus on and associate the key action-related region features of the entire video;

[0060] The cross-attention motion feature guidance module (C-MGA) is used to compare the motion-related regions of different videos in the task, to explicitly optimize the association between the motion-related regions in the given task, to potentially increase the inter-class difference, and to reduce the intra-class difference. ​

[0061] The overall architecture of the present application is as follows:

[0062] First, the input data is preprocessed, including the following operations: random cropping, horizontal flipping and color jittering.

[0063] For the composition of the input data: support set video S ∈ R NK×T×3×224×224 , query set video Q ∈ R M ×T×3×224×224 , where T represents the number of frames sampled for each video, and M represents the number of query set videos in a task.

[0064] The preprocessed input data will be extracted by the feature extractor to extract spatial features: the feature extractor usually selects an existing model, such as ResNet, to generate a feature map with spatial dimensions ∈ R D×H×W , where D represents the number of channels of the feature map, and H and W represent the spatial dimensions of the feature map, which will be used as the input of the subsequent algorithm.

[0065] The spatially scaled feature map extracted by the feature extractor will pass through the self-attention motion feature guide module (S-MGA) to capture the correlation between the key features of the action-related regions between each video frame, including: inputting a single video feature, performing convolution operation on the feature to reduce the dimension of the feature and reduce the amount of calculation while retaining key information; performing time difference feature extraction on the video features after convolution dimension reduction to calculate the time difference between video frames, thereby capturing motion information; based on the result of the time difference, extracting key motion features related to the action; calculating the self-association score matrix between the frames or features within the video to highlight the importance of key features. The input single video feature uses 3D deep convolution to extract features from the input video, and outputs key features related to the action between each video frame, which are used in subsequent video-level perception or cross-attention motion feature guide modules;

[0066] The specific network structure diagram of the self-attention motion feature guide module is shown in Figure 2 , and the specific operation is as follows:

[0067] The feature map is down-sampled by a factor of r1=8 through a 1×1 convolution to reduce the amount of calculation and the number of parameters.

[0068] The time difference feature is extracted by the time difference feature extraction operation, and the specific operation is that the feature of the previous frame is smoothed spatially after passing through a 2D deep convolution with a convolution kernel size of 3×1, and then subtracted from the feature of the next frame to generate the forward time difference feature of each frame, which can be expressed by the formula as follows:

[0069] C(F i ,F i+1 )=F i- Φ2(Fi, Fi+1) = Φ1(C(Fi, Fi+1)) i+1 )

[0070] where C(Fi, Fi+1) denotes the aligned temporal difference between frame Fi and frame Fi+1, Φ2 is a 2D depth convolution, and the extraction of the reverse temporal difference feature is similar.

[0071] The motion feature is extracted from the temporal difference feature by a motion feature extraction operation, which can be expressed by the following formula:

[0072]

[0073] where N = 4 in practice corresponds to the number of different spatial scales of CNN j , each scale extracts motion information from different receptive fields, and follows a three-branch implementation: (1) short connection, (2) 3x3 convolution, and (3) average pooling, 3x3 convolution, and bilinear upsampling, (4) bilinear upsampling, 3x3 convolution, and average pooling. Conv is a channel convolution used to aggregate multi-scale motion features. Bidirectional and multi-scale motion features contain rich motion cues between frames, so these features can be used to capture the association of precise motion-related regions in each video.

[0074] The motion-related self-association score matrix SA i of the frame-level feature F i is obtained by a self-attention mechanism of the bidirectional motion feature, to explicitly express the degree of association of the motion-related regions in F i with the adjacent frames at the patch level, and the specific calculation method is as follows:

[0075]

[0076] where μ denotes the softmax function, D is the dimension of the frame-level feature, and M(Fi, Fi+1) is the motion feature extracted by the difference between Fi and Fi+1 frames. Each video frame produces a corresponding self-association score matrix, which explicitly indicates the degree of association of adjacent frames at the patch level, thereby guiding the association of motion-related region features in the video.

[0077] The output single association video operation is used to operate on the self-association score matrix and the deep convolution enhanced video feature, and the calculation method is as follows:

[0078] AF i = λ·SA i Φ3(F) i +F i ,i = 1K L

[0079] Wherein, Lambda is a learnable parameter, L is the number of sampled frames in the video, Phi3 is a 3D deep convolution using a 3x1x1 convolution kernel, which enhances the modeling ability of the motion-related area with less spatial change in the time dimension, and clearly highlights the motion-related area features in the original frame-level features F i With higher correlation with the motion-related area at the patch level in the adjacent frame.

[0080] The video-level perception module receives the output from the self-attention motion feature guiding module (S-MGA) and operates on the output of the self-attention motion feature guiding module to obtain continuous attention to the area closely related to motion in a single video, providing a more comprehensive understanding for subsequent comparison of the differences and connections between different videos. The specific network structure diagram of the video-level perception module is shown in Figure 3 As follows:

[0081] The video frame F i ∈R T×D×H×W .

[0082] The time sequence MLP operation is used to operate on the video-level features output by the S-MGA, which pays more attention to the interaction of dynamic features in the time dimension, and the calculation method is as follows:

[0083] G i,*,*,* =W2·sigma(W1·F i,*,*,* )+F i,*,*,* i=1K L

[0084] Wherein, sigma() is a ReLU nonlinear function, and W1 and W2 are LxL matrices.

[0085] The channel MLP operation is used to operate on the features output by the time sequence MLP, which pays more attention to the interaction of dynamic features in the channel dimension, and the calculation method is as follows:

[0086] H *,j,*,* =W4·sigma(W3·G *,j,*,* )+G *,j,*,* i=1K D

[0087] Wherein, W3 and W4 are DxD matrices.

[0088] The video-level perception module effectively expands the receptive field along the time dimension in the feature space, enhances the global context perception in time, and thus realizes continuous attention and correlation of the key action-related area features of the entire video, providing a more comprehensive understanding of the video data.

[0089] The cross-attention motion feature guide module (C-MGA) operates on the video features output by the video perception module for the entire task, including: first, the C-MGA module receives video data from the query video set (set video Q) as input, at the same time, the C-MGA module also receives video data from the support video set (set video S), which is used for comparison and association with the query video; using 3D deep convolution, the input query video and support video are respectively subjected to preliminary feature extraction, generating respective video features, and performing convolution dimension reduction; the extracted video features are subjected to a time difference operation to calculate the time difference between video frames, thereby capturing motion information; based on the results of the time difference, action-related key motion features are extracted; by calculating the cross-association score matrix, the cross-attention mechanism is applied to compare the motion features between the query video and the support video; after the above processing, the C-MGA module outputs the processed feature representation, which can be used for subsequent classification, identification or other tasks.

[0090] The specific network structure diagram of the cross-attention motion feature guide module is as shown in Figure 4 Specifically, the C-MGA adopts convolution to compress the feature dimension by a ratio r2, and then uses the same method as the S-MGA to extract bidirectional and multi-scale motion features for the entire task. Subsequently, for a query video containing T frames, the C-MGA calculates T motion-related cross-association score matrices, indicating the degree of correlation between the query video and all support videos in the entire task at the patch level. This enables the present application to explicitly optimize the association between motion-related regions in a given specific task, potentially increasing the difference between classes while reducing the difference within classes.

[0091] For each query video, the C-MGA generates T unique motion-related cross-association score matrices, each corresponding to a frame in the video. Then, for the i-th frame (i∈[1, T]) of the m-th (m∈[1, |Q|]) video in the query set Q, the calculation method of its motion-related cross-association score matrix is as follows:

[0092]

[0093] Where μ represents the softmax function, M Q,m represents the motion features of the m-th video in the query set Q, M S represents the motion features of the entire support set S; M Q,m ∈R L×HW×D / r2 , M S ∈R L×NKHW×D / r2 Therefore, CA Q,m,i ∈R HW×NKHW .

[0094] The output associated support set video and query set video feature operation is used for cross association score matrix and deep convolution enhanced video feature operation, and the calculation method is as follows:

[0095] A Q,m,i = lambda1 * CA Q,m,i Phi3 (F Q,m ) i + lambda2 * Phi3 (F Q,m ) i + F Q,m,i ,

[0096] A S,n,i = lambda2 * Phi3 (F S,n ) i + F S,n,i

[0097] Wherein, the range of variable n is 1 to N*K, indicating the nth video in the support set, Phi3 is 3D deep convolution, Fx,x,x represents frame level feature, this method maximizes the difference between different categories of motion related areas and minimizes the difference between the same categories, ensures more accurate task specific attention mechanism, in several selected categories with similar sub-actions, the C-MGA module accurately focuses on the motion related areas in the same category, so as to produce more distinctive task specific features.

[0098] Then, the output distinctive task specific features will be combined with the existing metric function for small sample behavior recognition classification, so as to realize higher detection accuracy.

[0099] The model of the application is superior to a large number of existing works in 1-shot and 5-shot settings on all data sets. The 1-shot accuracy on the classic small sample small sample behavior recognition data sets such as UCF101, HMDB51, Kinetics, SSv2-Small and SSv2-Full can reach: 88.0%, 63.5%, 75.0%, 42.8% and 56.7%. The 5-shot accuracy can reach: 97.4%, 79.3%, 88.0%, 61.8% and 71.1%. The effectiveness of the brain-like inductive reasoning mechanism algorithm based on attention mechanism is verified.

[0100] The above embodiments are only preferred embodiments of the application and cannot be considered as limiting the scope of the application. Any equivalent changes and improvements made within the scope of the application should still belong to the patent coverage of the application. Any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the application are equivalent replacement methods and are included in the protection scope of the application.

Claims

1. A brain-like small sample behavior recognition algorithm based on motion guided attention mechanism, characterized by The method comprises the following steps: 1) preprocessing input data, the input data comprising support set videos and query set videos, the preprocessing comprising random cropping, horizontal flipping and color jittering; 2) the preprocessed input data is subjected to a feature extractor to extract spatial features; 3) the feature map with spatial scales extracted by the feature extractor is subjected to a self-attention motion feature guiding module S-MGA to capture the correlation between the key features of the action-related regions between each video frame; The self-attention motion feature guiding module S-MGA comprises: using a time difference operation to extract bidirectional and multi-scale motion features to correlate the motion-related regions between adjacent frames in a video; S-MGA uses bidirectional and multi-scale motion features based on a self-attention mechanism to learn a motion-related self-correlation scoring matrix, which explicitly represents the correlation degree of adjacent frames at the patch level, thereby guiding the correlation of the motion-related region features in the input video; 4) a video-level perception module receives the output from the self-attention motion feature guiding module S-MGA to continuously focus on and correlate the key action-related region features of the entire video; The video-level perception module comprises: using the idea of a multi-layer perception machine to efficiently fuse the temporal dynamic features and channel features in the video, thereby completing continuous attention to the action-related regions in the entire video; 5) the entire task video features output by the video perception module are subjected to operation by a cross-attention motion feature guiding module C-MGA to compare the motion-related regions of different videos in the task, explicitly optimize the correlation between the motion-related regions in the given task, potentially increase the inter-class difference, and reduce the intra-class difference; The cross-attention motion feature guiding module C-MGA comprises: correlating the motion-related regions of different videos in the same task, C-MGA uses the same method as S-MGA to extract bidirectional and multi-scale motion features for the entire task; for a query video comprising L frames, C-MGA calculates L motion-related cross-correlation scoring matrices based on a cross-attention mechanism to represent the correlation degree of the query video and all support videos in the entire task at the patch level, thereby guiding the correlation of the motion-related region features between different videos; through such correlation, the motion-related region relationship within the task can be directly learned, promoting the same type of motion-related region to obtain a higher score and the different type of motion-related region to obtain a lower score; 6) outputting task-specific features with discriminativeness in combination with an existing metric function for small sample behavior recognition classification to achieve higher detection accuracy.

2. The brain-inspired small sample behavior recognition algorithm based on motion-guided attention mechanism according to claim 1, characterized in that In step 2), the feature extractor uses ResNet.

3. The brain-inspired small sample behavior recognition algorithm based on motion-guided attention mechanism of claim 1, wherein In step 3), the feature map with spatial scales extracted by the feature extractor is subjected to the self-attention motion feature guiding module S-MGA to capture the correlation between the key features of the action-related regions between each video frame, comprising the following steps: 3.1) a convolution dimension reduction operation is used to reduce the dimension of the input video features to reduce the calculation amount and parameter amount of subsequent operations; 3.2) Temporal difference feature extraction operation, used for operating on the reduced dimension features, to extract the corresponding temporal difference features, which aims to simulate the form of optical flow, which contains rich motion features; 3.3) Motion feature extraction operation, used for operating on the temporal difference features, to extract bidirectional multi-scale motion features; 3.4) Self-association score matrix calculation operation, used for operating on the motion features, the calculation method of the self-association score matrix is as follows: wherein, η denotes a softmax function, D is the dimension of frame-level features, M(F i ,F i+1 ) is the motion feature extracted by frame difference between F i and F i+1 , SA i of frame-level features F i is used to explicitly express the degree of association of the motion-related regions in F i with the adjacent frames at the patch level; each video frame produces a corresponding self-association score matrix, which explicitly indicates the degree of association of the adjacent frames at the patch level, thereby guiding the association of the motion-related region features in the video; 3.5) output single associated video operation for operating on the self-association score matrix and the deep convolution enhanced video features, self-association score matrix SA i The calculation of the motion-related region feature association in the guided video is as follows: AF i = λ · SA i Φ3(F) i +F i i = 1,..., L where λ is a learnable parameter, L is the number of sampled frames in the video, and Φ3 is a 3D deep convolution using a 3x1x1 convolution kernel to enhance the modeling ability of motion-related regions with smaller spatial changes in the time dimension, to clearly highlight the motion-related region features in the original frame-level features that have higher correlation with the motion-related regions in the adjacent frames at the block level.

4. The brain-inspired small sample behavior recognition algorithm based on motion-guided attention mechanism of claim 1, wherein In step 4), the video-level perception module receives the output from the self-attention motion feature guide module S-MGA, continuously paying attention to and associating the key action-related region features of the entire video, including the following steps: 4.1) Temporal MLP operation, used for operating on the video-level features output by S-MGA, which pays more attention to the interaction of dynamic features in the time dimension, using the calculation method: G i,*,*,* = W2 · σ(W1 · F i,*,*,* )+ F i,*,*,* i = 1,..., L where σ() is the ReLU nonlinear function, W1 and W2 are LxL matrices, and L is the number of sampled frames in the video; 4.2) Channel MLP operation, used for operating on the features output by the temporal MLP, which pays more attention to the interaction of dynamic features in the channel dimension, using the calculation method: H *,j,*,* = W4·σ(W3·G *,j,*,* )+ G *,j,*,* j = 1,..., D where W3 and W4 are DxD matrices, and D is the dimension of the frame-level features; The video-level perception module effectively expands the receptive field along the time dimension in the feature space, enhancing the global context perception in time, thereby continuously paying attention to and associating the key action-related region features of the entire video, providing a more comprehensive understanding of the video data.

5. The brain-inspired small sample behavior recognition algorithm based on motion-guided attention mechanism of claim 1, wherein In step 5), the cross-attention motion feature guide module C-MGA operates on the entire task video features output by the video perception module, including the following steps: 5.1) Convolution dimension reduction operation, used for reducing the dimension of the input support set video features and query set video features, to reduce the computational load and parameter quantity of subsequent operations; 5.2) Temporal difference feature extraction operation, used for operating on the reduced dimension support set video features and query set video features, to extract the corresponding temporal difference features, which aims to simulate the form of optical flow, containing rich motion features. It should be noted that the temporal difference features extracted in S-MGA are for a single video, while in C-MGA, the temporal difference features are extracted for all support set and query set videos in a task; 5.3) Motion feature extraction operation, used for operating on the support set video temporal difference features and query set temporal difference features, to extract bidirectional multi-scale motion features for the entire task; 5.4) Cross-association attention score matrix calculation, used for operating on the motion features of the entire task, using the calculation method: wherein, η denotes a softmax function, MQ,m denotes the motion feature of the mth video in the query set Q, D is the dimension of the frame-level feature, MS denotes the motion feature of the entire support set S, M(F i ,F i+1 ) is the motion feature extracted by F i and F i+1 frame difference; 5.5) Output associated support set video and query set video feature operations for cross-association scoring matrix and deep convolution enhanced video feature operations, cross-association scoring matrix guides different video motion related region feature association computation as follows: A Q,m,i = λ1·CA Q,m,i Φ3(F Q,m ) i + λ2·Φ3(F Q,m ) i + F Q,m,i , A S,n,i = λ2· Φ3(F S,n ) i + F S,n,i where n ranges from 1 to N x K, indicating the nth video in the support set, Φ3is a 3D depth convolution, F x,x,x denotes frame-level features.

Images