A small sample behavior recognition method, system and device
By constructing a few-shot behavior recognition model based on the R(2+1)D network and combining feature embedding, hybrid attention and prototype attention modules, the problem of behavior recognition under few-shot datasets is solved, and effective video behavior recognition and model generalization are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2023-02-20
- Publication Date
- 2026-06-05
AI Technical Summary
Existing deep learning-based behavior recognition tasks require a large amount of data and are difficult to train effectively on small sample datasets. Furthermore, traditional methods cannot effectively process the temporal information of video sequences, leading to model overfitting and insufficient data transfer capabilities.
A few-sample behavior recognition model is constructed, using an R(2+1)D network as the backbone network, combined with a feature embedding module, a hybrid attention module, and a prototype attention module. The feature embedding module processes the spatial and temporal features of the video, the hybrid attention module enhances the features, the prototype attention module performs classification, and the polarization attention mechanism and the spatiotemporal residual module are used to optimize feature extraction and classification.
It achieves effective video behavior recognition on small sample datasets, improves the model's recognition accuracy and generalization ability, and solves the problem of insufficient temporal information processing in traditional methods.
Smart Images
Figure CN116311504B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of few-sample behavior recognition technology, and in particular to a few-sample behavior recognition method, system and device. Background Technology
[0002] Action recognition is an important branch of video understanding, aiming to classify videos containing human movements. Unlike image recognition tasks, human movements in video action recognition are dynamic. In recent years, with the continuous improvement of large-scale video datasets and the development of deep learning technology, research on action recognition has made significant progress. Traditional deep learning-based action recognition tasks require a large amount of labeled video data as the underlying data support, resulting in high human costs. Current traditional deep learning algorithms are only good at mastering a specific task; even with similar datasets, well-trained models are difficult to transfer to other datasets. In real-world industries, fields such as defense, military, and medical diagnosis struggle to obtain large amounts of labeled video data. Directly training deep learning methods on small datasets often leads to model overfitting. Some researchers have attempted to expand datasets using simple data augmentation methods such as rotating and cropping video and image data, but the results have been unsatisfactory. Deep learning uses neural network designs that mimic the operation of neurons in the human brain. However, humans don't need to be trained with thousands of data points to learn to classify new things; they can accurately identify a new object category by looking at just a few or even a single image. Driven by the need to reduce reliance on large-scale datasets and to meet the demands of human-like learning, research into behavior recognition using small sample data has emerged.
[0003] Few-shot behavior recognition is built upon few-shot learning. Few-shot learning is set as an N-way, K-shot learning task. Here, 'way' represents the number of classes sampled in the training data, and 'shot' represents the total number of samples in each class. When the number of sampling classes N=3 and the total number of samples K=3, the training sampling process is as follows: Figure 1As shown in the diagram. First, for all categories in the dataset, N (N=3 in the diagram) categories are randomly selected as training categories. In the diagram, categories 2, 3, and 5 are selected. Then, from the total samples of the already selected categories, K (K=3 in the diagram), represented by green squares, are randomly selected to form the support set used for training. The total number of samples in the support set is K*N. Finally, from the remaining unselected samples for the training categories, X (X=3 in the diagram), represented by orange squares, are selected to form the query set. The query set contains X*N samples. All samples in the support set are labeled, while samples in the query set are unlabeled. The core objective of few-shot learning is to determine the similarity between samples in the query set and samples in the support set, accurately classifying the samples in the query set. Each sampling process is called an episode.
[0004] like Figure 1 As shown, the goal of few-shot action recognition is to classify an unlabeled query video into an action category within a support set, where the number of samples for each action category is limited. Few-shot action recognition is a challenging computer vision problem. Videos consist of multiple frames and are temporally sequential; recognizing human behavior within the video requires combining contextual video frame information for understanding. Action recognition, as one of the most important branches of video understanding, has significant application value in real-world scenarios. However, traditional deep learning-based action recognition tasks require substantial data dependencies, and the models lack data transfer capabilities, making it difficult to expand application scenarios. Researching methods for few-shot action recognition can better address this practical situation and improve recognition accuracy. Therefore, research on few-shot-based action recognition methods is of great significance, possessing important theoretical and practical application value.
[0005] Currently, methods for few-shot action recognition tasks can generally be divided into two categories: generative methods and metric learning-based methods. Generative methods aim to improve recognition accuracy by expanding the dataset's sample size. Metric learning-based methods utilize meta-learning, obtaining a feature vector space through processing and then calculating the classification result by measuring the distance to the support vectors. Most mainstream methods currently employ metric learning. For example, STRM adds a spatiotemporal enrichment module and a temporal relationship modeling module to TRX. After enhancing the features, a classifier measures and classifies the intermediate layer features to further enhance feature discriminability. STRM has achieved state-of-the-art performance. However, most few-shot learning methods, including STRM, use 2D networks as their basic framework. 2D networks cannot handle the temporal information of video sequences when extracting features, so these methods must process the temporal information of features piecemeal. Summary of the Invention
[0006] This invention provides a method for identifying behavior in a small sample, comprising:
[0007] A few-shot behavior recognition model is constructed, which includes a feature embedding module for simultaneously extracting and processing spatial and temporal features of the video, a hybrid attention module for capturing and processing the spatial context features of the video, and a prototype attention module for implementing feature classification of the video.
[0008] The video to be processed is preprocessed to obtain a support set with category labels and a query set without category labels. Both the support set and the query set are formed by video frames with time sequence.
[0009] The support set and query set are input into the small sample behavior recognition model;
[0010] Based on the feature embedding module, the support set and query set are processed to obtain the support feature set and query feature set, respectively.
[0011] At least based on the hybrid attention module, the features in the supporting feature set and the query feature set are enhanced;
[0012] Based on the prototype attention module, the support features in the enhanced support feature set are mapped to different first vector spaces using the target mapping function. At the same time, the query features in the enhanced query feature set are mapped to different second vector spaces using the target mapping function. The distances between each first vector space and the second vector space are compared, and based on the comparison results, it is determined that the query feature with the closest distance in the vector space belongs to the same category as the support feature.
[0013] As an optional embodiment, the feature embedding module consists of a 2D convolutional network and a 1D convolutional network. The 2D convolutional network is used to process the spatial data of the video, and the 1D convolutional network is used to process the temporal features of the video to obtain the temporal relationship of multiple video frames in the support set and query set.
[0014] As an optional embodiment, the hybrid attention module introduces a serial polarized attention mechanism, which performs data processing on the input channel, the width and height of the supporting features or query features of the few-sample behavior recognition model based on the polarized attention mechanism.
[0015] As an optional embodiment, the enhancement processing of features in the supporting feature set and the query feature set based at least on the hybrid attention module includes:
[0016] Based on the polarized attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then enters a 1x1 first convolutional layer and a second convolutional layer respectively to transform into first data and second data. The input channel of the first data is compressed to 1, and the input channel dimension of the second data is C / 2, where C is the number of input channels of the hybrid attention module.
[0017] The first data is augmented based on the objective function;
[0018] The second data is matrix-multiplied with the enhanced first data, and a 1x1 convolution and the feature normalization layer in the hybrid attention module are applied to the multiplication result to increase the input channel dimension of the second data to C.
[0019] The first data and the second data are dynamically mapped to control the weights in the hybrid attention module to be between 0 and 1.
[0020] The weights are multiplied by each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding input channel.
[0021] As an optional embodiment, the enhancement processing of features in the supporting feature set and the query feature set based at least on the hybrid attention module includes:
[0022] Based on the polarized attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then transformed into third and fourth data by 1x1 convolutional layers respectively.
[0023] The third data is spatially compressed so that its width and height are converted to form a 1*1 size;
[0024] The spatial dimension of the fourth data remains constant;
[0025] The compressed third data is then subjected to information enhancement processing;
[0026] The fourth data is dynamically mapped to the information-enhanced third data to control the weights in the hybrid attention module to be between 0 and 1.
[0027] The weights are multiplied by each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding space.
[0028] As an optional embodiment, the hybrid attention module further includes a spatiotemporal residual module;
[0029] The method further includes:
[0030] Based on the spatiotemporal residual module, 2D convolution is performed on the features in the supporting feature set and the query feature set in the spatial dimension to the intermediate subspace;
[0031] Based on the activation function and BN layer in the spatiotemporal residual module, the features convolved into the intermediate subspace are processed, and the features are then subjected to 1D convolution on the time axis.
[0032] Based on the features of 1D convolution processed by the pooling layer in the spatiotemporal residual module, the enhanced support feature set and query feature set are obtained.
[0033] As an optional embodiment, the prototype attention module has a multi-head self-attention layer;
[0034] The method further includes:
[0035] Based on the enhanced support features and query features obtained by the prototype attention module through the multi-head self-attention layer, attention scores between features are obtained.
[0036] The attention score and the corresponding enhanced support feature or query feature are input and added together, and then combined with the objective function for mapping processing.
[0037] As an optional embodiment, the step of comparing the distances between each of the first vector spaces and the second vector spaces, and determining, based on the comparison results, that the query feature with the closest vector space distance belongs to the same category as the supporting feature, includes:
[0038] The prototype attention module calculates the distance between the first vector space corresponding to different supporting features and the second vector space corresponding to different query features based on the metric space, and determines that the query feature and the supporting feature with the closest vector space distance belong to the same category based on the comparison results.
[0039] The metric space is learned by the prototype attention module during feature processing in the multi-head self-attention layer.
[0040] Another embodiment of the present invention also provides a small sample behavior recognition system, comprising:
[0041] A construction module is used to build a few-shot behavior recognition model. The few-shot behavior recognition model includes a feature embedding module for simultaneously extracting and processing spatial and temporal features of the video, a hybrid attention module for capturing and processing the spatial context features of the video, and a prototype attention module for implementing feature classification of the video.
[0042] The preprocessing module is used to preprocess the video to be processed to obtain a support set with category labels and a query set without category labels. Both the support set and the query set are formed by video frames with time sequence.
[0043] The input module is used to input the support set and query set into the small sample behavior recognition model;
[0044] The few-sample behavior recognition model is used to process the support set and query set according to the feature embedding module to obtain a support feature set and a query feature set respectively; to enhance the features in the support feature set and query feature set according to the hybrid attention module; to map the support features in the enhanced support feature set to different first vector spaces according to the target mapping function according to the prototype attention module, and to map the query features in the enhanced query feature set to different second vector spaces according to the target mapping function; to compare the distance between each first vector space and second vector space, and to determine the category to which the query feature and support feature with the closest vector space distance belong based on the comparison results.
[0045] Another embodiment of the present invention also provides a small sample behavior recognition device, comprising:
[0046] At least one processor; and,
[0047] A memory communicatively connected to the at least one processor; wherein,
[0048] The memory stores instructions that can be executed by the at least one processor to implement the few-sample behavior recognition method as described in any of the embodiments above.
[0049] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.
[0050] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0052] Figure 1 This is a flowchart illustrating the process of a small-sample behavior recognition method in an existing scheme.
[0053] Figure 2 This is a flowchart of a small sample behavior recognition method in an embodiment of the present invention.
[0054] Figure 3 This is a structural framework diagram of the small sample behavior recognition model in an embodiment of the present invention.
[0055] Figure 4 This is a flowchart illustrating the application of the small sample behavior recognition method in this embodiment of the invention.
[0056] Figure 5 This is a structural diagram of the main modules of the R(2+1)D network in an embodiment of the present invention.
[0057] Figure 6 This is a structural diagram of the polarization attention mechanism in an embodiment of the present invention.
[0058] Figure 7 This is a structural block diagram of the small sample behavior recognition system in an embodiment of the present invention. Detailed Implementation
[0059] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.
[0060] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this disclosure will be apparent to those skilled in the art.
[0061] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.
[0062] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.
[0063] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention, which have the features described in the claims and are therefore all within the scope of protection defined herein.
[0064] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.
[0065] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.
[0066] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.
[0067] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0068] like Figure 2 As shown, this embodiment of the invention provides a method for identifying behavior in small samples, including:
[0069] S101: Construct a few-shot behavior recognition model, which includes a feature embedding module for simultaneously extracting and processing spatial and temporal features of the video, a hybrid attention module for capturing and processing spatial context features of the video, and a prototype attention module for realizing feature classification of the video.
[0070] S102: Preprocess the video to be processed to obtain a support set with category labels and a query set without category labels. Both the support set and the query set are formed by video frames with time sequence.
[0071] S103: Input the support set and query set into the small sample behavior recognition model;
[0072] S104: Based on the feature embedding module, the support set and query set are processed to obtain the support feature set and query feature set, respectively;
[0073] S105: At least based on the hybrid attention module, the features in the supporting feature set and the query feature set are enhanced;
[0074] S106: Based on the prototype attention module, the support features in the enhanced support feature set are mapped to different first vector spaces using the target mapping function. At the same time, the query features in the enhanced query feature set are mapped to different second vector spaces using the target mapping function. The distance between each first vector space and the second vector space is compared, and based on the comparison results, it is determined that the query feature with the closest distance in the vector space belongs to the same category as the support feature.
[0075] Few-shot behavior recognition aims to train a model on a dataset with extremely limited sample sizes, accurately classifying unlabeled videos into labeled video categories, thus achieving video classification. Most current methods employ metric learning and use 2D networks as the backbone, requiring fragmented processing of temporal information of features. To address this issue, this embodiment proposes a hybrid attention prototype network for few-shot behavior recognition, a few-shot behavior recognition model with a structure different from existing solutions. The model algorithm consists of a feature embedding module, a hybrid attention module, and a prototype attention module. The feature embedding module utilizes R(2+1)D from the video understanding domain as the backbone network to jointly process temporal and spatial information using video domain knowledge. In the hybrid attention module, the model simultaneously enhances spatial and temporal information. The prototype attention module follows the architecture of the prototype network; before metric matching, this embodiment adds a multi-head attention mechanism to learn the relationships between sample points in the vector space.
[0076] When applying, such as Figure 3 As shown, a support set and a query set can be given as inputs to a few-shot behavior recognition model (hereinafter referred to as the model). The model processes the data to map the support features in the support set and the query features in the query set to a vector space. The sample points (query feature vectors) in the query set are used to perform a metric matching with the sample points (support feature vectors) in the support set in the vector space. The support feature vector with the highest similarity, i.e., the support feature vector with the smallest distance, is selected as the category label of the corresponding query feature vector.
[0077] To simplify the explanation of the above embodiments, N=3, K=1, and the explanation focuses only on a single query video. Specifically, the support set contains N types of video actions. Each individual video segment is extracted into L video frames, which serve as the input to the model. The video frames from the support set are processed by the temporal feature extraction module (i.e., the feature embedding module) to obtain the support features Es. The video frame data from the query set also undergoes a similar feature embedding module to obtain the query features Eq. That is, as shown... Figure 4 As shown, the model in this embodiment has two sets of modules. Each set includes a feature embedding module, a hybrid attention module, and a prototype attention module. The two sets of modules are used to process the support set and the query set, respectively, and corresponding modules in the two sets share parameters, such as the feature embedding modules in the two sets sharing parameters and having the same structure. Next, the obtained features are further enhanced by the hybrid attention module. Then, the enhanced features in the support set are mapped to different vector spaces (D1...D2) based on a mapping function. NFor each feature in the query set, the same mapping function is used to map it to a vector space G, and then the vector space G is compared with multiple different vector spaces (D1...D2). N The distance to the nearest supporting feature vector is used to select the category to which the query sample belongs.
[0078] Furthermore, such as Figure 5 As shown, the feature embedding module in this embodiment consists of a 2D convolutional network and a 1D convolutional network. The feature embedding module in this embodiment adopts an R(2+1)D network. That is, the feature embedding module is an R(2+1)D network, which is evolved from 3D CNN. The feature embedding module in this embodiment decomposes the complete 3D convolution in 3D CNN into a 2D convolution and a 1D convolution. The 2D convolutional network is used to process the spatial data of the video, and the 1D convolutional network is used to process the temporal features of the video. This is because the three-dimensional spatiotemporal convolution fuses spatial and dynamic information together, which is not easy to optimize, while two-dimensional and one-dimensional convolutions are easier to optimize. Through the feature embedding module designed in this embodiment, the temporal relationship of multiple video frames in the support set and query set can be obtained.
[0079] Furthermore, the hybrid attention module in this embodiment is designed to better extract the spatial and temporal contextual information of the video. Firstly, in order to capture the differences between different categories of features at different spatial locations, such as... Figure 6 As shown, this embodiment introduces a serial polarized attention mechanism (PSA) to extract and enhance spatial context features within the video frame's image and channels. Based on the polarized attention mechanism, data processing is performed on the input channel, the width, and the height of the support or query features of the few-sample behavior recognition model, without affecting the batch or temporal dimensions of the data. In other words, the PSA processes only a single video frame within the support set or query set.
[0080] Specifically, when performing enhancement processing on features in the supporting feature set and the query feature set based at least on a hybrid attention module, it includes:
[0081] S107: Based on the polarization attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then enters the first and second convolutional layers of 1x1 respectively to transform into first data and second data. The input channel of the first data is compressed to 1, and the input channel dimension of the second data is C / 2, where C is the number of input channels of the hybrid attention module.
[0082] S108: Perform information enhancement processing on the first data based on the objective function;
[0083] S109: Multiply the second data with the enhanced first data by matrix multiplication, and then connect the multiplication result to a 1x1 convolution and a feature normalization layer in the hybrid attention module to increase the input channel dimension of the second data to C.
[0084] S110: Dynamically map the first data and the second data to control the weights in the hybrid attention module to be between 0 and 1;
[0085] S111: Perform a dot product of the weights with each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature on the corresponding input channel.
[0086] For example, the feature data output by the feature embedding module are denoted as E(si) and E(q), respectively, and serve as the input to the hybrid attention module. For ease of explanation, the input to the PSA (module) is denoted as x in this embodiment. Combined with... Figure 6 As can be seen, PSA has two branches: a channel branch and a spatial branch. Based on the principles of PSA, this embodiment aims to completely collapse one dimension of the feature while maintaining high resolution in the orthogonal directions. Therefore, in cascaded PSA, the input data x is also divided into two branches, entering 1x1 convolutional layers F_q and F_v respectively, and being transformed into q (first data) and v (second data). The input channels of q are compressed to 1, while the input channel dimension of v is reduced to C / 2, where C is the number of input channels in the hybrid attention module. Although the input channels are dimensionality-reduced, they still maintain a relatively high dimension. Since the input channel dimension of q is compressed, this embodiment preferably uses the Softmax function to enhance the information in q, thereby increasing the attention range. Then, q and v are multiplied by matrix, and a 1x1 convolution and LayerNorm layer (feature normalization layer) are applied to the multiplication result to increase the C / 2 dimension of the input channels to C dimension. Finally, the Sigmoid function is used for dynamic mapping of the processed q and v. Based on the Sigmoid function, the parameters (weights) of all PSA modules can be kept between 0 and 1. Based on this weight, the initial input data x is multiplied by a dot product, thus enabling the x features to receive channel-wise attention enhancement.
[0087] Furthermore, when performing enhancement processing on features in the supporting feature set and the query feature set based at least on the hybrid attention module, it also includes:
[0088] S112: Based on the polarization attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then entered into the third and fourth convolutional layers of 1x1 respectively to transform into the third data and the fourth data.
[0089] S113: Compress the spatial dimension of the third data to convert its width and height into a 1*1 size;
[0090] S114: Keep the spatial dimension of the fourth data constant;
[0091] S115: Perform information enhancement processing on the compressed third data;
[0092] S116: Dynamically map the fourth data to the information-enhanced third data to control the weights in the hybrid attention module to be between 0 and 1;
[0093] S117: Perform a dot product of the weights with each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding space.
[0094] For example, since the PSA in this embodiment has two branches, after the input data X has received channel-level attention enhancement, it needs to enter the spatial branch for spatial attention enhancement. Figure 5 The specific structure of the spatial branch is shown. Similar to the channel attention module mentioned above, the spatial attention module also starts with two 1x1 convolutional layers. These two convolutional layers process the input features x... c The data is converted to k and v respectively. Then, based on the polarization filtering concept of PSA, the GlobalPooling function is used on the k feature to compress its spatial dimension. The width and height of the k feature are converted to a 1*1 size, while the spatial dimension of the v feature remains unchanged, still at a relatively large level. Since the spatial dimension of the k feature is compressed, this embodiment also uses the Softmax function to enhance the information of k. Finally, similar to channel branching, the obtained k and v are processed using the Sigmoid function to obtain spatial attention weights, which are then compared with the input feature x. c Perform a dot product to complete x c Spatial attention is enhanced. The above processing is illustrated in the formula:
[0095] x c =x⊙σ sm (F z (F q (x)×φ(F v (x))))
[0096] Where, σ sm This refers to the softmax function.
[0097] Preferably, in this embodiment, a spatiotemporal residual module is added after the PSA module to complete the construction of the hybrid attention module. That is, the hybrid attention module in this embodiment also includes a spatiotemporal residual module. Based on the spatiotemporal residual module, while further optimizing spatial features, feature enhancement in the temporal context dimension can be achieved, making up for the shortcomings of the PSA module in temporal processing.
[0098] Specifically, the method further includes:
[0099] S118: Based on the spatiotemporal residual module, perform 2D convolution on the features in the supporting feature set and the query feature set in the spatial dimension to the intermediate subspace;
[0100] S119: Based on the activation function in the spatiotemporal residual module and the BN layer, the features convolved into the intermediate subspace are processed, so that the features are convolved in 1D on the time axis;
[0101] S120: Based on the pooling layer in the spatiotemporal residual module, the features of 1D convolution are processed to obtain the enhanced support feature set and query feature set.
[0102] For example, the input data E(si) and E(q) are first subjected to a 1*d*d 2D convolution in the spatial dimension to an intermediate subspace. Then, after processing by the ReLU activation function and BN layer, a k*1*1 1D convolution is performed in the temporal dimension to enhance feature processing. Finally, the result after enhanced temporal feature processing is processed by the adaptive pooling layer of the spatiotemporal residual module to obtain the final output of the hybrid attention module. To prevent network degradation as the number of network layers increases, this embodiment uses residual connections to connect the PSA module and the spatiotemporal residual module.
[0103] Furthermore, the prototype attention module in this embodiment is used to achieve the final classification of unlabeled videos. In few-shot classification problems, due to insufficient training data, general classification algorithms tend to overfit, resulting in a large error between the classification result and the actual result. To reduce the impact of overfitting caused by insufficient data, this embodiment uses a metric-based prototype network to perform metric classification on the features learned by the previous network, increasing the network's generalization ability.
[0104] Specifically, the prototype attention module in this embodiment has a multi-head self-attention layer;
[0105] The method further includes:
[0106] S121: Based on the enhanced support features and query features obtained from the prototype attention module of the multi-head self-attention layer, the attention scores between features are obtained;
[0107] S122: The attention score and the corresponding enhanced support features or query features are added together as input, and then the target function is used for mapping.
[0108] Among these, when comparing the distances between each first vector space and the second vector space, and determining, based on the comparison results, that the query feature with the closest distance in the vector space belongs to the same category as the supporting feature, the process includes:
[0109] The prototype attention module calculates the distance between the first vector space corresponding to different support features and the second vector space corresponding to different query features based on the metric space, and determines the category to which the query feature with the closest vector space distance belongs to the same category as the support feature based on the comparison results.
[0110] The metric space is learned by the prototype attention module during feature processing in the multi-head self-attention layer.
[0111] For example, after processing by the network module described above, this embodiment represents the obtained feature as x. Before matching the feature vectors, this embodiment first processes the feature x using a standard multi-head self-attention layer. Here, N is the length of the input sequence, and D is the hidden dimension. This embodiment uses a projection matrix W with three learnable parameters. q W k W v To calculate the query, key, and value in the attention layer, the specific mapping relationship can be represented as follows:
[0112] x q =W q x, x k =W k x, x v =W v x
[0113] After obtaining the mapping output of query, key, and value, the obtained x is further processed according to the number h of multi-head attention heads used. q x k x v The data is divided into h equal parts. Each head then calculates its own self-attention mechanism, and the results from each head are concatenated. Finally, the concatenated result is processed through a learnable parameter matrix W. o The samples are then fused. This yields attention scores between the samples, which are added to the initial feature input as a benefit. The above process can be represented as:
[0114]
[0115] Where d k It represents x kThe dimension, σ sm This represents the softmax function. Both the query set and the support set undergo processing by the multi-head attention module described above. However, it's important to note that during query set processing, both the key and value inputs are provided by the support set. Specifically, this can be represented as:
[0116]
[0117] Furthermore, the model proposed in this embodiment strengthens the connection between class prototype samples, greatly assisting in the classification of sample points with small inter-class differences and the classification of outlier and overlapping sample points. After multi-head attention processing, the prototype attention module learns a metric space in which classification is performed by calculating the distance between samples in the query set and the prototype representations (i.e., vector spaces) of each class in the support set. The sample is classified as the one closest to the prototype representation of the class it is closest to. This embodiment preferably uses the cosine function to measure the distance between sample points (feature vectors). Specifically, the distance function D can be expressed as the formula:
[0118]
[0119] Where Π refers to the total number of sample points, and Q G S represents the sample points (query feature vector) in the query set. G This represents the sample points (support feature vectors) in the support set. Furthermore, this embodiment preferably uses the cross-entropy cost function to calculate the network loss, where... Let represent the predicted label and y represent the true label. The loss function of the entire model can be simply expressed as:
[0120]
[0121] The small sample behavior recognition method described in the above embodiments can achieve the following technical effects:
[0122] 1. Compared to existing few-sample behavior recognition methods, this embodiment draws inspiration from traditional video understanding tasks and uses R(2+1)D as the backbone network to extract features from video behavior sequences. The R(2+1)D network can jointly model the spatial and temporal dimensions of the input information, without requiring separate processing of the temporal dimension.
[0123] 2. This embodiment proposes a hybrid attention module. In the hybrid attention module, a polarized self-attention mechanism is introduced to enhance the spatial dimension information features. After the polarized self-attention mechanism, this embodiment also adds an additional residual-connected (2+1)D convolution layer to simultaneously process spatial and temporal information, thus fully enhancing the temporal and spatial features.
[0124] 3. Small sample data is extremely prone to overfitting. Based on this consideration, this embodiment also proposes a prototype attention module. The prototype attention module follows the architecture of the prototype network and adds a multi-head attention mechanism before measuring matching to learn the relationship between sample points between classes in the vector space, effectively solving the overfitting problem in small sample training.
[0125] like Figure 7 As shown, another embodiment of the present invention also provides a small sample behavior recognition system 100, comprising:
[0126] The building module is used to build a few-shot behavior recognition model. The few-shot behavior recognition model includes a feature embedding module for simultaneously extracting and processing spatial and temporal features of the video, a hybrid attention module for capturing and processing spatial context features of the video, and a prototype attention module for implementing feature classification of the video.
[0127] The preprocessing module is used to preprocess the video to be processed to obtain a support set with category labels and a query set without category labels. Both the support set and the query set are formed by video frames with time sequence.
[0128] The input module is used to input the support set and query set into the few-sample behavior recognition model;
[0129] The few-sample behavior recognition model is used to process the support set and query set according to the feature embedding module to obtain the support feature set and query feature set respectively; the hybrid attention module enhances the features in the support feature set and query feature set; the prototype attention module maps the support features in the enhanced support feature set to different first vector spaces based on the target mapping function, and at the same time maps the query features in the enhanced query feature set to different second vector spaces based on the target mapping function. The distance between each first vector space and the second vector space is compared, and based on the comparison results, it is determined that the query feature with the closest distance in the vector space belongs to the same category as the support feature.
[0130] As an optional embodiment, the feature embedding module consists of a 2D convolutional network and a 1D convolutional network. The 2D convolutional network is used to process the spatial data of the video, and the 1D convolutional network is used to process the temporal features of the video to obtain the temporal relationship of multiple video frames in the support set and query set.
[0131] As an optional embodiment, the hybrid attention module introduces a serial polarized attention mechanism, which performs data processing on the input channel, the width and height of the supporting features or query features of the few-sample behavior recognition model.
[0132] As an optional embodiment, at least based on the hybrid attention module, features in the supporting feature set and the query feature set are enhanced, including:
[0133] Based on the polarized attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then enters the first and second convolutional layers of 1x1 respectively to transform into the first data and the second data. The input channel of the first data is compressed to 1, and the input channel dimension of the second data is C / 2, where C is the number of input channels of the hybrid attention module.
[0134] Information enhancement processing is performed on the first data based on the objective function;
[0135] The second data is matrix-multiplied with the enhanced first data, and a 1x1 convolution and a feature normalization layer from the hybrid attention module are applied to the multiplication result to increase the input channel dimension of the second data to C.
[0136] The first and second data are dynamically mapped to ensure that the weights in the hybrid attention module are all between 0 and 1.
[0137] The weights are multiplied by each feature in both the support feature set and the query feature set to enhance the attention of each feature on the corresponding input channel.
[0138] As an optional embodiment, at least based on the hybrid attention module, features in the supporting feature set and the query feature set are enhanced, including:
[0139] Based on the polarization attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, which are then transformed into third and fourth data by 1x1 convolutional layers respectively.
[0140] The third data is spatially compressed so that its width and height are converted to form a 1*1 size;
[0141] Keep the spatial dimension of the fourth data constant;
[0142] The compressed third-party data is then subjected to information enhancement processing.
[0143] The fourth data and the information-enhanced third data are dynamically mapped to control the weights in the hybrid attention module to be between 0 and 1.
[0144] The weights are multiplied by each feature in both the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding space.
[0145] As an optional embodiment, the hybrid attention module also includes a spatiotemporal residual module;
[0146] The system also includes:
[0147] The spatiotemporal residual module performs 2D convolutions on the features in the support feature set and query feature set in the spatial dimension to an intermediate subspace. Based on the activation function and BN layer in the spatiotemporal residual module, the features convolved to the intermediate subspace are processed, and then 1D convolutions are performed on the feature in the time axis. Then, the features that have undergone 1D convolutions are processed by the pooling layer in the spatiotemporal residual module to obtain the enhanced support feature set and query feature set.
[0148] As an optional embodiment, the prototype attention module has a multi-head self-attention layer;
[0149] The prototype attention module is also used for:
[0150] Based on the enhanced support features and query features obtained from the prototype attention module of the multi-head self-attention layer, attention scores between features are obtained.
[0151] The attention score and the corresponding enhanced support features or query features are added together as input, and then combined with the objective function for mapping processing.
[0152] As an optional embodiment, the distances between each first vector space and the second vector space are compared, and based on the comparison results, it is determined that the query feature with the closest distance in the vector space belongs to the same category as the supporting feature, including:
[0153] The prototype attention module calculates the distance between the first vector space corresponding to different support features and the second vector space corresponding to different query features based on the metric space, and determines the category to which the query feature with the closest vector space distance belongs to the same category as the support feature based on the comparison results.
[0154] The metric space is learned by the prototype attention module during feature processing in the multi-head self-attention layer.
[0155] Another embodiment of the present invention also provides a small sample behavior recognition device, comprising:
[0156] At least one processor; and,
[0157] A memory communicatively connected to the at least one processor; wherein,
[0158] The memory stores instructions that can be executed by the at least one processor to implement the few-sample behavior recognition method as described in any of the embodiments above.
[0159] Furthermore, one embodiment of the present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the small sample behavior recognition method described above. It should be understood that the various solutions in this embodiment have the corresponding technical effects in the above method embodiments, and will not be repeated here.
[0160] Furthermore, embodiments of the present invention also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions that, when executed, cause at least one processor to perform a few-sample behavior recognition method as described in the embodiments above.
[0161] It should be noted that the computer storage medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.
[0162] Furthermore, those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0163] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.
[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0165] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0166] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
[0167] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.
Claims
1. A method for identifying behavior in small samples, characterized in that, include: A few-shot behavior recognition model is constructed, which includes a feature embedding module for simultaneously extracting and processing spatial and temporal features of the video, a hybrid attention module for capturing and processing the spatial context features of the video, and a prototype attention module for implementing feature classification of the video. The video to be processed is preprocessed to obtain a support set with category labels and a query set without category labels. Both the support set and the query set are formed by video frames with time sequence. The support set and query set are input into the small sample behavior recognition model; Based on the feature embedding module, the support set and query set are processed to obtain the support feature set and query feature set, respectively. At least based on the hybrid attention module, the features in the supporting feature set and the query feature set are enhanced; Based on the prototype attention module, the support features in the enhanced support feature set are mapped to different first vector spaces using the target mapping function. At the same time, the query features in the enhanced query feature set are mapped to different second vector spaces using the target mapping function. The distances between each first vector space and the second vector space are compared, and based on the comparison results, it is determined that the query feature with the closest vector space distance belongs to the same category as the support feature. The hybrid attention module introduces a serial polarized attention mechanism, which performs data processing on the input channel, the width and height of the supporting features or query features of the few-sample behavior recognition model based on the polarized attention mechanism. The enhancement processing of features in the supporting feature set and the query feature set based at least on the hybrid attention module includes: Based on the polarized attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then enters a 1x1 first convolutional layer and a second convolutional layer respectively to transform into first data and second data. The input channel of the first data is compressed to 1, and the input channel dimension of the second data is C / 2, where C is the number of input channels of the hybrid attention module. The first data is augmented based on the objective function; The second data is matrix-multiplied with the enhanced first data, and a 1x1 convolution and the feature normalization layer in the hybrid attention module are applied to the multiplication result to increase the input channel dimension of the second data to C. The first data and the second data are dynamically mapped to control the weights in the hybrid attention module to be between 0 and 1. The weights are multiplied by each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding input channel.
2. The small sample behavior recognition method according to claim 1, characterized in that, The feature embedding module consists of a 2D convolutional network and a 1D convolutional network. The 2D convolutional network is used to process the spatial data of the video, and the 1D convolutional network is used to process the temporal features of the video to obtain the temporal relationship of multiple video frames in the support set and query set.
3. The small sample behavior recognition method according to claim 1, characterized in that, The enhancement processing of features in the supporting feature set and the query feature set based at least on the hybrid attention module includes: Based on the polarized attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then transformed into third and fourth data by 1x1 convolutional layers respectively. The third data is spatially compressed to its width and height. Dimensions; The spatial dimension of the fourth data remains constant; The compressed third data is then subjected to information enhancement processing; The fourth data is dynamically mapped to the information-enhanced third data to control the weights in the hybrid attention module to be between 0 and 1. The weights are multiplied by each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding space.
4. The small sample behavior recognition method according to claim 1, characterized in that, The hybrid attention module also includes a spatiotemporal residual module; The method further includes: Based on the spatiotemporal residual module, 2D convolution is performed on the features in the supporting feature set and the query feature set in the spatial dimension to the intermediate subspace; Based on the activation function and BN layer in the spatiotemporal residual module, the features convolved into the intermediate subspace are processed, and the features are then subjected to 1D convolution on the time axis. Based on the features of 1D convolution processed by the pooling layer in the spatiotemporal residual module, the enhanced support feature set and query feature set are obtained.
5. The small sample behavior recognition method according to claim 1, characterized in that, The prototype attention module has a multi-head self-attention layer; The method further includes: Based on the enhanced support features and query features obtained by the prototype attention module through the multi-head self-attention layer, attention scores between features are obtained. The attention score and the corresponding enhanced support feature or query feature are input and added together, and then combined with the objective function for mapping processing.
6. The small sample behavior recognition method according to claim 5, characterized in that, The step of comparing the distances between each of the first vector spaces and the second vector spaces, and determining, based on the comparison results, that the query feature with the closest vector space distance belongs to the same category as the supporting feature, includes: The prototype attention module calculates the distance between the first vector space corresponding to different supporting features and the second vector space corresponding to different query features based on the metric space, and determines the category to which the query feature with the closest vector space distance belongs to the same category as the supporting feature based on the comparison results. The metric space is learned by the prototype attention module during feature processing in the multi-head self-attention layer.
7. A small-sample behavior recognition system, characterized in that, include: A construction module is used to build a few-shot behavior recognition model. The few-shot behavior recognition model includes a feature embedding module for simultaneously extracting and processing spatial and temporal features of the video, a hybrid attention module for capturing and processing the spatial context features of the video, and a prototype attention module for implementing feature classification of the video. The preprocessing module is used to preprocess the video to be processed to obtain a support set with category labels and a query set without category labels. Both the support set and the query set are formed by video frames with time sequence. The input module is used to input the support set and query set into the small sample behavior recognition model; A small sample behavior recognition model is used to process the support set and query set according to the feature embedding module to obtain the support feature set and query feature set respectively. The hybrid attention module performs enhancement processing on the features in the support feature set and the query feature set; According to the prototype attention module, the support features in the enhanced support feature set are mapped to different first vector spaces based on the target mapping function, and the query features in the enhanced query feature set are mapped to different second vector spaces based on the target mapping function. The distance between each first vector space and the second vector space is compared, and based on the comparison results, it is determined that the query feature with the closest vector space distance belongs to the same category as the support feature. The hybrid attention module introduces a serial polarized attention mechanism, which performs data processing on the input channel, the width and height of the supporting features or query features of the few-sample behavior recognition model based on the polarized attention mechanism. The enhancement processing of features in the support feature set and query feature set based on the hybrid attention module includes: Based on the polarized attention mechanism, each feature in the supporting feature set and the query feature set is divided into two branches, and then enters a 1x1 first convolutional layer and a second convolutional layer respectively to transform into first data and second data. The input channel of the first data is compressed to 1, and the input channel dimension of the second data is C / 2, where C is the number of input channels of the hybrid attention module. The first data is augmented based on the objective function; The second data is matrix-multiplied with the enhanced first data, and a 1x1 convolution and the feature normalization layer in the hybrid attention module are applied to the multiplication result to increase the input channel dimension of the second data to C. The first data and the second data are dynamically mapped to control the weights in the hybrid attention module to be between 0 and 1. The weights are multiplied by each feature in the supporting feature set and the query feature set to achieve attention enhancement for each feature in the corresponding input channel.
8. A small-sample behavior recognition device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to implement the few-sample behavior recognition method as described in any one of claims 1-6.