A small sample behavior recognition method based on edge-guided motion information promotion network

Abstract

The application discloses a small sample behavior recognition method based on edge guidance motion information promotion network, and relates to computer vision technology.A. Given a video dataset, randomly extract T frames of each video to form a new video frame sequence.B. The video frame sequence is input into a feature extractor to obtain video frame features.Meanwhile, the sampled video frame sequence is input into an edge information extractor to obtain edge information features.C. The video frame features and the edge information features generated in the step B are respectively input into a motion information promotion module to obtain promoted video frame features and promoted edge information features.D. The two kinds of features promoted in the step C are fused and input into a predictor to obtain a category prediction result, and a category loss is generated.E. The edge information features promoted in the step C are input into the predictor to obtain an edge loss, and the category loss generated in the step D is combined with the edge loss to form a loss function of the network.Compared with current mainstream small sample behavior recognition methods, the classification performance is improved.

Description

Technical Field

[0001] This invention relates to computer vision technology, specifically to a few-sample behavior recognition method based on edge-guided motion information enhancement networks. Background Technology

[0002] In recent years, behavior recognition has played a crucial role in the multimedia field, attracting increasing attention due to its diverse applications in traffic management, security monitoring, and autonomous driving. However, it remains a challenging task because videos typically contain complex temporal information that is difficult to model. The performance of most existing methods still falls far short of the capabilities of the human visual system. Therefore, the field of behavior recognition holds significant research value. Deep learning has achieved remarkable success in behavior recognition. However, modern deep learning methods heavily rely on large amounts of manually labeled data, and acquiring this data is extremely costly. In contrast, humans can learn new concepts using prior knowledge even with only a few samples. To alleviate the dependence on massive amounts of data, few-shot learning has become a popular research area, aiming to accomplish corresponding tasks using limited labeled data.

[0003] Considering that deep learning models may suffer from low generalization ability when trained with only a small amount of labeled data, many researchers have focused on few-shot image classification, achieving promising results. However, directly applying few-shot image classification methods to few-shot action recognition tasks does not yield ideal results. Compared to images, videos have a more complex structure, containing rich and important contextual information. Therefore, few-shot action recognition is a promising and challenging research direction, with its main challenge lying in how to fully utilize the contextual information in videos. Current few-shot action recognition methods have achieved good results, but they often neglect the effective use of edge information and motion information. Summary of the Invention

[0004] The purpose of this invention is to address the problem of spatiotemporal information mining in few-sample behavior recognition by providing a few-sample behavior recognition method based on an edge-guided motion information enhancement network that can simultaneously and effectively capture edge information and motion information and has good recognition performance.

[0005] The method of this invention comprises two key components: an edge information extractor and a motion information enhancement module. Specifically, the edge information extractor, which includes a semantic enhancement module, acquires edge information features of video frames. These edge information features are then used as guidance to fuse with the original features. Furthermore, the motion information enhancement module adaptively captures motion-sensitive features in the video. It employs a self-gating mechanism to highlight motion-sensitive regions of the video from a larger temporal receptive field.

[0006] This invention includes the following steps:

[0007] A. Given a small sample behavior recognition dataset containing several videos, each consisting of several video frames; a sparse temporal sampling strategy is used to sample the videos to obtain a video frame sequence containing T video frames; specifically, firstly, a video V is randomly divided into T video segments; then, one frame is randomly selected from each video segment; therefore, the T randomly selected video frames constitute a new video frame sequence V′={I1,I2,…,I… i ,…,I T}; where I i This represents a video frame randomly selected from the i-th video segment.

[0008] B. Input the video frame sequence generated in step A into the feature extractor (ResNet-50) to obtain the video frame features F = {f1, f2, ..., f...} t ,…,f T}; where f t Let E represent the features of the t-th video frame; simultaneously, the video frame sequence generated in step A is also input into the edge information extractor to obtain the edge information features E = {e1, e2, ..., e}. t ,…,e T Specifically, the edge information extractor includes an edge extraction operator and a semantic enhancement (CE) module.

[0009] For the CE module, assuming its input is X, it is processed using value transformation functions, query transformation functions, and key transformation functions respectively. After the value transformation function and the key transformation function, a pyramid pooling block is added to obtain the value vector, query vector, and key vector. After certain processing of these three, the context information is obtained. This process is defined as follows:

[0010] C=P(V(X))·S([P(K(X))·Q(X)]),

[0011] Where C represents context information; Q(·), K(·), and V(·) represent query transformation function, key transformation function, and value transformation function, respectively; P(·) represents pyramid pooling block; and S(·) represents activation function. The context information is then processed by convolutional layers and concatenated with the original input X. Finally, it is processed by convolutional layers, batch regularization layers, and random deactivation layers to obtain the final module output Y. The CE module imitates the human brain's visual system, enriching spatial information and highlighting foreground targets, which is beneficial for more effectively extracting edge information.

[0012] C. Input the video frame features F and edge information features E generated in step B into the motion enhancement (ME) module to obtain the enhanced video frame features. and the enhanced edge information features The ME module adaptively emphasizes motion-related regions in the video using a global self-gating mechanism, suppressing irrelevant information from a larger temporal receptive field. Specifically, it first uses a 1×1 2D convolutional layer conv1 to reduce the number of feature channels, thereby reducing computational cost; then, it uses a global average pooling layer to process the pooled features X. avg Then calculate the transformation time difference D between adjacent frames. t The specific definition is as follows:

[0013]

[0014] Here, conv2 represents a 1×1 channel transform convolutional layer; the difference value at the last time point is set to 0, i.e., D. T =0; To expand the temporal receptive field, a self-gating mechanism is used to activate motion-sensitive regions in the video; an accumulation operation is used to calculate the accumulated difference A. t The specific definition is as follows:

[0015]

[0016] Set A1 = D1; then use a learnable parameter λ to adjust and balance the effects of accumulated difference and time difference; the specific definition is as follows:

[0017] λ=σ(L([A t D t ])),

[0018] M t =λ·D t +(1-λ)·A t ,

[0019] Where σ(·) represents the sigmoid function; L(·) represents the multilayer perceptron; [·,·] represents the concatenation operation; M t The adjusted motion features are then concatenated to obtain the final adjusted features M = [M1, ..., M2]. T Then, a convolutional layer conv3 is used to adjust the number of channels and activate the motion-sensitive channel to obtain the enhanced features. The specific definitions are as follows:

[0020]

[0021] Where ⊙ represents channel multiplication operation;

[0022] D. The enhanced video frame features obtained in step C and the enhanced edge information features Fusion to enhance edge information features The enhanced video frame features are used as guidance to enrich the features, and then the fused features are input into the predictor to obtain the classification prediction. The cross-entropy loss is used to calculate the class loss. and the enhanced edge information features The fusion process is specifically defined as follows:

[0023]

[0024] Here, conv4 represents a convolutional layer.

[0025] E. The enhanced edge information features obtained in step C. The input predictor yields the edge loss, which, together with the category loss generated in step D, forms the network's loss function, used to train the network.

[0026] In step A, the number of frames in the video frame sequence is set to T = 8.

[0027] This invention proposes an effective few-shot behavior recognition method based on edge-guided motion information enhancement networks, incorporating edge information as guidance. To effectively utilize edge information in videos, this invention proposes an edge information extractor containing a novel semantic enhancement module. The semantic enhancement module enriches spatial information and highlights foreground objects, facilitating more effective edge information extraction. Then, the edge information is used as guidance and fused with the original features to enhance the features of the video frames. Furthermore, motion information is crucial for behavior recognition. Motion and edge information are complementary; rich motion information helps to effectively extract edge information, and vice versa. Therefore, this invention proposes a novel motion information enhancement module that adaptively captures motion information in videos. This module utilizes a global self-gating mechanism to adaptively emphasize motion-related regions in the video, suppressing irrelevant information from a larger temporal receptive field. In this way, the method of this invention can effectively and comprehensively utilize edge and motion information to achieve accurate few-shot behavior recognition. Moreover, edge and motion information are mutually reinforcing; effectively extracting edge information helps to better capture motion information, and vice versa. This method addresses the spatiotemporal information mining problem in small-sample behavior recognition, achieving accuracy superior to mainstream small-sample behavior recognition methods. This invention is primarily applied to behavior recognition in situations with insufficient labeled data (i.e., small samples). Therefore, this invention can be applied to various intelligent systems, such as smart homes and smart surveillance. It can analyze and determine the behavior categories in videos to aid the system in making further decisions. Attached Figure Description

[0028] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the Semantic Enhancement Module (CE module).

[0030] Figure 3 This is a schematic diagram of the motion information enhancement module. Detailed Implementation

[0031] The method of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. This embodiment is implemented under the premise of the technical solution of the present invention, and provides implementation methods and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.

[0032] like Figure 1 As shown, the implementation of this embodiment of the invention includes the following steps:

[0033] A. Given a small sample behavior recognition dataset containing several videos, each video consisting of several video frames; this invention uses a sparse temporal sampling strategy to sample the videos, aiming to obtain a video frame sequence containing T video frames; specifically, firstly, a video V is randomly divided into T video segments; then, one frame is randomly extracted from each video segment; therefore, the T randomly extracted video frames constitute a new video frame sequence V′={I1,I2,…,I… i ,…,I T}; where I i This represents a video frame randomly selected from the i-th video segment; the number of frames in the video frame sequence is set to T = 8.

[0034] B. Input the video frame sequence generated in step A into the feature extractor (ResNet-50) to obtain the video frame features F = {f1, f2, ..., f...} t ,…,f T}; where f t Let E represent the features of the t-th video frame; simultaneously, the video frame sequence generated in step A is also input into the edge information extractor to obtain the edge information features E = {e1, e2, ..., e}. t ,…,e T Specifically, the edge information extractor includes an edge extraction operator (the Canny operator is selected here) and a semantic enhancement (CE) module.

[0035] like Figure 2As shown, for the CE module, assuming the input is X, it is processed using value transformation, query transformation, and key transformation functions respectively. After the value and key transformation functions, a pyramid pooling block is added to process the features, resulting in value vectors, query vectors, and key vectors. The pyramid pooling block improves edge features by generating a hierarchical global context prior containing rich information at different scales. Compared to the simplest global average pooling operation, the pyramid pooling block can preserve more spatial relationships and reduce information loss. It first uses multi-scale pooling kernels to generate multiple pooled feature maps, then uses 1×1 convolutional layers to reduce the channel dimension of these multi-scale feature maps. Then, these low-dimensional feature maps are upsampled to obtain new feature maps of the same size as the input feature map of the pyramid pooling block. Finally, the upsampled features are concatenated to obtain the final pyramid pooled features. The above process is defined as follows:

[0036] C=P(V(X))·S([P(K(X))·Q(X)]),

[0037] Where C represents context information; Q(·), K(·), and V(·) represent the query transformation function, key transformation function, and value transformation function, respectively; P(·) represents the pyramid pooling block; and S(·) represents the softmax activation function. The context information is then processed by convolutional layers and concatenated with the original input X. Finally, it passes through convolutional layers, batch regularization layers, and random deactivation layers to obtain the final module output Y. The CE module mimics the human brain's visual system, enriching spatial information and highlighting foreground targets, which is beneficial for more effectively extracting edge information.

[0038] C. Input the video frame features F and edge information features E generated in step B into the motion enhancement (ME) module to obtain the enhanced video frame features. and the enhanced edge information features

[0039] like Figure 3 As shown, the ME module adaptively emphasizes motion-related regions in the video using a global self-gating mechanism, suppressing irrelevant information from a larger temporal receptive field. Specifically, assuming its input is X (which in this invention is the video frame feature F and edge information feature E), a 1×1 2D convolutional layer conv1 is first used to reduce the number of feature channels, thereby reducing computational cost; then a global average pooling layer is used to process the pooled feature X. avg ; Calculate the transformation time difference D between adjacent frames t The specific definition is as follows:

[0040]

[0041] Where conv2 represents a 1×1 channel transform convolutional layer, and t represents the time axis dimension of the video;

[0042] Set the difference value at the last time point to 0, i.e., D. T =0; To expand the temporal receptive field, a self-gating mechanism is used to activate motion-sensitive regions in the video to obtain the final enhanced features; an accumulation operation is used to calculate the accumulated difference A. t The specific definition is as follows:

[0043]

[0044] Set A1 = D1; use a learnable parameter λ to adjust and balance the effects of accumulated differences and time differences; the specific definition is as follows:

[0045] λ=σ(L([A t D t ])),

[0046] M t =λ·D t +(1-λ)·A t ,

[0047] Where σ(·) represents the sigmoid function; L(·) represents the multilayer perceptron; [·,·] represents the concatenation operation; M t The adjusted motion features are then concatenated to obtain the final adjusted features M = [M1, ..., M2]. T Then, a convolutional layer (conv3) is used to adjust the number of channels and activate the motion-sensitive channels to obtain the enhanced features. (In this invention, the enhanced video frame features) and the enhanced edge information features The specific definition is as follows:

[0048]

[0049] Here, ⊙ represents the channel multiplication operation.

[0050] D. The enhanced video frame features obtained in step C and the enhanced edge information features Fusion, that is, using the enhanced edge information features This serves as a guide to further enrich video frame features. The fused features are then input into a similarity category predictor, which calculates the similarity "distance" between the query features and the support features to classify the query video, determining which category it belongs to in the support set, ultimately obtaining the behavior category recognition result. Cross-entropy loss is then used to calculate the category loss; the enhanced video frame features... and the enhanced edge information features The fusion process is specifically defined as follows:

[0051]

[0052] Here, conv4 represents a convolutional layer.

[0053] E. The enhanced edge information features obtained in step C. Input the same similarity category predictor as mentioned in step D to obtain the edge loss, which together with the category loss generated in step D forms the network's loss function. This function is used to train the network and optimize the parameters to improve the recognition accuracy of the behavior category mentioned in step D.

[0054] This invention conducts experiments on four datasets: UCF101, HMDB51, SSv2, and Kinetics, and compares them with other state-of-the-art methods. For fairness, following the processing methods of other methods, this invention crops video frames to a size of 224*224; uses a ResNet-50 pre-trained on ImageNet as the feature extractor; and trains the network end-to-end using a stochastic gradient descent optimizer. For experiments on the SSv2 dataset, the learning rate is set to 0.001 due to its large data volume. The learning rate is also set to 0.0001 for experiments on other datasets. The average accuracy across 10,000 tasks randomly selected from the test set is used as the final performance evaluation. Table 1 shows the comparative experimental results on the UCF101 and HMDB51 datasets.

[0055] Table 1

[0056]

[0057] Table 2 shows the comparative experimental results on the SSv2 and Kinetics datasets.

[0058] Table 2

[0059]

[0060] The bold text in the tables indicates the best results by comparison. Observing Tables 1 and 2 reveals that, compared to these state-of-the-art methods, this invention achieves optimal classification performance in most cases. For example, as shown in Table 2, on the SSv2 dataset, this invention achieves comprehensive superiority over the STRM method published at CVPR 2022 (a top conference in computer vision). Especially in the 1-shot setting, this invention surpasses STRM in accuracy by as much as 8.2%, because with less data, fully extracting and utilizing the information in the data (such as the edge and motion information that this invention focuses on) becomes even more important. These comparative experimental results fully validate the effectiveness of this invention.

[0061] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A few-sample behavior recognition method based on edge-guided motion information enhancement network, characterized in that... Includes the following steps: A. Given a small sample behavior recognition dataset containing several videos, sample the videos and randomly select... The frames form a new video frame sequence; B. Input the sampled video frame sequence obtained in step A into the feature extractor to obtain video frame features; at the same time, input the sampled video frame sequence into the edge information extractor to obtain edge information features. C. Analyze the video frame features generated in step B. and edge information features Each feature is input into the ME module to obtain the enhanced video frame features. and the enhanced edge information features ; The ME module adaptively emphasizes motion-related regions in a video using a global self-gating mechanism, suppressing irrelevant information from a larger temporal receptive field; specifically, let its input be... First, adopt a 2D convolutional layers Reduce the number of feature channels to reduce computational cost; use a global average pooling layer to process the pooled features. ; Calculate the transformation time difference between adjacent frames The specific definition is as follows: in, Represent a Channel transformation convolutional layer; set the difference value at the last time point to 0, i.e. To expand the temporal receptive field, a self-gating mechanism is used to activate motion-sensitive regions in the video; an accumulation operation is used to calculate the accumulated difference. The specific definition is as follows: set up Use a learnable parameter. Adjustments are made to balance the effects of accumulated differences and time differences; the specific definitions are as follows: in, Represents the sigmoid function; Represents a multilayer perceptron; Indicates a splicing operation; The adjusted motion features are then concatenated to obtain the final adjusted features. Then use a convolutional layer Adjusting the number of channels and activating motion-sensitive channels yields improved features. The specific definition is as follows: in, This indicates a channel multiplication operation; D. Fuse the enhanced video frame features and enhanced edge information features obtained in step C, and input the fused features into the similarity category predictor, that is, calculate the similarity "distance" between the query features and the support features to classify the query video, determine which category it belongs to in the support set, obtain the behavior category recognition result, and calculate the category loss. E. Input the improved edge information features obtained in step C into the same similarity category predictor as in step D to obtain the edge loss. This edge loss, together with the category loss generated in step D, forms the network's loss function, which is used to train the network and optimize the network parameters to improve the recognition accuracy of the behavior category in step D.

2. The few-sample behavior recognition method based on edge-guided motion information enhancement network as described in claim 1, characterized in that... In step A, each video is randomly selected. Frames form a new video frame sequence, with each video consisting of several video frames; a sparse temporal sampling strategy is used to sample the video to obtain a sequence containing... A sequence of video frames; specifically, first, a video... Randomly divided into Each video segment is divided into several parts, and then one frame is randomly selected from each video segment; the randomly selected frame... These video frames constitute a new video frame sequence. ;in, Indicates from the first Video frames randomly selected from a video segment.

3. The few-sample behavior recognition method based on edge-guided motion information enhancement network as described in claim 1, characterized in that... The number of frames in the video frame sequence is set to .

4. The few-sample behavior recognition method based on edge-guided motion information enhancement network as described in claim 1, characterized in that... In step B, the feature extractor uses ResNet-50. Obtaining video frame features involves inputting the video frame sequence generated in step A into the ResNet-50 feature extractor to acquire the video frame features. ;in, Indicates the first Features of each video frame; The edge information extractor consists of an edge extraction operator and a CE module; the video frame sequence generated in step A is input into the edge information extractor to obtain edge information features. ; Specifically, for the CE module, let its input be... The query vector, query vector, and key vector are processed using value transformation functions, query transformation functions, and key transformation functions, respectively. After the value transformation function and the key transformation function, a pyramid pooling block is added to obtain the value vector, query vector, and key vector. These three vectors are then processed to obtain the context information. This process is defined as follows: in, Indicates contextual information; , and These represent query transformation functions, key transformation functions, and value transformation functions, respectively. Represents a pyramid pooling block; Indicates the activation function; Contextual information is processed by convolutional layers and combined with the original input. After splicing and merging, the modules are processed through convolutional layers, batch regularization layers, and random deactivation layers to obtain the final module output. The CE module mimics the human brain's visual system, enriching spatial information, highlighting foreground targets, and effectively extracting edge information.

5. The few-sample behavior recognition method based on edge-guided motion information enhancement network as described in claim 1, characterized in that... In step D, the enhanced video frame features and enhanced edge information features obtained in step C are fused, and the fused features are input into a similarity category predictor. This involves calculating the similarity "distance" between the query features and the support features to classify the query video, determining which category it belongs to in the support set, obtaining the behavior category identification result, and calculating the category loss. Specific steps include: The enhanced video frame features obtained in step C and the enhanced edge information features Fusion, that is, using the enhanced edge information features As a guide, the enhanced video frame features were further enriched. The fused features are then input into a similarity category predictor to obtain the behavior category recognition result, and the category loss is calculated using cross-entropy loss; the enhanced video frame features and the enhanced edge information features The fusion process is specifically defined as follows: in, This represents a convolutional layer.

Images