A small sample action recognition method based on a graph neural network

Abstract

A small sample action recognition method based on a graph neural network, comprising the following steps: S1, acquiring a video and extracting video features; S2, remodeling and enhancing the video features to obtain video timing features; S3, performing average pooling operation on all video timing features to form corresponding node features, and using the node features to construct edge features, the node features and the edge features are input into a pre-trained graph network for feature propagation and update, and task-oriented features of a query video and a class support set video are calculated; S4, performing class matching on the task-oriented features to obtain a class prediction value of the query video; S5, identifying the action of the actor in the query video through the class prediction value, wherein the method can accurately and quickly identify the action of the actor in the actual scene with only a small amount of labeled training data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and particularly to a small sample action recognition method based on a graph neural network. BACKGROUND

[0002] Compared with ordinary action recognition, few-shot action recognition only needs limited labeled samples to quickly learn new categories. It can avoid large-scale, time-consuming and labor-intensive data annotation usually related to supervised tasks, making it more suitable for industrial applications. Based on this advantage, more and more attention has been paid to the field of few-shot action recognition. However, due to the limited learning materials of few-shot action recognition, it is challenging to learn a well-generalized model.

[0003] Few-shot image classification uses a few-shot training method to use a small number of labeled training samples from similar tasks to represent a large number of labeled training samples. In recent years, few-shot image classification research can mainly be divided into two categories: adaptation-based and metric-based. The adaptation-based method aims to find a network initialization that can be fine-tuned by a small number of labeled data to adapt to unknown tasks, called "gradient by gradient". The classic adaptation method includes meta-learning (MAML, Model-Agnostic Meta-Learning), Reptile. The metric-based method aims to learn a feature space and compare task features through different matching strategies, called "learning to compare". Representative methods include the prototype network Prototypical Networks, the matching network Matching Networks. There are also many methods that aim to improve these methods or are inspired by these methods and belong to metric-based.

[0004] The core idea of few-shot action recognition is similar to few-shot image classification, but the former task is more complex than the latter due to the additional time dimension. Adaptation-based methods (e.g., MetaUVFS) are less concerned in few-shot action recognition due to high computational resource and experimental time requirements. Existing research mainly applies metric learning, but focuses on different aspects. Some methods focus on the enhancement of feature representation. For example, STRM adopts local and global enhancement modules for spatio-temporal modeling, HyRSM uses hybrid relational modeling to learn task-specific embeddings, and SloshNet utilizes a feature fusion architecture search module to exploit low-level spatial features and long-term and short-term temporal modeling modules to encode complementary global and local temporal representations. Other methods focus on the matching strategy of class prototypes. For example, OTAM proposes a temporal alignment module to calculate distance values between the query video and the support set video, TRX matches each query sub-sequence with all sub-sequences in the support set, and HyRSM designs a bidirectional average Hausdorff metric to more flexibly find the correspondence between different videos. In addition, TRPN and MORN focus on combining visual and semantic features, and AMeFu-Net focuses on utilizing depth information to assist learning. However, the existing methods still have the problems of low accuracy and slow identification of target actions in the few-labeled sample recognition task. SUMMARY

[0005] In view of the above problems, the present application provides a few-shot action recognition method based on a graph neural network, which can accurately and quickly identify actions in actual scenes with only a small amount of labeled training data.

[0006] In order to achieve the above purpose, the present application provides a few-shot action recognition method based on a graph neural network, comprising:

[0007] Step S1: obtaining a query video and a class support set video, and inputting them into a pre-trained feature extraction network for feature extraction to obtain a query video feature and a class support set video feature;

[0008] Step S2: inputting all video features in step S1 into a pre-trained feature enhancement network for remodeling and enhancement to obtain video temporal features; the video temporal features include query video temporal features and class support set video temporal features;

[0009] Step S3: performing average pooling operation on all video time sequence features in step S2 to form corresponding node features, and constructing edge features using the node features, the edge features being used to represent the category relevance between any two node features, the node features and the edge features being input into a pre-trained graph network for feature propagation and updating, and task-oriented features of the query video and the category support set videos being calculated according to results output by the graph network after propagation and updating;

[0010] Step S4: performing category matching on the task-oriented features of the query video and the category support set videos through a pre-trained hybrid matching network to obtain a category prediction value of the query video;

[0011] Step S5: identifying the action of the actor in the query video through the category prediction value.

[0012] Preferably, step S3 includes the following steps:

[0013] Step S31: performing average pooling operation on all video time sequence features in step S2 to form corresponding node features, denoted as V; and constructing edge features using the node features, denoted as A;

[0014] Node features

[0015] v i denotes the i-th node feature, denotes the total number of node features of the category support set videos and the query video in the graph network, and the node features are initialized;

[0016] Edge features

[0017] a ij denotes the edge feature between the i-th node feature and the j-th node feature, denotes the total number of node features of the category support set videos and the query video in the graph network, and the edge features are initialized using real category labels;

[0018] Step S32: the node features and the edge features are input into a pre-trained L-layer graph network for feature propagation and updating, and the steps of propagation and updating specifically include: all initial node features and initial edge features are propagated to the first layer graph network, the updated node features and edge features output by the first layer graph network are calculated, the updated node features and edge features output by the first layer graph network are then propagated to the next layer graph network for continuous updating, and the updating is performed layer by layer until the final updated node features and edge features are output by the last layer graph network, and L is an integer greater than or equal to 1;

[0019] Step S33: selecting the edge feature related to the node feature of the video to be queried from the final updated edge feature by using the Select operation to obtain the associated edge feature;

[0020] Step S34: calculating the graph network guidance feature of the video to be queried and the category support set video through the associated edge feature;

[0021] Step S35: fusing the graph network guidance feature of the video to be queried and the category support set video with the video time sequence feature of the video to be queried and the category support set video respectively to obtain the task-oriented feature of the video to be queried and the category support set video.

[0022] Preferably, in step S31:

[0023] The initialization of the node feature is specifically:

[0024]

[0025] represents the initial node feature, represents the video feature after the average pooling operation of the video time sequence feature, represents the union set of the category support set video and the video to be queried;

[0026] The initialization of the edge feature is specifically:

[0027]

[0028] represents the initial edge feature, y i , y j respectively represent the real category of the i-th and j-th video, represents the number of category support set videos.

[0029] In step S32, the update of the node feature is a process of weighted aggregation of the node features and the edge features of other layers, which is specifically:

[0030]

[0031]

[0032] represents the i-th node feature of the l-th layer, represents the j-th node feature of the l-1-th layer, is a multi-layer perceptron for converting features; Cat represents the Concat operation between matrices; and respectively represent the edge feature between the i-th node and the j-th node or the h-th node of the l-1-th layer;

[0033] The updating of the edge feature is performed after the updating of the node feature, and the sum of all edge feature values remains unchanged, specifically:

[0034]

[0035] represents the edge feature between the ith node and the jth node in the lth layer, is a function for calculating the similarity between two node features; and respectively represent the ith, jth and hth node features in the lth layer;

[0036] In step S33, the associated edge feature is specifically:

[0037]

[0038] M siam represents the associated edge feature, represents the number of videos to be queried, represents the number of videos in the category support set, represents the edge feature of the ith and jth nodes in the Lth layer, and Select means selecting the edge feature related to each video to be queried from all edge features;

[0039] In step S34, the associated edge feature calculation formula includes:

[0040]

[0041] wherein Us represents an unsqueeze operation, R represents a repeat operation, f FFN is a feedforward network, f emb is a multilayer perceptron, represents matrix cross multiplication; and respectively represent the node features of the video to be queried and the video in the category support set; and respectively represent the graph network guided features of the video to be queried and the video in the category support set; Cat represents the Concat operation between matrices; and T represents the number of frames of the video.

[0042] Preferably, the fusion formula in step 35 is specifically:

[0043]

[0044] f fuse is a multilayer perceptron, and respectively represent the video temporal features of the query video and the category support set video, and respectively represent the task-oriented features of the query video and the category support set video, Cat represents the Concat operation between matrices, and Us represents the unsqueeze operation.

[0045] Preferably, step S2 comprises the following steps:

[0046] Step S21: Let F represent the video features, and reshape the video features F into F seq1 and F seq2 , is a real set, where N represents the number of videos, T is the number of frames of the video, H and W are the length and width of the video feature map respectively, and C represents the dimension of the video feature map;

[0047] Step S22: input F seq1 into the temporal multi-layer perception to obtain the implicit temporal feature H T , insert the implicit temporal feature H T into F seq1 , so that F seq1 contains semantic information, and obtain the spatial block relationship modeling feature F tp after the learnable spatial attention operation, and obtain the temporal spatial block relationship modeling feature F tp ′ after the spatial attention operation.

[0048] Step S23: first perform the learnable channel shift operation on F seq2 , and then perform the spatial attention operation to obtain the temporal channel relationship modeling feature F tc .

[0049] Step S24: perform weight summation on the temporal spatial block relationship modeling feature F tp and the temporal channel relationship modeling feature F tc to obtain the video temporal feature Specifically:

[0050]

[0051] where β ∈ [0, 1] is a hyperparameter.

[0052] Preferably, in step S22, the implicit temporal feature H T , specifically:

[0053] H T = relu(W t1 F seq1 )W t2 +F seq1

[0054] wherein W t1 and are learnable weights for the interaction of the temporal information of different video frames; relu is a ReLU function; the temporal information is modeled by W t1 and W t2 to obtain the temporal semantic information of the video features;

[0055] temporal-spatial block relationship modeling feature F tp , specifically:

[0056]

[0057] wherein n is a spatial block index, and gap is a positive integer for controlling the frequency of spatial block shifting;

[0058] In step S23, the learnable channel shifting operation is specifically:

[0059]

[0060] K c,i represents the temporal convolution kernel weight of the cth channel, i represents the relative position relative to the current frame, i∈{-1,0,1}, is the input feature of the cth channel, is the output feature of the cth channel; After the spatial attention operation, the temporal-channel relationship modeling feature F tc is obtained; wherein t and c represent the time dimension and the channel dimension of the video feature mapping, respectively;

[0061] Preferably, the video feature F and the spatial block relationship modeling feature F tp in step S21 and step S22 are reshaped, and then subjected to spatial attention operation and weighted summation to obtain the temporal-spatial block relationship modeling feature F tp ', specifically:

[0062]

[0063] SA spa is the spatial attention operation, is reshaped from the video feature F, is reshaped from the spatial block relationship modeling feature F tp , and γ∈[0,1] is a hyperparameter;

[0064] Preferably, step S4 comprises the following steps:

[0065] Step S41: using and respectively represent the task-oriented features of the query video and the category support set videos, the task-oriented features of the query video and the task-oriented features of the category support set videos performing frame-level Hausdorff matching measurement operation to obtain frame-level matching measurement

[0066] Step S42: performing tuple-level Hausdorff matching measurement operation to obtain tuple-level matching measurement and the task-oriented features of the category support set videos performing tuple-level Hausdorff matching measurement operation to obtain tuple-level matching measurement

[0067] Step S43: calculating hybrid matching measurement by frame-level matching measurement and tuple-level matching measurement Specifically

[0068]

[0069] wherein a∈[0, 1] is a hyperparameter;

[0070] Step S44: calculating category prediction value by matching measurement

[0071] Preferably, in step S41, the frame-level Hausdorff matching measurement operation is specifically:

[0072]

[0073] and respectively represent the video features of the mth category support set video in the kth category and the video features of the pth query video; wherein represents the i-th frame feature of p,j the j-th frame feature of p T represents the number of frames of the video;

[0074] In step S42, the tuple-level Hausdorff matching measurement operation is specifically:

[0075]

[0076] and ​​respectively represent the i-th frame image feature of the m-th category support set video in the k-th category and the j-th frame image feature of the p-th video to be queried; i1, i2 represent the indexes of the i1, i2-th frames of the category support set video respectively, j1, j2 represent the indexes of the j1, j2-th frames of the video to be queried respectively; PE is position encoding, and T represents the number of frames of the video; is the number of tuples.

[0077] Preferably, the feature extraction network in step S1, the feature enhancement network in step S2, the graph network in step S3 and the hybrid matching network in step S4 are pre-trained, and the pre-training step comprises:

[0078] Step S61: obtaining a video to be queried and a category support set video, using a server to generate training videos with a length of T from all the obtained videos respectively, and inputting all the training videos into a feature extraction network for feature extraction to obtain video features of the video to be queried and the category support set video, wherein T≥2; the video to be queried is represented as Q={q1, q2, …, q T}, and the category support set video is represented as The category support set video comprises N categories; N is a positive integer;

[0079] Step S62: inputting all the video features in step S61 into a feature enhancement network respectively for remodeling and enhancement to obtain video time sequence features; the video time sequence features comprise video time sequence features of the video to be queried and the category support set video;

[0080] Step S63: performing average pooling operation on all the video time sequence features in step S62 to form corresponding node features, and using the node features to construct edge features, the edge features being used to represent the category correlation between any two node features, the node features and the edge features being input into a graph network for feature propagation and updating, and task-oriented features of the video to be queried and the category support set video being calculated according to the output results of the graph network after propagation and updating, and a loss being obtained through graph measurement

[0081] Step S64: performing category matching on the task-oriented features of the video to be queried and the category support set video through a hybrid matching network to obtain a category prediction value and a loss

[0082] Step S65: loss function of the action recognition method is the number of tuples. and the values of and, and is minimized through a standard cross-entropy loss;

[0083] Step S66: continuously optimizing the loss function The network parameters are then optimized until the network converges, resulting in a pre-trained feature extraction network, a pre-trained feature enhancement network, a pre-trained graph network, and a pre-trained hybrid matching network.

[0084] Compared with the prior art, the beneficial effects of the present invention are:

[0085] This invention provides an action recognition method based on graph neural networks. Guided by the graph network, it explicitly optimizes the correlation between intra-class and inter-class features between the query video and the videos in the category support set. It exhibits remarkable performance in recognizing similar category actions without any dataset or task type bias.

[0086] This invention proposes a feature enhancement network. After video features are subjected to learnable spatial block shifting and learnable channel shifting operations, spatial attention operations are performed on each to obtain temporal spatial block modeling features and temporal channel block modeling features. The two are then weighted and summed to obtain the video temporal features. The video temporal features obtained by this method have strong representation capabilities and help lay a solid foundation for the subsequent category matching process between the query video and the category support set videos.

[0087] This invention proposes a hybrid class prototype matching strategy based on a hybrid matching network. By fusing frame-level and tuple-level class prototype matching results, it can effectively handle video tasks with different styles.

[0088] The method of this invention can accurately and quickly identify target actions in tasks with a small number of labeled samples. In 5-way 1-shot and 5-shot tasks, the recognition accuracy reaches 74.9% and 87.4% on the Kinetics test set, respectively; 54.5% and 69.2% on the SSv2 test set, respectively; 61.2% and 76.9% on the HMDB51 test set, respectively; and 85.2% and 96.3% on the UCF101 test set, respectively. Attached Figure Description

[0089] Figure 1 A schematic diagram of the feature enhancement network constructed in this invention;

[0090] Figure 2 This is a schematic diagram of the graph network constructed in this invention;

[0091] Figure 3 This is a schematic diagram of the algorithm framework of an action recognition method based on graph neural networks according to the present invention;

[0092] Figure 4 This is a schematic diagram of the final output of the graph network edge features of the present invention.

[0093] Figure 5The attention visualization result schematic diagram of the target action recognition in the reasoning phase of the application. DETAILED DESCRIPTION

[0094] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work belong to the protection scope of the application.

[0095] The application provides a small sample action recognition method based on a graph neural network, comprising the following steps:

[0096] Step S1: obtaining a to-be-queried video and a category support set video, and inputting the to-be-queried video and the category support set video into a pre-trained feature extraction network respectively to obtain to-be-queried video features and category support set video features;

[0097] Step S2: inputting all the video features in step S1 into a pre-trained feature enhancement network respectively to reshape and enhance the video features to obtain video time sequence features; the video time sequence features comprise to-be-queried video time sequence features and category support set video time sequence features;

[0098] Step S3: performing average pooling operation on all the video time sequence features in step S2 to form corresponding node features, and constructing edge features by using the node features, wherein the edge features are used to represent the category correlation between any two node features; the node features and the edge features are input into a pre-trained graph network to perform feature propagation and update; and task-oriented features of the to-be-queried video and the category support set video are calculated according to the output results of the graph network after propagation and update;

[0099] Step S4: performing category matching on the task-oriented features of the to-be-queried video and the category support set video through a pre-trained hybrid matching network to obtain a category prediction value of the to-be-queried video;

[0100] Step S5: identifying the action of an actor in the to-be-queried video through the category prediction value.

[0101] The application provides an action recognition method based on a graph neural network, which can explicitly optimize the correlation between the features within and between the to-be-queried video and the category support set video under the guidance of the graph neural network. Without any dataset and task type preference, the method shows amazing performance in identifying similar category actions.

[0102] In the embodiment, step S2 comprises the following steps:

[0103] Step S21: using F to represent the video features, and reshaping the video features F into Fseq1 and F seq2 , Let N be a set of real numbers, where N represents the number of videos, T is the number of frames in the video, H and W are the width and height of the video feature map, respectively, and C represents the dimension of the video feature map.

[0104] Step S22: F seq1 The input is fed into a temporal multilayer perceptron to obtain implicit temporal features H. T Implicit temporal features H T It possesses rich spatiotemporal information in video, and incorporates implicit temporal features H T Insert into F seq1 In the middle, make F seq1 Includes semantic information, specifically, single-frame video features F seq1 It contains semantic information of all corresponding video frames, and obtains spatial block relationship modeling features F through learnable spatial block shifting operations. tp After performing spatial attention operations, the temporal spatial block relationship modeling feature F is obtained. tp ′;

[0105] Implicit temporal features H T Specifically:

[0106] H T =relu(W t1 F seq1 W t2 +F seq1

[0107] Among them W t1 and These are learnable weights used for the interaction of temporal information between different video frames; ReLU is the ReLU function; through W... t1 and W t2 Modeling temporal information yields temporal semantic information of video features;

[0108] Temporal spatial block relationship modeling feature F tp Specifically:

[0109]

[0110] Where n is the spatial block index, and gap is a positive integer used to control the frequency of spatial block shifting;

[0111] Step S23: F seq2 First, a learnable channel shift operation is performed, followed by a spatial attention operation, to obtain the temporal channel relationship modeling feature F. tc The learnable channel shifting operation is a 1D channel temporal convolution, used to learn an independent convolution kernel for each channel.

[0112] The learnable channel shifting operations are as follows:

[0113]

[0114] K c,i Let represent the temporal convolution kernel weight of the c-th channel, and let i represent the relative position with respect to the current frame, i∈{-1,0,1}. It is the input feature of the c-th channel. It is the output feature of the c-th channel; The temporal channel relationship modeling feature F is obtained after spatial attention operation. tc Where t and c represent the time dimension and channel dimension of the video feature mapping, respectively.

[0115] Step S24: Model the temporal spatial block relationship features F tp Modeling features F related to time channels tc Video temporal features are obtained by summing the weights. Specifically:

[0116]

[0117] Where β∈[0,1] is a hyperparameter.

[0118] This invention proposes a feature enhancement network. Video features are subjected to learnable spatial block shifting and learnable channel shifting operations, respectively, and then each is further subjected to spatial attention operations to obtain temporal spatial block modeling features and temporal channel block modeling features. These two features are then weighted and summed to obtain the video temporal features. The video temporal features obtained by this method have strong representation capabilities, which helps to lay a solid foundation for the subsequent category matching process between the query video and the category support set videos. This invention constructs a learnable complementary temporal modeling unit based on the feature enhancement network, such as... Figure 1 As shown, it specifically includes a temporal spatial block relationship modeling module and a temporal channel relationship modeling module, which are used to perform dense temporal modeling in the spatial and channel domains. The temporal spatial block relationship modeling module is used to perform learnable spatial block shifting operations on video features, and performs well in terms of the temporal information of aggregated image spatial blocks. The temporal channel relationship modeling learns the temporal offset of the channel by performing learnable channel shifting operations on video features. Therefore, the feature enhancement network can enhance video features in a dense and learnable way to obtain sufficient temporal relationships in the spatial and channel dimensions.

[0119] Due to the above-mentioned temporal spatial block relationship modeling characteristics F tpThis invention only sparsely collects temporal information from different video frames within a single frame, sacrificing the original spatial information within each frame. To alleviate this problem, in this embodiment, the invention employs a weighted summation of spatial information and spatiotemporal attention results. Specifically, the video feature F and the spatial block relationship modeling feature F from steps S21 and S22 are weighted summations. tp After reshaping, spatial attention operations and weighted summation are performed to obtain the temporal spatial block relationship modeling feature F. tp ′, specifically:

[0120]

[0121] SA spa For spatial attention operations, It is reconstructed from video feature F. Feature F modeled by spatial block relations tp Reconstructed from scratch, γ∈[0,1] is a hyperparameter;

[0122] Step S3 includes the following steps:

[0123] Step S31: Perform average pooling on all video temporal features from step S2 to form corresponding node features, denoted by V; and construct edge features using the node features, denoted by A; such as Figure 2 As shown, this invention constructs a class prototype building unit guided by a graph neural network based on a graph network, and uses the graph network to guide the learning of task-oriented features in the class prototype building process, explicitly optimizing the intra-class and inter-class correlations of video features. The specific implementation process is as follows:

[0124] Definition diagram Used to build category support set videos and search for videos The relationship between them.

[0125] Node features

[0126] v i Represents the feature of the i-th node. The total number of node features representing the categories supporting the video set and the video to be queried within the representative graph network is used to initialize the node features; the specific initialization of node features is as follows:

[0127]

[0128] Indicates the initial node characteristics. The video features are those resulting from average pooling after performing a process on the temporal features of the video. The representative category supports the union of the set video and the video to be queried;

[0129] edge feature

[0130] denotes an edge feature between the ith node feature and the jth node feature, denotes a union set of the category support set video and the to-be-queried video, and the edge feature is initialized using the real category label; the initialization of the edge feature is specifically:

[0131]

[0132] denotes an initial edge feature, y i , y j denote real categories of the ith and jth videos, respectively, denotes a number of category support set videos.

[0133] Step S32: The node features and the edge features are respectively input into the pre-trained L-layer graph network for feature propagation and updating, and the propagation and updating steps specifically include: all initial node features and initial edge features are propagated to the first layer graph network, the first layer graph network updated node features and edge features are calculated and output, the first layer graph network updated node features and edge features are then propagated to the next layer graph network for continuous updating, and the updating is performed layer by layer until the last layer graph network outputs the final updated node features and edge features, and L is an integer greater than or equal to 1;

[0134] The updating of the node features is a process of weighted aggregation of the node features and the edge features of other layers, and is specifically:

[0135]

[0136] denotes an ith node feature of an lth layer, denotes a jth node feature of an (l-1)th layer, is a multi-layer perception, used for converting features; Cat represents a Concat operation between matrices; and denote edge features between an ith node and a jth node or an hth node of an (l-1)th layer, respectively; wherein b e {1, 2}, the edge feature includes two matrices of node correlation and dissimilarity, b is 1 when the correlation matrix is represented, and b is 2 when the dissimilarity matrix is represented;

[0137] The updating of the edge features is performed after the updating of the node features, and is performed through the similarity or dissimilarity between the node features, and the sum of all edge feature values remains unchanged, and is specifically:

[0138]

[0139] represents the edge feature between the i-th node and the j-th node in the l-th layer, is a function for calculating the similarity between two node features; and represent the i-th, j-th and h-th node features in the l-th layer, respectively;

[0140] Step 33: Select the edge features related to the node features of the video to be queried from the final updated edge features using the Select operation to obtain the associated edge features; the Select operation refers to selecting the edge features related to the video features to be queried from all the output edge features, further forming a total of new edge features M siam ; the associated edge features are specifically:

[0141]

[0142] M siam represents the associated edge features, represents the number of query set videos, represents the number of support set videos. represents the edge features of the i-th and j-th nodes in the L-th layer, and since we only need the relevance features, we take i.e. b = 1. The Select means selecting the edge features related to each query video from all the edge features, thereby forming a total of new edge features.

[0143] Step 34: Calculate the graph network guided features of the video to be queried and the category support set videos through the associated edge features; the associated edge features calculation formula includes:

[0144]

[0145] wherein Us represents the unsqueeze operation, R represents the repeat operation, f FFN is a feedforward network, f emb is a multilayer perceptron, represents matrix cross multiplication; and represent the node features of the video to be queried and the category support set videos, respectively; and represent the graph network guided features of the video to be queried and the category support set videos, respectively; Cat represents the Concat operation between matrices; T represents the number of frames of the video. The task-oriented feature F task is obtained by fusing the enhanced temporal feature and the feature F graph guided by the graph network.To preserve the temporal characteristics of the features, each video feature to be queried has its special set of support video features of the category, and the intra-class and inter-class correlations of the video features are explicitly optimized through the guidance of the graph neural network.

[0146] Step S35: Fuse the graph network guided features of the video to be queried and the category support set videos with the video temporal features of the video to be queried and the category support set videos to obtain task-oriented features of the video to be queried and the category support set videos.

[0147] The fusion formula is specifically:

[0148]

[0149] Wherein f fuse is a multi-layer perceptron, and represent the video temporal features of the video to be queried and the category support set videos respectively, and represent the task-oriented features of the video to be queried and the category support set videos respectively, Cat represents the Concat operation between matrices, and Us represents the unsqueeze operation.

[0150] Step S4 includes the following steps:

[0151] Step S41: Use and to represent the task-oriented features of the video to be queried and the category support set videos respectively, and perform frame-level Hausdorff matching measurement operation on the task-oriented features of the video to be queried and the task-oriented features of the category support set videos to obtain frame-level matching measurement

[0152] For frame-level matching, bidirectional mean Hausdorff measurement is used for operation, and the frame-level Hausdorff matching measurement operation is specifically:

[0153]

[0154] and represent the video features of the mth category support set video in the kth category and the video features of the pth video to be queried respectively; wherein represents the i-th frame feature of , q p,j represents the j-th frame feature of q p , and T represents the number of frames of the video;

[0155] Step S42: Perform frame-level Hausdorff matching measurement operation on the task-oriented features of the video to be queried Task-oriented features of video support set and category The tuple-level Hausdorff matching measurement operation is performed to obtain a tuple-level matching measurement In step S42, for the prototype matching at the tuple level, two frames are combined into a tuple, and all combinations are traversed to obtain the tuple-level matching measurement of T frames The tuple-level Hausdorff matching measurement operation is specifically:

[0156]

[0157] and respectively represent the i-th frame image feature of the m-th category support set video in the k-th category and the j-th frame image feature of the p-th video to be queried; i1 and i2 respectively represent the indexes of the i1-th and i2-th frames of the category support set video, j1 and j2 respectively represent the indexes of the j1-th and j2-th frames of the video to be queried; PE is position encoding, and T represents the number of frames of the video; is the number of tuples.

[0158] Step S43: calculating a hybrid matching measurement by frame-level matching measurement and tuple-level matching measurement Specifically,

[0159]

[0160] wherein α is an hyperparameter in [0, 1];

[0161] Step S44: calculating a category prediction value by the matching measurement

[0162] The application proposes a hybrid category prototype matching strategy based on a hybrid matching network, and the fusion of frame-level and tuple-level category prototype matching results can effectively process video tasks with different styles.

[0163] The feature extraction network in step S1, the feature enhancement network in step S2, the graph network in step S3 and the hybrid matching network in step S4 are pre-trained, and the pre-training step includes:

[0164] Step S61: obtaining a video to be queried and a category support set video, using a server to generate training videos with a length of T from all obtained videos respectively, and inputting all training videos into a feature extraction network for feature extraction to obtain video features to be queried and category support set video features, wherein T is greater than or equal to 2; the video to be queried is represented as Q={q1, q2, …, q T}, and the category support set video is represented as ​​The category support set of videos includes N categories; N is a positive integer; in this embodiment, T=8, N=5. During training, some basic methods can be used for data augmentation, such as random horizontal flipping, cropping, and color jittering.

[0165] Step S62: Input all the video features from step S61 into the feature enhancement network for reshaping and enhancement to obtain video temporal features; the video temporal features include the temporal features of the video to be queried and the video temporal features of the category support set;

[0166] Step S63: Perform average pooling on all video temporal features from Step S62 to form corresponding node features, and construct edge features using the node features. The edge features are used to represent the class correlation between any two node features. The node features and edge features are respectively input into the graph network for feature propagation and updating. Based on the output of the graph network after propagation and updating, calculate the task-oriented features of the query video and the class support set videos, and obtain the loss through graph metric. like Figure 3 As shown;

[0167] Step S64: Perform category matching on the task-oriented features of the query video and the category support set videos through a hybrid matching network to obtain the category prediction value and loss.

[0168] Step S65: Loss function of the action recognition method for and The sum of the values ​​will Minimize using a standard cross-entropy loss;

[0169] Step S66: Continuously optimize the loss function The network parameters are then optimized until the network converges, resulting in a pre-trained feature extraction network, a pre-trained feature enhancement network, a pre-trained graph network, and a pre-trained hybrid matching network.

[0170] The few-shot action recognition method of this embodiment is used for the recognition task. A test video clip generation unit is executed on the server, uniformly sampling eight frames of a video as input, and the input image is center-cropped during testing. ResNet-50 is used as the feature extractor, and ImageNet pre-trained weights are employed. For the complementary temporal modeling unit, W... t1 W t2 It consists of two single-layer multilayer perceptrons, with the gap set to 2. For the prototype building unit guided by a graph neural network, a single-layer graph neural network (GNN) is applied to obtain task-oriented features. The few-shot action recognition method of this invention assigns the query video in the test video to the closest matching video. of the class, and the action of the actor in the video to be queried is identified through the class prediction value.

[0171] The present application visualizes the final output of the edge feature of the graph network in the recognition task process. Figure 4 As shown in the figure, the graph network is also called a graph neural network, and the value of the edge feature can be regarded as a similarity score between two video features. From the visualization result, it can be seen that the graph neural network as a guide can well optimize the intra-class and inter-class correlation of the video features, and the updated edge feature is very close to the similarity matrix corresponding to the true class label. At the same time, the intermediate output recognition result of the graph neural network obtained by using the edge feature in the accuracy calculation area can also achieve high accuracy.

[0172] The present application visualizes the attention visualization result graph of target action recognition in the recognition task. Figure 5 As shown in the figure, the attention visualization result of the target action recognition of the present application is compared with the attention map generated without using the learnable complementary temporal modeling module (in the middle) and the attention map generated using the learnable complementary temporal modeling module (on the right) compared with the left original RGB image. The attention map generated without using the learnable complementary temporal modeling module contains many irrelevant or scattered attention areas. For example, many frames in the "horse riding" action are obviously paying attention to the background and irrelevant objects, thereby diverting the attention to the action. In contrast, the attention map generated using the learnable complementary temporal modeling module is strongly related to the subject action. Specifically, the frames in the "skiing" action are paying attention to the skier, and the frames in the "tennis swing" action are paying attention to the tennis player. These observations provide empirical evidence that the learnable complementary temporal modeling module enhances the effect of spatio-temporal representation.

[0173] In the recognition task of the present application, the present application enables the system to accurately and quickly recognize the target action in a small number of labeled sample tasks. In the 5-way 1-shot and 5-shot tasks (1-shot task refers to 1 sample per class in the support set, and 5-shot task refers to 5 samples per class in the support set), the recognition accuracy on the Kinetics test set is 74.9% and 87.4% respectively, the recognition accuracy on the SSv2 test set is 54.5% and 69.2% respectively, the recognition accuracy on the HMDB51 test set is 61.2% and 76.9% respectively, and the recognition accuracy on the UCF101 test set is 85.2% and 96.3% respectively.

[0174] ​While the application has been described with reference to particular embodiments thereof, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present application. It will be apparent to those skilled in the art that numerous modifications can be made within the scope of the present application as defined by the appended claims. It is intended that all such modification fall within the spirit and scope of the present application. It will be understood that the features described in connection with separate embodiments can be used in other embodiments.

Claims

1. An action recognition method based on graph neural networks, comprising: Step S1: Obtain the video to be queried and the category support set videos, and input them into the pre-trained feature extraction network to extract features of the video to be queried and the category support set videos respectively; Step S2: Input all the video features from step S1 into the pre-trained feature enhancement network for reshaping and enhancement to obtain video temporal features; the video temporal features include the temporal features of the video to be queried and the video temporal features of the category support set; Step S3: Perform average pooling on all video temporal features from Step S2 to form corresponding node features, and use the node features to construct edge features. The edge features are used to represent the class correlation between any two node features. The node features and edge features are respectively input into a pre-trained graph network for feature propagation and updating. Based on the output of the graph network after propagation and updating, calculate the task-oriented features of the query video and the category support set video respectively. Step S4: Perform category matching on the task-oriented features of the video to be queried and the category support set videos through a pre-trained hybrid matching network to obtain the category prediction value of the video to be queried; Step S5: Identify the actions of the actors in the query video using the category prediction values; Step S3 includes the following steps: Step S31: Perform average pooling on all the video temporal features from step S2 to form corresponding node features, denoted by V; and use the node features to construct edge features, denoted by A; Node features v i Represents the feature of the i-th node. The total number of node features supporting the categories of the video set and the video to be queried within the representative graph network is initialized. Edge features a ij This represents the edge feature between the feature of node i and the feature of node j. The total number of node features supporting the categories within the graph network and the video to be queried is represented by the actual category labels used to initialize the edge features. Step S32: The node features and edge features are respectively input into the pre-trained L-layer graph network for feature propagation and updating. The propagation and updating steps specifically include: all initial node features and initial edge features are propagated to the first layer graph network, and the updated node features and edge features of the first layer graph network are calculated and output. The updated node features and edge features of the first layer graph network are then propagated to the next layer graph network for further updating, and so on until the last layer graph network outputs the final updated node features and edge features. L is an integer greater than or equal to 1. Step S33: Use the Select operation to select edge features related to the node features of the video to be queried from the finally updated edge features to obtain the associated edge features; Step S34: Calculate the graph network guided features of the query video and the category support set videos using the associated edge features; Step S35: Fuse the graph network-guided features of the video to be queried and the category support set videos with the video temporal features of the video to be queried and the category support set videos respectively to obtain the task-oriented features of the video to be queried and the category support set videos.

2. The action recognition method based on graph neural networks according to claim 1, characterized in that, In step S31: The initialization of node features is specifically as follows: Indicates the initial node characteristics. The video features are those resulting from average pooling after performing a process on the temporal features of the video. The representative category supports the union of the set video and the video to be queried; The specific steps for initializing edge features are as follows: Represents the initial edge features, y i y j Let i and j represent the real categories of the i-th and j-th videos, respectively. The number of videos that a category can support; In step S32, the node feature update is a weighted aggregation process using node features and edge features from other layers, specifically as follows: This represents the feature of the i-th node in the l-th layer. This represents the feature of the j-th node in the (l-1)-th layer. It is a multilayer perceptron used for feature transformation; Cat represents the concat operation between matrices; and These represent the edge features between the i-th node and the j-th node or the h-th node in the (l-1)-th layer, respectively; Edge features are updated after node features are updated, based on the similarity or dissimilarity between node features, while the sum of all edge feature values ​​remains unchanged. Specifically: This represents the edge feature between the i-th node and the j-th node in the l-th layer. It is a function used to calculate the similarity between features of two nodes; and Let i, j, and h represent the features of the i-th, j-th, and h-th nodes in the l-th layer, respectively. In step S33, the features of the associated edges are specifically as follows: M siam Represents the characteristics of associated edges. This represents the number of videos to be queried. The number of videos that a category can support. The edge features represent the i-th and j-th nodes in the L-th layer; Step S34, the formula for calculating the features of associated edges includes: Where Us represents the unsqueeze operation, R represents the repeat operation, and f FFN It is a feedforward network, f emb It is a multilayer perceptron. Represents the matrix cross product; and These represent the node features of the video to be queried and the video in the category support set, respectively; and These represent the graph network guided features of the video to be queried and the video of the category support set, respectively; Cat represents the Concat operation between matrices; T represents the number of frames in the video.

3. The action recognition method based on graph neural networks according to claim 1, characterized in that, The fusion formula in step S35 is as follows: f fuse It is a multilayer perceptron. and These represent the temporal features of the video to be queried and the video in the category support set, respectively. and These represent the task-oriented features of the video to be queried and the category support set video, respectively. Cat represents the Concat operation between matrices, and Us represents the unsqueeze operation.

4. The action recognition method based on graph neural networks according to claim 1, characterized in that, Step S2 includes the following steps: Step S21: Let F represent the video features, and reshape the video features F into F1 respectively. seq1 and F seq2 , Let N be a set of real numbers, where N represents the number of videos, T is the number of frames in the video, H and W are the width and height of the video feature map, respectively, and C represents the dimension of the video feature map. Step S22: F seq1 The input is fed into a temporal multilayer perceptron to obtain implicit temporal features H. T The implicit time feature H T Insert into F seq1 In the middle, make F seq1 Including semantic information, and through learnable spatial block shifting operations, the spatial block relation modeling feature F is obtained. tp After performing spatial attention operations, the temporal spatial block relationship modeling feature F is obtained. tp ′; Step S23: F seq2 First, a learnable channel shift operation is performed, followed by a spatial attention operation, to obtain the temporal channel relationship modeling feature F. tc ; Step S24: Model the temporal spatial block relationship features F tp Modeling features F related to time channels tc Video temporal features are obtained by summing the weights. Specifically: Where β∈[0,1] is a hyperparameter.

5. The action recognition method based on graph neural networks according to claim 4, characterized in that, In step S22, the implicit time feature H T Specifically: H T =relu(W t1 F seq1 )IN t2 +F seq1 Among them W t1 and These are learnable weights used for the interaction of temporal information between different video frames; ReLU is the ReLU function; through W... t1 and W t2 Modeling temporal information yields temporal semantic information of video features; The temporal spatial block relationship modeling feature F tp Specifically: Where n is the spatial block index, and gap is a positive integer used to control the frequency of spatial block shifting; In step S23, the learnable channel shift operation is specifically as follows: K c,i Let represent the temporal convolution kernel weight of the c-th channel, and let i represent the relative position with respect to the current frame, i∈{-1,0,1}. It is the input feature of the c-th channel. It is the output feature of the c-th channel; The temporal channel relationship modeling feature F is obtained after spatial attention operation. tc .

6. The action recognition method based on graph neural networks according to claim 4, characterized in that, The video features F and spatial block relationship modeling features F from steps S21 and S22 are used. tp After reshaping, spatial attention operations and weighted summation are performed to obtain the temporal spatial block relationship modeling feature F. tp ′, specifically: SA spa For spatial attention operations, It is reconstructed from video feature F. Feature F modeled by spatial block relations tp Reconstructed from the original, γ∈[0,1] is a hyperparameter.

7. The action recognition method based on graph neural networks according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: Use and These represent the task-oriented features of the video to be queried and the category support set video, respectively. and category support for task-oriented features of video sets Perform a frame-level Hausdorff matching metric operation to obtain the frame-level matching metric. Step S42: Task-oriented features of the video to be queried and category support for task-oriented features of video sets Performing a tuple-level Hausdorff matching metric operation yields a tuple-level matching metric. Step S43: Measuring frame-level matching tuple-level matching metrics Calculate the mixed matching metric Specifically: Where α∈[0,1] is a hyperparameter; Step S44: By matching metrics The predicted values ​​for the categories are calculated.

8. The action recognition method based on graph neural networks according to claim 7, characterized in that, In step S41, the frame-level Hausdorff matching metric operation specifically involves: and Let represent the video features of the m-th category supported video in the k-th category and the video features of the p-th query video, respectively. in express The i-th frame feature, q p,j q p The j-th frame features, where T represents the number of frames in the video; In step S42, the tuple-level Hausdorff matching metric operation is specifically as follows: and Let represent the video features of the m-th category supported video in the k-th category and the video features of the p-th query video, respectively. in express The i-th frame feature, q p,j q p The features of the j-th frame; i1 and i2 represent the indices of the i1th and i2th frames of the video in the category support set, respectively; j1 and j2 represent the indices of the j1th and j2th frames of the video to be queried, respectively; PE stands for position encoding, and T represents the number of frames in the video; The number of tuples.

9. The action recognition method based on graph neural networks according to claim 1, characterized in that: The feature extraction network in step S1, the feature enhancement network in step S2, the graph network in step S3, and the hybrid matching network in step S4 are pre-trained. The pre-training steps include: Step S61: Obtain the query video and category support set videos. Use the server to generate training videos of length T from all the obtained videos. Input all training videos into a feature extraction network to extract features from the query video and category support set videos, where T≥2. The query video is represented as Q={q1,q2,…,q T }, the category supports video representation as The category supports video sets comprising N categories; N is a positive integer. Step S62: Input all the video features from step S61 into the feature enhancement network for reshaping and enhancement to obtain video temporal features; the video temporal features include the temporal features of the video to be queried and the video temporal features of the category support set; Step S63: Perform average pooling on all video temporal features from step S62 to form corresponding node features, and construct edge features using the node features. The edge features are used to represent the class correlation between any two node features. The node features and edge features are respectively input into a graph network for feature propagation and updating. Based on the output of the graph network after propagation and updating, calculate the task-oriented features of the query video and the class support set videos, and obtain the loss through graph metrics. Step S64: Perform category matching on the task-oriented features of the query video and the category support set videos through a hybrid matching network to obtain the category prediction value and loss. Step S65: Loss function of the action recognition method for and The sum of the values ​​will Minimize using a standard cross-entropy loss; Step S66: Continuously optimize the loss function The network parameters are then optimized until the network converges, resulting in a pre-trained feature extraction network, a pre-trained feature enhancement network, a pre-trained graph network, and a pre-trained hybrid matching network.

Images