A few-shot video action recognition method and system based on hierarchical task information

Abstract

The application provides a few-shot video action recognition method based on hierarchical task information, comprising obtaining a video dataset, constructing multiple tasks each having N+1 groups of videos and preprocessing; using a low-order feature extraction network, extracting low-order features of N+1 groups of videos in each task, and generating adaptive parameters of each task for a high-order feature extraction network according to the low-order features of N groups of support set videos in each task; based on the adaptive parameters, adjusting the high-order feature extraction network used by each task to extract high-order features of N+1 groups of videos; calculating the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task according to the extracted N+1 high-order features; according to the two similarities calculated for each task, a group of support set videos meeting predetermined conditions is correspondingly filtered out from N groups of support set videos of each task. The application is used to solve the problems of poor generalization performance and insufficient measurement efficiency in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video recognition, and particularly relates to a few-shot video action recognition method and system based on hierarchical task information. BACKGROUND

[0002] With the enhancement of big data and computing power, deep learning related technologies have been rapidly developed. In the field of deep learning, few-shot learning has been applied to many downstream tasks. For example, video saliency detection, video tracking and video target detection tasks. Of course, few-shot action recognition task is one of the downstream tasks, and the task aims to train a classifier with a few labeled video samples, and classify unknown actions according to the similarity with known actions.

[0003] However, the current few-shot action recognition method still has two problems: (1) under the condition that the sample related to the action recognition video is less, the deep learning model trained has poor generalization performance, and is difficult to be applied to new tasks; (2) different video contents have feature semantic correlation information, so that the deep learning model trained has weak ability to measure the semantic correlation between video features, resulting in insufficient measurement efficiency and accuracy.

[0004] Therefore, in order to solve the above two problems, it is necessary to provide a few-shot video action recognition method with high generalization performance and high measurement efficiency. SUMMARY

[0005] The technical problem to be solved by the embodiments of the present application is to provide a few-shot video action recognition method and system based on hierarchical task information, to solve the problems of poor generalization performance and insufficient measurement efficiency in the prior art few-shot video action recognition method.

[0006] In order to solve the above technical problems, the embodiments of the present application provide a few-shot video action recognition method based on hierarchical task information, which comprises the following steps:

[0007] S1, a video data set with X action categories is obtained, and based on the video data set, a plurality of tasks are constructed, and the video in each task is preprocessed; wherein each task has N groups of support set videos with different action categories and 1 group of query set videos with different content from the N groups of support set videos; X and N are positive integers, and X>N;

[0008] S2, using a low-order feature extraction network, extracting low-order features of N+1 groups of videos contained in each task to obtain N groups of high-order features corresponding to the support set videos and a single high-order feature corresponding to the query set video in each task, and obtaining adaptive parameters in the high-order feature extraction network of each task according to the N groups of high-order features corresponding to the support set videos in each task;

[0009] S3, based on the adaptive parameters in the high-order feature extraction network of each task, adjusting the three-dimensional convolution layer in the high-order feature extraction network of each task, and using the adjusted high-order feature extraction network corresponding to each task to extract the N+1 low-order features extracted by each task to obtain N groups of high-order features corresponding to the support set videos and a single high-order feature corresponding to the query set video in each task;

[0010] S4, according to the N groups of high-order features corresponding to the support set videos and the single high-order feature corresponding to the query set video in each task, calculating the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task;

[0011] S5, according to the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task, a group of support set videos meeting the predetermined condition is correspondingly screened out from the N groups of support set videos in each task.

[0012] Among them, the step S1 specifically includes:

[0013] Constructing a video data set, the video data set including action videos of X action categories;

[0014] Dividing the data set into M tasks, and each task containing N groups of videos of different action categories as support set videos, and randomly selecting 1 group of videos completely inconsistent with the content of the support set videos as the query set video in the N categories;

[0015] Preprocessing each group of support set videos and query set videos in each task.

[0016] Among them, the step of preprocessing each group of support set videos and query set videos includes video frame sampling and frame image processing after sampling, specifically:

[0017] Randomly selecting the initial position of the first frame image of each group of videos, and then continuously sampling 16 frames;

[0018] Crop each frame into an image block with a long and wide resolution of 224x224, and perform normalization to obtain N+1 groups of normalized videos, denoted as {S p1 , S p2 , ..., S pN , Qpi}; where S p1 ,S p2 ,...,S pN Q represents the support set videos of N different action categories after normalization; pi Let i represent the support set video of a group of action categories i after normalization; i∈N.

[0019] Specifically, step S2 includes:

[0020] The low-order feature extraction network is determined to consist of a three-dimensional convolutional layer, a normalization layer, and a max pooling layer.

[0021] In each task, N+1 sets of videos are processed through a single 3D convolutional layer in the low-order feature extraction network, which has 64 3D convolutional kernels and 64 output channels, to obtain F. conv and F conv After processing by the normalization layer in the low-order feature extraction network, F is obtained. batchnorm Furthermore, F batchnorm The input is processed by the max pooling layer in the low-order feature extraction network to obtain N low-order features F corresponding to the N support set videos. SL And a single low-order feature F corresponding to a set of query set videos. QL :

[0022]

[0023] F batchnorm =Batchnorm3(F conv );

[0024] in, It is a three-dimensional convolutional layer containing 64 3×3×3 three-dimensional convolutional kernels and 64 output channels. n ={S n1 ,S n2 ,...,S nN Q ni Batchnorm3() represents the normalization operation;

[0025] Solve for the N low-order features F corresponding to the N support set videos in each task. SL Their respective expected values ​​μ and standard deviations σ are expressed by the formula Info=G(μ,diag(σ)). 2), and further inputting the multivariate Gaussian output value Info of each task into a full connection layer for linear transformation to convert into three-dimensional convolution adaptive parameters P of each task for the high-order feature extraction network adaptive ; wherein P adaptive =f(Info); Info represents the multivariate Gaussian output value of the expectation μ and the standard deviation σ; diag() represents a diagonal matrix construction function.

[0026] The step S3 specifically comprises:

[0027] Based on the adaptive parameters P of each task for the high-order feature extraction network in the high-order feature extraction network of each task adaptive , the three-dimensional convolution layer in the high-order feature extraction network of each task is adjusted; wherein the three-dimensional convolution layer in the high-order feature extraction network of each task is a three-dimensional convolution with 64 three-dimensional convolution kernels of 3×3×3 and an output channel number of 64.

[0028] In each task, the N+1 low-order features are processed by the three-dimensional convolution with 64 three-dimensional convolution kernels of 3×3×3 and an output channel number of 64 in the adjusted high-order feature extraction network to obtain N high-order features F SH corresponding to the N groups of support set videos and a single high-order feature F QH corresponding to the query set video:

[0029]

[0030]

[0031] The step S4 specifically comprises:

[0032] The first step is to calculate the spatio-temporal self-attention similarity of each task, specifically:

[0033] In each task, according to the N high-order features F SH corresponding to the N groups of support set videos and the single high-order feature F QH corresponding to the query set video, and through the formula M ST =mean(F SH ×(F SH ) T ), the spatio-temporal self-attention vector M ST of each group of support set videos is obtained, and through the formula M QT =mean(F QH ×(F QH ) T ), the spatio-temporal self-attention vector MQT ; wherein, mean() represents the operation of taking the average value; T represents the transpose;

[0034] Determine the intra-task learner K inner () is composed of a two-dimensional convolution layer, a Relu nonlinear activation layer, a two-dimensional convolution layer and a Softmax logistic regression function;

[0035] The spatio-temporal self-attention vector M ST and the spatio-temporal self-attention vector M QT of the query set video in each task are input into the intra-task information learner K inner (), respectively, to obtain the optimized self-attention vector A ST of each group of support set videos in each task and the optimized self-attention vector A QT of the query set video in each task; ST = K inner (M ST ); A QT = K inner (M QT );

[0036] The enhanced spatio-temporal self-information features T ST of each group of support set videos in each task are obtained by the formula T SH = F ST × A SH + F ST , and the enhanced spatio-temporal self-information features T QT of the query set video in each task are obtained by the formula T QH = F QT × A QH + F QT ;

[0037] The cosine distance between the enhanced spatio-temporal self-information features T T of the query set video in each task and the enhanced spatio-temporal self-information features T ST of each group of support set videos in the same task is calculated by the formula Sim QT = Dcos(T QT , T ST ), to obtain the N spatio-temporal self-attention similarity Sim T of each task:

[0038] The second step is to calculate the relevant cross-attention measure similarity of each task, which is specifically:

[0039] In each task, according to the N high-order features F SH corresponding to the N groups of support set videos and the single high-order feature FQH , and the relevant cross-attention vector M SR of each group of support set videos is obtained by the formula M SH = mean(F QH ·(F T ) SR ), and the relevant cross-attention vector M QR of the query set video is obtained by the formula M QH = mean(F SH ·(F T ) QR ;

[0040] The relevant cross-attention vector M SR of each group of support set videos in each task and the relevant cross-attention vector M QR of the query set video are input into the intra-task information learner K inner () respectively, to obtain the optimized relevant cross-attention vector A SR of each group of support set videos of each task and the optimized relevant cross-attention vector A QR of the query set video of each task; wherein A SR = K inner (M SR ); A QR = K inner (M QR );

[0041] The reinforced spatio-temporal self-information feature R SR of each group of support set videos in each task is obtained by the formula R SH = F SR × A SH + F SR , and the reinforced spatio-temporal self-information feature R QR of the query set video in each task is obtained by the formula R QH = F QR × A QH + F QR ;

[0042] The cosine distance between the reinforced spatio-temporal self-information feature R R of the query set video in each task and the reinforced spatio-temporal self-information feature R SR of each group of support set videos in the same task is calculated by the formula Sim QR = Dcos(R QR , R SR ), to obtain N relevant cross-attention similarities Sim R of each task.

[0043] The step S5 specifically comprises:

[0044] In each task, according to the obtained N spatio-temporal self-attention similarities Sim T and N related cross-attention similarities Sim R , the spatio-temporal self-attention similarities Sim T and the cross-attention similarities Sim R of the same action category are determined, and the N similarities Sim corresponding to the query set video and the N groups of support set videos are obtained through the formula Sim = a * Sim T + (1-a) * Sim R .

[0045] The N similarities Sim corresponding to each task are sorted from large to small, and the support set video corresponding to the maximum similarity Sim is filtered out from the N groups of support set videos in each task.

[0046] The embodiment of the application also provides a few-shot video action recognition system based on hierarchical task information, comprising:

[0047] a video task construction unit configured to obtain a video dataset having X action categories, and construct a plurality of tasks based on the video dataset, and pre-process the videos in each task; wherein each task has N groups of support set videos with different action categories and 1 group of query set videos with different contents from the N groups of support set videos; X and N are positive integers, and X > N;

[0048] a low-order feature extraction unit configured to extract low-order features of N+1 groups of videos in each task using a low-order feature extraction network, to obtain N low-order features corresponding to the N groups of support set videos and a single low-order feature corresponding to the 1 group of query set videos in each task, and obtain adaptive parameters for a high-order feature extraction network in each task according to the N low-order features corresponding to the N groups of support set videos in each task;

[0049] a high-order feature extraction unit configured to adjust a three-dimensional convolution layer in the high-order feature extraction network used in each task based on the adaptive parameters for the high-order feature extraction network in each task, and extract high-order features of the N+1 low-order features extracted in each task using the adjusted high-order feature extraction network corresponding to each task, to obtain N high-order features corresponding to the N groups of support set videos and a single high-order feature corresponding to the 1 group of query set videos in each task;

[0050] a similarity calculation unit configured to calculate spatio-temporal self-attention similarities and related cross-attention similarity measures of each task according to the N high-order features corresponding to the N groups of support set videos and the single high-order feature corresponding to the 1 group of query set videos in each task.

[0051] The video recognition unit is configured to filter out a set of support set videos meeting a predetermined condition from the N groups of support set videos of each task according to the spatio-temporal self-attention similarity and the related cross-attention similarity.

[0052] The embodiment of the present application has the following beneficial effects:

[0053] 1. The present application solves the problem of few samples by meta-learning. The purpose of meta-learning is to enable the deep neural network to "learn to learn" by designing the neural network and pre-training with a small number of samples, so that the neural network model can quickly generalize to other scenarios, and the problem of poor generalization performance and insufficient measurement efficiency in the existing few-sample video action recognition method can be solved.

[0054] 2. The present application can make the deep learning model have high generalization performance and high measurement performance with a small number of samples, and dynamically generate adaptive parameters of the feature extraction network according to the input task to construct a task-related feature space, thereby improving the model generalization.

[0055] 3. The present application combines the intra-task information learner with the self-attention mechanism and the cross-attention mechanism to mine intra-task feature key self-information and mutual information, strengthen the feature spatio-temporal key information and the correlation between features, and thereby improve the measurement efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings obtained according to these drawings without creative labor are still within the scope of the present application.

[0057] Figure 1 A flowchart of a few-sample video action recognition method based on hierarchical task information provided by the embodiment of the present application;

[0058] Figure 2 A schematic diagram of a frame in the pre-processing process of part of the video in a few-sample video action recognition method based on hierarchical task information provided by the embodiment of the present application;

[0059] Figure 3 A logic working framework diagram of a few-sample video action recognition method based on hierarchical task information provided by the embodiment of the present application;

[0060] Figure 4A hierarchical task information based few-shot video action recognition structure diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings.

[0062] As Figure 1 shown, a hierarchical task information based few-shot video action recognition method provided by an embodiment of the present application includes the following steps:

[0063] Step S1, a video dataset with X action categories is acquired, and based on the video dataset, a plurality of tasks are constructed, and the video in each task is preprocessed; wherein each task has N sets of support set videos with different action categories and 1 set of query set videos inconsistent with the content of the N sets of support set videos; X and N are positive integers, and X>N;

[0064] The specific process is as follows: first, a video dataset is constructed, which includes action videos of X action categories; wherein actions of the same category are classified into one category.

[0065] Secondly, the dataset is divided into M tasks, and each task includes N sets of videos with different action categories as support set videos, and 1 set of videos inconsistent with the content of the support set videos as query set videos are randomly selected from the N categories. Therefore, each task includes N+1 sets of videos. A task is composed of {S N ,Q i} video set, i=1,2,...,N. Wherein, S N represents the support set video composed of N sets of different action category videos, Q i represents the query set video of action category i video.

[0066] Finally, each set of support set videos and query set videos in each task is preprocessed, which specifically includes video frame sampling and frame image processing after sampling. At this time, the process of processing is as follows: the initial position of the first frame image of each set of videos is randomly selected, and then 16 frames are continuously sampled; each frame is cropped into an image block with a long and wide resolution of 224x224, and normalized to obtain N+1 sets of videos after normalization, which are denoted as {S p1 ,S p2 ,...,S pN ,Q pi}; wherein, S p1 ,S p2 ,...,S pNQ represents the support set videos of N different action categories after normalization processing; Q pi Q represents the support set videos of one action category i after normalization processing; i∈N. Wherein, Figure 2 is a schematic diagram of one frame in the pre-processing process of part of the video.

[0067] Step S2, using a low-order feature extraction network, extracting the low-order features of N+1 groups of videos contained in each task to obtain N low-order features corresponding to N groups of support set videos and a single low-order feature corresponding to one group of query set videos in each task, and obtaining adaptive parameters in the high-order feature extraction network according to the N low-order features corresponding to the N groups of support set videos in each task;

[0068] The specific process is as follows: first, it is determined that the low-order feature extraction network is composed of one layer of three-dimensional convolution layer, one layer of normalization layer and one layer of maximum pooling layer.

[0069] Secondly, in each task, N+1 groups of videos are processed by one layer of three-dimensional convolution layer with 64 3×3×3 three-dimensional convolution kernels and output channel of 64 in the low-order feature extraction network to obtain F conv , and F conv is processed by the normalization layer in the low-order feature extraction network to obtain F batchnorm , and further F batchnorm is input to the maximum pooling layer in the low-order feature extraction network to obtain N low-order features F SL corresponding to N groups of support set videos and a single low-order feature F QL corresponding to one group of query set videos:

[0070]

[0071] F batchnorm =Batchnorm3(F conv );

[0072] Wherein, is one layer of three-dimensional convolution layer containing 64 3×3×3 three-dimensional convolution kernels and output channel number of 64, V n ={S n1 ,S n2 ,...,S nN ,Q ni}; Batchnorm3() represents the normalization operation;

[0073] Finally, the expectation μ and the standard deviation σ of N low-order features F SL corresponding to N groups of support set videos in each task are solved respectively, and the formula Info=G(μ,diag(σ2 ), and further inputting the multivariate Gaussian output value Info of each task into a full connection layer for linear transformation to convert into three-dimensional convolution adaptive parameters P of each task for the high-order feature extraction network adaptive ; wherein P adaptive = f(Info); Info represents the multivariate Gaussian output value of the expectation μ and the standard deviation σ; diag() represents a diagonal matrix construction function.

[0074] Step S3, based on the adaptive parameters in the high-order feature extraction network of each task, adjusting the three-dimensional convolution layer in the high-order feature extraction network used by each task, and using the adjusted high-order feature extraction network corresponding to each task to perform high-order feature extraction on the N+1 low-order features extracted by each task to obtain N high-order features corresponding to the N groups of support set videos and a single high-order feature corresponding to the query set video in each task;

[0075] The specific process is as follows: first, based on the adaptive parameters P adaptive in the high-order feature extraction network of each task, adjusting the three-dimensional convolution layer in the high-order feature extraction network used by each task; wherein the three-dimensional convolution layer in the high-order feature extraction network used by each task is a three-dimensional convolution with 64 3x3x3 three-dimensional convolution kernels and an output channel number of 64.

[0076] Secondly, in each task, the N+1 low-order features are processed by the three-dimensional convolution with 64 3x3x3 three-dimensional convolution kernels and an output channel number of 64 in the three layers of the adjusted high-order feature extraction network to obtain N high-order features F SH corresponding to the N groups of support set videos and a single high-order feature F QH corresponding to the query set video:

[0077]

[0078]

[0079] Step S4, according to the N high-order features corresponding to the N groups of support set videos and the single high-order feature corresponding to the query set video in each task, calculating the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task;

[0080] The specific process is as follows: first, calculating the spatio-temporal self-attention similarity of each task, specifically:

[0081] (1) In each task, according to the N high-order features F SHand the single high-order feature F corresponding to the query set video of the 1st group QH , and the spatio-temporal self-attention vector M of the query set video is obtained through the formula M ST =mean(F SH ×(F SH ) T ) ST QT QH QH T QT ; wherein, mean() represents the operation of taking the average value; T represents transposition;

[0082] (2) The intra-task learner K inner () is composed of a two-dimensional convolution layer, a Relu nonlinear activation layer, a two-dimensional convolution layer and a Softmax logistic regression function;

[0083] (3) The spatio-temporal self-attention vector M ST of each group of support set videos and the spatio-temporal self-attention vector M QT of the query set video in each task are input into the intra-task information learner K inner () respectively, so as to obtain the optimized self-attention vector A ST of each group of support set videos in each task and the optimized self-attention vector A QT of the query set video in each task; wherein, A ST =K inner (M ST ); A QT =K inner (M QT );

[0084] (4) The enhanced spatio-temporal self-information feature T ST of each group of support set videos in each task is obtained through the formula T SH =F ST ×A SH +F ST , and the enhanced spatio-temporal self-information feature T QT of the query set video in each task is obtained through the formula T QH =F QT ×A QH +F QT ;

[0085] (5) The similarity Sim T between the query set video and each group of support set videos in each task is obtained through the formula Sim ST =Dcos(T QT ​​​​​), calculate the reinforcement spatio-temporal self-information feature T of the query set video in each task QT and the reinforcement spatio-temporal self-information feature T of each group of support set videos in the task ST , to obtain the N spatio-temporal self-attention similarity Sim of each task T .

[0086] Second step, calculate the relevant cross-attention metric similarity of each task, specifically:

[0087] (1) In each task, according to the N high-order features F SH corresponding to the N groups of support set videos and the single high-order feature F QH corresponding to the query set video, and through the formula M SR =mean(F SH ·(F QH ) T ), obtain the relevant cross-attention vector M SR of each group of support set videos, and through the formula M QR =mean(F QH ·(F SH ) T ), obtain the relevant cross-attention vector M QR of the query set video;

[0088] (2) Input the relevant cross-attention vector M SR of each group of support set videos in each task and the relevant cross-attention vector M QR of the query set video into the task internal information learner K inner () in the first step, to obtain the optimized relevant cross-attention vector A SR of each group of support set videos in each task and the optimized relevant cross-attention vector A QR of the query set video in each task; wherein A SR =K inner (M SR ); A QR =K inner (M QR );

[0089] (3) Through the formula R SR =F SH ×A SR +F SH , obtain the reinforcement spatio-temporal self-information feature R SR of each group of support set videos in each task, and through the formula R QR =F QH ×A QR +F QH , obtain the reinforcement spatio-temporal self-information feature RThe enhanced spatiotemporal self-information features R of the query set videos in each task are obtained. QR ;

[0090] (4) Using the formula Sim R =Dcos(R) SR R QR ), calculate the enhanced spatiotemporal self-information features R of the query set videos in each task. QR The enhanced spatiotemporal self-information features R of each set of support videos in the same task SR The cosine distance is used to obtain the N relevant cross-attention similarities Sim for each task. R .

[0091] Step S5: Based on the spatiotemporal self-attention similarity and the relevant cross-attention metric similarity of each task, select one set of support set videos that meet the predetermined conditions from the N sets of support set videos for each task.

[0092] The specific process is as follows: First, in each task, based on the obtained N spatiotemporal self-attention similarities Sim... T Similarity with N related cross-attention points Sim R The spatiotemporal self-attention similarity Sim belonging to the same action category was determined. T and cross-attention similarity Sim R And through the formula Sim = α × Sim T +(1-α)×Sim R This yields N similarity scores, Sim, between the query set video and the N support set videos; where α equals 0.25 or other decimals less than 1.

[0093] Secondly, the N similarity scores Sim obtained in each task are sorted from largest to smallest, and in the N support set videos of each task, the support set videos corresponding to the maximum similarity scores Sim are selected.

[0094] Therefore, the logical structure of steps S1 to S5 above can be summarized as shown in Figure 5. 3 The framework diagram shown.

[0095] like Figure 4 As shown in the figure, a few-sample video action recognition system based on hierarchical task information is provided in an embodiment of the present invention, comprising:

[0096] The video task construction unit 110 is configured to obtain a video dataset with X action categories, and construct a plurality of tasks based on the video dataset, and pre-process the video in each task; wherein each task has N sets of support set videos with different action categories and 1 set of query set videos inconsistent with the content of the N sets of support set videos; X and N are positive integers, and X>N;

[0097] The low-order feature extraction unit 120 is configured to extract low-order features of N+1 sets of videos in each task by using a low-order feature extraction network, to obtain N low-order features corresponding to the N sets of support set videos and a single low-order feature corresponding to the 1 set of query set videos in each task, and obtain adaptive parameters in the high-order feature extraction network of each task according to the N low-order features corresponding to the N sets of support set videos in each task;

[0098] The high-order feature extraction unit 130 is configured to adjust a three-dimensional convolution layer in the high-order feature extraction network used in each task based on the adaptive parameters in the high-order feature extraction network of each task, and perform high-order feature extraction on the N+1 low-order features extracted in each task by using the adjusted high-order feature extraction network corresponding to each task, to obtain N high-order features corresponding to the N sets of support set videos and a single high-order feature corresponding to the 1 set of query set videos in each task;

[0099] The similarity calculation unit 140 is configured to calculate a spatio-temporal self-attention similarity and a related cross-attention measure similarity of each task according to the N high-order features corresponding to the N sets of support set videos and the single high-order feature corresponding to the 1 set of query set videos in each task.

[0100] The video recognition unit 150 is configured to correspondingly screen out a set of support set videos meeting a predetermined condition from the N sets of support set videos in each task according to the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task.

[0101] The embodiment of the present application has the following beneficial effects:

[0102] 1. The present application solves the problem of few samples by meta-learning. The purpose of meta-learning is to enable the deep neural network to "learn to learn" by designing the neural network and pre-training with a small number of samples, so that the neural network model can quickly generalize to other scenarios, and can solve the problems of poor generalization performance and insufficient measurement efficiency in the existing few-sample video action recognition method;

[0103] 2. The present application can make the deep learning model have high generalization performance and high measurement performance with a small number of samples, and dynamically generate adaptive parameters of the feature extraction network according to the input task to construct a task-related feature space, thereby improving the model generalization.

[0104] 3. The application combines the task information learner with the self-attention mechanism and the cross-attention mechanism to mine the intra-task feature key self-information and mutual information, strengthen the feature space-time key information and the correlation between features, and further improve the measurement efficiency.

[0105] It is worth noting that in the above system embodiments, each unit included is only divided according to the functional logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for the convenience of mutual differentiation, and is not used to limit the protection scope of the application.

[0106] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc.

[0107] The above only discloses the preferred embodiments of the application, and of course cannot limit the scope of the application, so equivalent changes made according to the claims of the application are still within the scope of the application.

Claims

1. A method for few-shot video action recognition based on hierarchical task information, characterized in that, The method comprises the following steps: S1, acquiring a video dataset with X action categories, and based on the video dataset, constructing a plurality of tasks, and preprocessing the video in each task; wherein each task has N sets of action category different support set videos and 1 set of query set videos inconsistent with the content of the N sets of support set videos; X and N are positive integers, and X>N; S2, using a low-order feature extraction network to extract the low-order features of the N+1 sets of videos in each task to obtain N low-order features corresponding to the N sets of support set videos and a single low-order feature corresponding to the 1 set of query set videos in each task, and obtaining adaptive parameters in the high-order feature extraction network of each task according to the N low-order features corresponding to the N sets of support set videos in each task; S3, based on the adaptive parameters in the high-order feature extraction network of each task, adjusting the three-dimensional convolution layer in the high-order feature extraction network used by each task, and using the adjusted high-order feature extraction network corresponding to each task to extract the high-order features of the N+1 low-order features extracted by each task, to obtain N high-order features corresponding to the N sets of support set videos and a single high-order feature corresponding to the 1 set of query set videos in each task; S4, according to the N high-order features corresponding to the N sets of support set videos and the single high-order feature corresponding to the query set video in each task, calculating the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task; S5, according to the spatio-temporal self-attention similarity and the related cross-attention measure similarity of each task, corresponding to filtering out a set of support set videos meeting the predetermined condition from the N sets of support set videos of each task. 2.The method of claim 1, wherein, The step S1 specifically comprises: constructing a video dataset, the video dataset comprising action videos of X action categories; dividing the dataset into M tasks, and each task comprising N sets of videos of different action categories as support set videos, and randomly selecting 1 set of videos completely inconsistent with the content of the support set videos as query set videos from the N categories; preprocessing each set of support set videos and query set videos in each task. 3.The method of claim 2, wherein, The preprocessing of each set of support set videos and query set videos comprises video frame sampling and frame image processing after sampling, specifically: randomly selecting the initial position of the first frame image of each set of videos, and then continuously sampling 16 frames; Each frame is cropped into a 224×224 resolution image block, and then normalized. The resulting N+1 normalized video sequences are denoted as {S}. p1 ,S p2 ,...,S pN Q pi }; where S p1 ,S p2 ,...,S pN Q represents the support set videos of N different action categories after normalization; pi This represents the support set of videos for a group of action categories i after normalization; i N. 4.The method of claim 3, wherein, The step S2 specifically comprises: determining that the low-order feature extraction network is composed of one layer of three-dimensional convolution layer, one layer of normalization layer and one layer of maximum pooling layer; In each task, the N+1 groups of videos are processed by a three-dimensional convolution layer with 64 three-dimensional convolution kernels of 3*3*3 and 64 output channels in the low-level feature extraction network to obtain , and the is further processed by a normalization layer in the low-level feature extraction network to obtain , and the is further input into a max-pooling layer in the low-level feature extraction network to obtain N low-level features corresponding to the N groups of support set videos F SL and a single low-level feature corresponding to the 1 group of query set videos F QL : ； ； wherein, is a three-dimensional convolution layer including 64 three-dimensional convolution kernels of 3x3x3, and the number of output channels is 64, V n = { S n1 ,S n2 ,..., S nN ,Q ni }; denotes a normalization operation; Solving N low-order features F corresponding to N groups of support set videos in each task SL The respective expectation μ and standard deviation σ are obtained by formula The expectation μ and standard deviation σ solved for each task are multivariate Gaussian, and the output values of the multivariate Gaussian of each task , are all transmitted into a full connection layer for linear transformation to convert into three-dimensional convolution adaptive parameters of each task for high-order feature extraction network ; wherein ; The output value of the multivariate Gaussian of the expectation μ and the standard deviation σ is represented. The diagonal matrix construction function is represented.

5. The few-shot video action recognition method based on hierarchical task information of claim 4, wherein, The step S3 specifically comprises: Adaptive parameters in high-order feature extraction network for each task , adjusting three-dimensional convolution layers in the high-order feature extraction network used by each task; wherein the three-dimensional convolution layers in the high-order feature extraction network used by each task are all three layers of three-dimensional convolution with 64 three-dimensional convolution kernels of 3×3×3 and output channel number of 64. In each task, the N+1 low-order features are processed by three layers of three-dimensional convolution with 64 3x3x3 three-dimensional convolution kernels and 64 output channels in the high-order feature extraction network corresponding to the respective adjusted high-order feature extraction network, to obtain N high-order features corresponding to the N support set videos and a single high-order feature corresponding to the query set video : 。 6.The method of claim 5, wherein, The step S4 specifically comprises: First step, calculating the spatio-temporal self-attention similarity of each task, specifically: In each task, based on the N high-order features corresponding to the N sets of support set videos... And a single high-order feature corresponding to a set of query set videos. And through the formula This yields the spatiotemporal self-attention vector for each support set video. and through formula This yields the spatiotemporal self-attention vectors of the query set videos. ;in, This indicates the operation of calculating the average value; Indicates transpose; Determining in-task learner is composed of a two-dimensional convolution layer, a Relu nonlinear activation layer, a two-dimensional convolution layer, and Softmax a logistic regression function; spatial self-attention vectors of each group of support set videos in each task spatial self-attention vectors of query set videos respectively input into an intra-task information learner to obtain optimized spatial self-attention vectors of each group of support set videos in each task and optimized spatial self-attention vectors of query set videos in each task ; wherein ; ; Through formula The enhanced spatiotemporal self-information features of each support set video in each task are obtained. and through formula The enhanced spatiotemporal self-information features of the query set videos in each task are obtained. ; Through formula Calculate the enhanced spatiotemporal self-information features of the query set videos in each task. Features of enhanced spatiotemporal self-information of each set of support videos in the same task The cosine distance is used to obtain the N spatiotemporal self-attention similarities for each task. : Second step, calculating the related cross-attention measure similarity of each task, specifically: In each task, according to N high-order features corresponding to N groups of support set videos and a single high-order feature corresponding to a group of query set videos , and through the formula , the relevant cross-attention vector of each group of support set videos is obtained , and through the formula , the relevant cross-attention vector of the query set video is obtained ; the relevant cross-attention vectors of each group of support set videos in each task and the relevant cross-attention vectors of the query set videos respectively input into the intra-task information learner to obtain the optimized relevant cross-attention vectors of each group of support set videos in each task and the optimized relevant cross-attention vectors of the query set videos in each task ; wherein ; ; Through formula The enhanced spatiotemporal self-information features of each support set video in each task are obtained. and through formula The enhanced spatiotemporal self-information features of the query set videos in each task are obtained. ; The features of the reinforced spatio-temporal self-information of the query set video in each task are calculated by the formula The cosine distance between the features of the reinforced spatio-temporal self-information of each group of support set videos in the same task is obtained to obtain N relevant cross-attention similarities of each task .​​ 7. The few-shot video action recognition method based on hierarchical task information of claim 6, wherein, The step S5 specifically comprises: In each task, according to the obtained N spatio-temporal self-attention similarities and N related cross-attention similarities , the spatio-temporal self-attention similarities and the cross-attention similarities belonging to the same action category are determined, and N similarities corresponding to the query set video and the N groups of support set videos are obtained through the formula . ; N similarity degrees corresponding to each task are obtained The N similarity degrees corresponding to each task are obtained The support set video corresponding to the maximum similarity degree 8.A few-shot video action recognition system based on hierarchical task information, characterized in that, including: a video task construction unit configured to obtain a video dataset having X action categories, and construct a plurality of tasks based on the video dataset, and pre-process videos in each task; wherein each task has N sets of support set videos with different action categories and 1 set of query set videos inconsistent with the N sets of support set videos in content; X and N are positive integers, and X>N; a low-order feature extraction unit configured to extract low-order features of N+1 sets of videos in each task using a low-order feature extraction network, to obtain N low-order features corresponding to the N sets of support set videos and a single low-order feature corresponding to the 1 set of query set videos in each task, and obtain adaptive parameters for a high-order feature extraction network in each task according to the N low-order features corresponding to the N sets of support set videos in each task; a high-order feature extraction unit configured to adjust a three-dimensional convolution layer in the high-order feature extraction network used in each task based on the adaptive parameters for the high-order feature extraction network in each task, and perform high-order feature extraction on the N+1 low-order features extracted in each task using the adjusted high-order feature extraction network corresponding to each task, to obtain N high-order features corresponding to the N sets of support set videos and a single high-order feature corresponding to the 1 set of query set videos in each task; a similarity calculation unit configured to calculate a spatio-temporal self-attention similarity and a relevant cross-attention measure similarity of each task according to the N high-order features corresponding to the N sets of support set videos and the single high-order feature corresponding to the 1 set of query set videos in each task; a video recognition unit configured to filter out a set of support set videos meeting a predetermined condition from the N sets of support set videos in each task according to the spatio-temporal self-attention similarity and the relevant cross-attention measure similarity of each task.

Images