Action recognition method based on progressive few-shot knowledge distillation

Abstract

The present invention discloses an action recognition method based on progressive few-sample knowledge distillation. The present invention first uniformly samples the original video to obtain an RGB frame image sequence and an optical flow image sequence, and constructs a teacher network and a student network respectively; further, the original video is randomly sampled in units of segments to obtain a set of sampled video segments; then, the contribution of each video segment is measured through a contribution scoring module; at the same time, a diversified few-sample selection module is used to guide sample selection to obtain a small number of differentiated video samples for training; finally, by progressively distilling the selected sample set and dynamically adjusting the distillation temperature, the knowledge transfer of the teacher model to the student model is realized. The method of the present invention can not only fully measure the strength of the temporal correlation of video samples, but also accurately characterize the relationship between video segments and the current action model, realize the lightweight of the few-sample action recognition model, and facilitate the deployment of the model on resource-constrained edge computing devices.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, in particular to the field of visual model compression and action recognition, and specifically relates to an action recognition method based on progressive few-sample knowledge distillation. Background Art

[0002] Action recognition is an important computer vision task that aims to understand and identify actions in videos. It is widely used in video security, human-computer interaction, intelligent driving and other fields. In recent years, with the continuous development of deep neural networks, the models represented by single convolutional neural networks have gradually transformed into models based on transformers or multi-network structure fusion. Although the recognition accuracy has been improved to a certain extent, the complexity and number of parameters of the model have increased, which has increased the computing cost and training difficulty, prolonged the inference time, and increased the computing resource requirements for the deployment environment. Therefore, there is an urgent need to compress complex models, that is, model compression technology, which aims to reduce the complexity of the model by reducing the number of network parameters while maintaining model performance. This can improve training efficiency and shorten inference time, so that it can be deployed on edge and end devices with limited computing resources, which has important application value.

[0003] Different from object detection, action recognition models not only describe the spatial dependencies of objects, but also reflect the temporal dynamic characteristics of objects. For example, temporal information such as the target's motion trajectory, the continuity of the action, and the duration of the action are usually considered to better understand and identify specific actions or behaviors. In general, the existing methods can be divided into three categories: 1) 3D Convolutional Neural Network method: the dynamic change relationship of people or objects in the video is modeled in spatial and temporal dimensions through a 3D convolutional layer, so as to capture the spatiotemporal pattern of the action and improve the recognition performance; 2) Two-Stream Fusion method: a two-stream framework is used to extract the appearance features and motion features of the video respectively. Among them, the appearance stream features usually reflect the static visual attributes of the object such as shape, color, texture, etc., while the motion stream features capture the optical flow information by calculating the temporal difference between consecutive frames, revealing the change trend of the action and its influencing factors, such as movement speed, direction, etc.; 3) Hybrid Network method: usually a combination of different types of networks such as long short-term memory network (LSTM), convolutional neural network (CNN), recurrent neural network (RNN), graph convolutional neural network (Graph CNN) is used to capture video features of different spatiotemporal granularity and realize the interactive fusion of spatiotemporal features between these networks.

[0004] Knowledge distillation is a typical method of model compression. It mainly transfers the knowledge of a larger network (teacher model) to a smaller network (student model) to obtain a lightweight model, thereby achieving fast reasoning of learning tasks under resource constraints. The so-called knowledge usually refers to the patterns or rules captured by the teacher model on the training data, specifically involving network weights, biases and activation functions; according to the different ways of knowledge representation, it can be divided into three knowledge distillation methods: response-based, feature-based and relation-based. Among them, response-based distillation allows the student model to imitate the teacher model's response to input data, learn the teacher model's decision-making process and output features; feature-based distillation uses the abstract representation reflected by the intermediate layer features of the teacher model as knowledge and passes it to the student model; relation-based distillation uses the semantic correlation between network layers or samples as knowledge, which not only describes the semantic relationship between input samples and their feature graphs, but also transfers the sample feature graph relationship of the teacher model to the student model.

[0005] The shortcomings of the above methods are mainly reflected in the following aspects: (1) Whether it is the early two-stream architecture method or the method based on three-dimensional convolutional network, most of them can only characterize short-term temporal dependencies and have high time complexity; in addition, the collaborative training and information interaction of different types of networks in the hybrid network bring a certain computational burden, which increases the time complexity; (2) The traditional method uses random sampling to obtain video samples, which is prone to category imbalance and catastrophic forgetting problems. The former refers to the large difference in the number of samples of different categories, which makes the model learning favor the category with more samples; the latter refers to the fact that when the model uses training samples to learn parameters to process certain tasks, it may forget the old tasks that have been processed or the old samples that have been used, resulting in performance degradation; (3) When the feature-based distillation method compresses the action recognition model, it ignores the fact that different video samples have different distillation effects between the teacher and student models. For example, the video clips where the student model recognizes the wrong action often need higher weights; this requires more training samples and more training rounds to achieve satisfactory distillation performance, resulting in low distillation efficiency. Therefore, there is an urgent need to design an action recognition method that can not only fully capture the spatiotemporal relationship of the video, but also accurately reflect the impact of different video samples on the knowledge distillation performance under a given model compression rate. Summary of the invention

[0006] The purpose of the present invention is to provide an action recognition method based on progressive small-sample knowledge distillation in view of the deficiencies of the prior art. The method of the present invention calculates the sample distillation contribution and selects the sample with the largest contribution based on the temporal correlation strength of the video samples and the relationship between the video clips and the current action model; based on the sample contribution score and the sample diversity score, a small number of samples suitable for the current model training are screened to solve the sample class imbalance problem and catastrophic forgetting problem, perform efficient knowledge distillation, and finally realize the lightweight action recognition model.

[0007] The method of the present invention performs the following operations in sequence for a given video data set containing category labels:

[0008] Step (1) uniformly sampling the original video to obtain RGB frame image sequence and optical flow image sequence, respectively constructing a teacher network and a student network with a spatiotemporal feature fusion module as the core for action recognition;

[0009] Step (2) randomly samples the original video in segments to obtain a set of sampled video segments;

[0010] Step (3) constructing a video clip contribution score module, taking as input a set of sampled video clips, and using an adversarial perturbation sample generation module and a timing reconstruction module to output a contribution score for each video clip;

[0011] Step (4) construct a diversified few-sample selection module, with input as all video clips and their contribution scores, and use the self-distillation module and sample diversification module to output a small number of differentiated video samples;

[0012] Step (5) inputting the differentiated video samples into the teacher network and the student network, calculating the cross entropy loss and focal loss between the real action category and the action category predicted by the student network, calculating the distance loss between the updated teacher action category probability and the student action category probability, and optimizing the student network using the stochastic gradient descent method;

[0013] Step (6) Repeat steps (3) to (5) until the student model converges, uniformly sample the new video sample to obtain RGB frame image sequence and optical flow image sequence, input them into the student network to obtain the predicted action category probability, and select the category with the largest probability as the action category of the new video.

[0014] Further, step (1) is specifically:

[0015] (1-1). Sample the original video at a rate of s frames per second Perform uniform sampling, is a real number domain, t', w, h, c are the total duration, width, height, and number of channels of the video sample, 20≤s≤50, and the frame stream image sequence is obtained Among them, the subscript I represents the image, I k represents the kth RGB three-channel image with width w and height h in the sequence, n' represents the number of images in the frame stream image sequence, and k represents the frame stream image index;

[0016] (1-2) Based on the obtained frame flow image sequence, calculate the optical flow of 10 consecutive frames before and after to obtain the optical flow image sequence Among them, the subscript O represents the optical flow, O k represents the k-th optical flow image with width w, height h and channel dimension 10 corresponding to the frame flow image with index k;

[0017] (1-3). Use two VGG-16 networks pre-trained on the ImageNet dataset to form a teacher encoder. The VGG-16 network is a VGG network with 16 layers. Use two VGG-16 networks to extract frame flow image and optical flow image features respectively; Use two VGG-11 networks with randomly initialized parameters to construct a student encoder to extract frame flow image and optical flow image features. The VGG-11 network is a VGG network with 11 layers.

[0018] (1-4). Convert the frame stream image sequence Input into one of the VGG-16 networks of the teacher encoder to obtain the teacher spatial encoding features The superscript tea means teacher, and the subscript Represents Spatial, w1, h1 and c1 represent the width, height and number of channels of the encoded feature respectively;

[0019] The optical flow image sequence Input into one of the VGG-16 networks of the teacher encoder to obtain the teacher's temporal encoding features Subscript Indicates feature timing;

[0020] Frame stream image sequence and optical flow image sequence Input the two VGG-11 networks of the student encoder at the same time to obtain the student spatial encoding features With temporal coding features The superscript stu means student;

[0021] (1-5). The spatial encoding features in the teacher and student models and the optical flow image sequence are concatenated along the channel dimension. The features of the concatenated teacher and student models are and The subscript con represents feature concatenation; the teacher model uses a three-layer converter to fuse the concatenated features, where each layer of converter contains a multi-head self-attention and a multi-layer perceptron, and the output is The subscript m represents feature fusion. The student model uses three layers of three-dimensional large-core convolutional layers for feature extraction, where each large-core convolutional layer consists of n' large-core convolutional kernels. A layer normalization operation and two sets of 1×1 convolutions are used to map features to a higher dimension and map them back to the original channel dimension respectively. The output is

[0022] (1-6). Finally, the teacher model The fusion features of the student model The video actions are classified after a fully connected layer and a softmax(·) function.

[0023] Furthermore, step (2) is as follows: Uniformly sample n video clips, that is, Among them, the four-dimensional tensor X i represents the i-th video clip, and t is the number of video frames in the video clip.

[0024] Furthermore, step (3) is specifically:

[0025] (3-1). The video clip contribution score module consists of an adversarial perturbation sample generation module and a time series reconstruction calculation module;

[0026] (3-2) Construct an adversarial perturbation sample generation module to measure the robustness of the student model to the video sample. The input is the video clip X i , for video segment X i Add perturbation η to get attack sample Among them, θ is the parameter of the action recognition model, y is the action label of the action recognition model, is the cross entropy loss, is the loss function for training the current student model, the subscript ce represents the cross entropy, ∈>0 is a hyperparameter,

[0027] is the loss function for the original video clip X i The gradient of the original video clip X is , sign(·) is the sign function, when the input is positive, the result is 1, when the input is negative, the result is -1, and when the input is zero, the result is 0; i With attack sample X i ′ Input the student action recognition model respectively, and output the adversarial robustness score of the video clip according to the difference between the cross entropy losses of the two The subscript adv indicates adversarial;

[0028] (3-3) To measure the impact of the video clip on the student model, the input is the loss function of step (3-1) about the original video clip X i Gradient Perform l2 norm constraint and output gradient influence score The superscript gra indicates gradient;

[0029] (3-4) Construct a time reconstruction calculation module to measure the strength of the temporal relationship of the video clip action and reflect the frequency of occurrence of an action in a certain temporal interval. The input is the video clip X i , from X i Randomly extract K frames for mask processing, that is, assign 1 to all elements in its feature matrix to obtain the masked video clip The sampled video frames are reconstructed using the multi-head attention autoencoder model pre-trained on the ImageNet-21K dataset. The multi-head attention autoencoder contains 2 multi-head self-attentions and 8 multi-layer perceptrons to obtain the reconstructed video clips. Then calculate the reconstruction score Characterizes the strength of the temporal relationship of actions in video clips, and the superscript tem indicates the score timing;

[0030] (3-5) Calculate the final contribution of the video clip, specifically calculate the linear combination of the above three scores, and output the final contribution of the video clip G = αG adv +βG tem +(1-α-β)Ggra , 0<α<0.5 and 0<β<0.5 are hyper parameters.

[0031] Furthermore, step (4) is specifically:

[0032] (4-1). The diversified few-shot selection module consists of a self-distillation module and a sample diversification module, where few-shot refers to a small number of labeled training samples and diversified refers to differentiated training samples;

[0033] (4-2) Input is all video clips and their contribution scores G, and the clip with the highest contribution in each video Perform average pooling on its features in the time dimension to obtain the feature matrix of the i′th video in the k′th round The fragment with the highest impact factor is X max The feature matrix of the jth frame in the k′th round is N is the total number of samples, and the output is the feature matrix of each video in the kth round. One round means the process of passing the entire training data set through the neural network once;

[0034] (4-3) Construct a self-distillation module to alleviate the catastrophic forgetting problem in the process of few-sample distillation; the input is the k′th feature matrix of the i′th video and the characteristic matrix in k′-1 rounds The memory-enhanced feature matrix is ​​obtained by using the self-distillation strategy, namely is a hyperparameter; the output is the memory-enhanced feature matrix F for each video i ″;

[0035] (4-4). Construct a sample diversification module to select the videos required for training. The input is the final contribution score G of all training videos and the memory enhancement feature matrix F i ″, a method to solve the subset selection problem using the determinant point process method, selecting a subset with diversity from the sample set, the more similar the samples are, the less likely they are to be sampled at the same time;

[0036] This module starts from a complete dataset with N videos In the process, we select Q differentiated video samples to form a set in, Represents the j′th original video, whose feature matrix set Represents the k′th original video, whose feature matrix set Will gather Each feature matrix F j ″、 Flatten them into one-dimensional vectors respectively, The Q expanded one-dimensional vectors are concatenated into a comprehensive matrix Will The N expanded one-dimensional vectors are concatenated into a comprehensive matrix Utilizing parameters Measuring Collection The diversity of video samples, det(·) represents the determinant, is the identity matrix, and the superscript T indicates transpose;

[0037] (4-5). Optimize the objective function through the greedy algorithm: Among them, G j′ For collection The final contribution score of the j′th sample in , 0<j′≤Q, argmax(×) represents the index used to find the maximum value, this objective function selects the optimal training video set 0<λ<1 is a hyperparameter, outputting the optimal training video set

[0038] Furthermore, step (5) is specifically:

[0039] (5-1). Collect the best training videos Input the teacher model and the student model. In order to alleviate the problem of balancing the number of samples of different classes, calculate the action category label of the video sample in the form of one-hot encoding. and the student action category probability vector The focal loss between The superscript C is the number of action categories. Wherein, the subscript fl represents the focal loss and p is p stu The probability value of the correct action category output in , α′>0 and γ′>1 are both hyper parameters, and the probability of the teacher action category The soft label of the teacher model is obtained by processing with the softmax(·) function The softened logical value of the "i" class is expressed as in, Represents the logical value of the i-th category of the teacher model 1≤i″≤C, the category index 1≤j″≤C, T represents the distillation temperature, exp(×) represents the exponential function, the distillation temperature of each video sample is positively correlated with the final contribution G of the sample T=μG, μ≥0 is a hyperparameter, and the distillation temperature is adaptively adjusted for different samples; calculate the probability of the teacher action category and the student action category probability vector p stu The relative entropy between As a distance loss, the subscript kl represents the divergence;

[0040] (5-2). Calculate the final loss function ζ>0 is a hyperparameter, and the student network is optimized using stochastic gradient descent to perform adaptive progressive knowledge distillation using only a small number of labeled samples.

[0041] Furthermore, step (6) is specifically:

[0042] (6-1) Repeat steps (3) to (5) until the student model converges;

[0043] (6-2). For new videos Uniform sampling to obtain RGB frame image sequence and optical flow image sequence The kth RGB three-channel image in the sequence with width w0 and height h0 The k-th optical flow image with width w0, height h0 and channel dimension 10 corresponding to the frame flow image with index k

[0044] (6-3) The RGB frame image sequence obtained by sampling and optical flow image sequence Input the trained student model to obtain the new student action category probability vector Select the category with the highest probability As new video The action category of the new video is the largest category in the action category probability vector.

[0045] The action recognition method based on progressive few-sample knowledge distillation proposed in the present invention has the following characteristics: 1) using the adversarial perturbation sample generation module and the timing reconstruction calculation module, combined with the sample gradient, the sample distillation contribution is calculated and the sample with the largest contribution is selected accordingly; 2) through the video sample diversification module and the self-distillation module, a small number of differentiated video samples are screened and the distillation temperature is dynamically adjusted to perform progressive few-sample knowledge distillation to improve the accuracy of action recognition.

[0046] The present invention aims at the problem of missing the description of the relationship between the video clip and the current model during the distillation process, and has the following beneficial effects: 1) The strength of the temporal correlation of the video clip is measured by mask reconstruction, which solves the problem of high time complexity of describing the strength of the temporal correlation; 2) The self-distillation module is combined with the focal loss function to solve the problem of sample class imbalance and catastrophic forgetting; 3) Based on DPP diversified sampling and model contribution scores, a small number of differentiated video samples are screened, and progressive knowledge distillation is performed, fully considering the relationship between the video clip and the current action model, shortening the distillation time of the action recognition model. The present invention is applied to practical scenarios such as human-computer interaction, intelligent driving, monitoring and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0048] Action recognition methods based on progressive few-shot knowledge distillation, such as Figure 1 As shown, for a given video data set with category labels, the following operations are performed in sequence:

[0049] Step (1) uniformly samples the original video to obtain RGB frame image sequences and optical flow image sequences, and constructs a teacher network and a student network with a spatiotemporal feature fusion module as the core for action recognition. Specifically:

[0050] (1-1) Sample the original video at a rate of 30 frames per second Perform uniform sampling, is the real number domain, t', w, h, c are the total duration, width, height, and number of channels of the video sample, respectively, and the frame stream image sequence is obtained Among them, the subscript I represents the image Image, I k represents the kth RGB three-channel image with width w and height h in the sequence, n' represents the number of images in the frame stream image sequence, and k represents the frame stream image index;

[0051] (1-2) Based on the obtained frame flow image sequence, calculate the optical flow of 10 consecutive frames before and after to obtain the optical flow image sequence Among them, the subscript O represents the optical flow, O k represents the k-th optical flow image with width w, height h and channel dimension 10 corresponding to the frame flow image with index k;

[0052] (1-3). Use two VGG-16 networks pre-trained on the ImageNet dataset to form a teacher encoder. The VGG-16 network is a VGG network (Visual Geometry Group) with 16 layers. Use two VGG-16 networks to extract frame flow image and optical flow image features respectively; Use two VGG-11 networks with randomly initialized parameters to construct a student encoder to extract frame flow image and optical flow image features. The VGG-11 network is a VGG network with 11 layers.

[0053] (1-4). Convert the frame stream image sequence Input into one of the VGG-16 networks of the teacher encoder to obtain the teacher spatial encoding features The superscript tea means teacher, and the subscript Represents Spatial, w1, h1 and c1 represent the width, height and number of channels of the encoded feature respectively;

[0054] The optical flow image sequence Input into one of the VGG-16 networks of the teacher encoder to obtain the teacher's temporal encoding features Subscript Indicates the characteristic temporal;

[0055] Frame stream image sequence and optical flow image sequence Input the two VGG-11 networks of the student encoder at the same time to obtain the student spatial encoding features With temporal coding features The superscript stu means student;

[0056] (1-5). The spatial encoding features in the teacher and student models and the optical flow image sequence are concatenated along the channel dimension. The features of the concatenated teacher and student models are and The subscript con represents feature concatenation; the teacher model uses a three-layer transformer to fuse the concatenated features, where each layer of transformer contains a multi-head self-attention and a multi-layer perceptron, and the output is The subscript m represents feature fusion. The student model uses three layers of three-dimensional large-core convolutional layers for feature extraction, where each large-core convolutional layer consists of n' large-core convolutional kernels. A layer normalization (LayerNorm) operation and two sets of 1×1 convolutions are used to map features to a higher dimension and map them back to the original channel dimension respectively. The output is

[0057] (1-6). Finally, the teacher model The fusion features of the student model The video actions are classified after a fully connected layer and a softmax(·) function.

[0058] Step (2) randomly samples the original video in segments to obtain a set of sampled video segments. Uniformly sample n video clips, that is, Among them, the four-dimensional tensor X i represents the i-th video clip, and t is the number of video frames in the video clip.

[0059] Step (3) constructs a video clip contribution score module, which takes as input a set of sampled video clips, and uses an adversarial perturbation sample generation module and a time sequence reconstruction module to output the contribution score of each video clip. Specifically:

[0060] (3-1). The video clip contribution score module consists of an adversarial perturbation sample generation module and a time series reconstruction calculation module;

[0061] (3-2) Construct an adversarial perturbation sample generation module to measure the robustness of the student model to the video sample. The input is the video clip X i , for video segment X i Add perturbation η to get attack sample Among them, θ is the parameter of the action recognition model, y is the action label of the action recognition model, is the cross entropy loss, is the loss function for training the current student model. The subscript ce represents cross-entropy, ∈>0 is a hyperparameter. is the loss function for the original video clip X i The gradient of the original video clip X is , sign(·) is the sign function, when the input is positive, the result is 1, when the input is negative, the result is -1, and when the input is zero, the result is 0; i With attack sample X i ′ Input the student action recognition model respectively, and output the adversarial robustness score of the video clip according to the difference between the cross entropy losses of the two The subscript adv means Adversarial;

[0062] (3-3) To measure the impact of the video clip on the student model, the input is the loss function of step (3-1) about the original video clip X i Gradient Perform l2 norm constraint and output gradient influence score The superscript gra represents gradient;

[0063] (3-4) Construct a time reconstruction calculation module to measure the strength of the temporal relationship of the video clip action and reflect the frequency of occurrence of an action in a certain temporal interval. The input is the video clip X i , from X i Randomly extract K frames for mask processing, that is, assign 1 to all elements in its feature matrix to obtain the masked video clip The sampled video frames are reconstructed using the multi-head attention autoencoder model pre-trained on the ImageNet-21K dataset. The multi-head attention autoencoder contains 2 multi-head self-attentions and 8 multi-layer perceptrons to obtain the reconstructed video clips. (Pre-select some video clips for fine-tuning) and then calculate the reconstruction score Characterizes the temporal relationship strength of actions in video clips. The superscript tem indicates the temporal score.

[0064] (3-5) Calculate the final contribution of the video clip, specifically calculate the linear combination of the above three scores, and output the final contribution of the video clip G = αG adv +βG tem +(1-α-β)G gra , 0<α<0.5 and 0<β<0.5 are hyper parameters.

[0065] Step (4) constructs a diversified small sample selection module, which takes all video clips and their contribution scores as input, and uses the self-distillation module and sample diversification module to output a small number of differentiated video samples. Specifically:

[0066] (4-1). The diversified few-shot selection module consists of a self-distillation module and a sample diversification module, where few-shot refers to a small number of labeled training samples and diversified refers to differentiated training samples;

[0067] (4-2) Input is all video clips and their contribution scores G, and the clip with the highest contribution in each video Perform average pooling on its features in the time dimension to obtain the feature matrix of the i′th video in the k′th epoch The fragment with the highest impact factor is X max The feature matrix of the jth frame in the k′th epoch is N is the total number of samples, and the output is the feature matrix of each video in the kth epoch. One epoch means passing the entire training data set through the neural network once (i.e., one forward propagation and one back propagation);

[0068] (4-3) Construct a self-distillation module to alleviate the catastrophic forgetting problem in the process of few-sample distillation; the input is the k′th feature matrix of the i′th video and the characteristic matrix in k′-1 rounds The memory-enhanced feature matrix is ​​obtained by using the self-distillation strategy, namely is a hyperparameter; the output is the memory-enhanced feature matrix F′ for each video i′ ;

[0069] (4-4). Construct a sample diversification module to select the videos required for training. The input is the final contribution score G of all training videos and the memory enhancement feature matrix F′ i′ , using the determinantal point process method (DPP: Determinantal Point Processes) to solve the subset selection problem, select a subset with diversity from the sample set. The more similar the samples are, the less likely they are to be sampled at the same time;

[0070] This module starts from a complete dataset with N videos In the process, we select Q differentiated video samples to form a set in, Represents the j′th original video, whose feature matrix set Represents the k′th original video, whose feature matrix set Will gather Each feature matrix F′ j′ , Flatten them into one-dimensional vectors respectively, The Q expanded one-dimensional vectors are concatenated into a comprehensive matrix Will The N expanded one-dimensional vectors are concatenated into a comprehensive matrix Utilizing parameters Measures the diversity of video samples in set S, det(·) represents the determinant, is the identity matrix, and the superscript T indicates transpose;

[0071] (4-5). Optimize the objective function through the greedy algorithm: Among them, G j′ For collection The final contribution score of the j′th sample in , 1<j′≤Q, argmax(·) represents the index used to find the maximum value, this objective function selects the optimal training video set 0<λ<1 is a hyperparameter, outputting the optimal training video set

[0072] Step (5) Input the differentiated video samples into the teacher network and the student network, calculate the cross-entropy loss and focal loss between the real action category and the action category predicted by the student network, calculate the distance loss between the updated teacher action category probability and the student action category probability, and optimize the student network using the stochastic gradient descent method. Specifically:

[0073] (5-1). Collect the best training videos Input the teacher model and the student model. In order to alleviate the problem of balancing the number of samples of different classes, calculate the action category label of the video sample in the form of one-hot encoding. and the student action category probability vector The focal loss between The superscript C is the number of action categories. Among them, the subscript fl represents the focal loss, and p is p stuThe probability value of the correct action category output in , α′>0 and γ′>1 are both hyper parameters, and the probability of the teacher action category The soft label of the teacher model is obtained by processing with the softmax(·) function The softened logical value of the "i" class is expressed as in, Represents the logical value of the i-th category of the teacher model 1≤j″≤C, the category index 1≤j″≤C, T represents the distillation temperature, exp(×) represents the exponential function, the distillation temperature of each video sample is positively correlated with the final contribution G of the sample T=μG, μ≥0 is a hyperparameter to achieve adaptive adjustment of the distillation temperature for different samples; calculate the probability of the teacher action category and the student action category probability vector p stu The relative entropy between As the distance loss, the subscript kl represents the divergence Kullback-Leibler;

[0074] (5-2). Calculate the final loss function The student network is optimized using stochastic gradient descent as a hyperparameter, and adaptive progressive knowledge distillation is performed using only a small number of labeled samples.

[0075] Step (6) Repeat steps (3) to (5) until the student model converges, uniformly sample the new video sample to obtain the RGB frame image sequence and the optical flow image sequence, input them into the student network to obtain the predicted action category probability, and select the category with the largest probability as the action category of the new video. Specifically:

[0076] (6-1) Repeat steps (3) to (5) until the student model converges;

[0077] (6-2). For new videos Uniform sampling to obtain RGB frame image sequence and optical flow image sequence The kth RGB three-channel image in the sequence with width w0 and height h0 The k-th optical flow image with width w0, height h0 and channel dimension 10 corresponding to the frame flow image with index k

[0078] (6-3) The RGB frame image sequence obtained by sampling and optical flow image sequence Input the trained student model to obtain the new student action category probability vector Select the category with the highest probability As new video The action category of the new video is the largest category in the action category probability vector.

[0079] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An action recognition method based on progressive few-sample knowledge distillation, characterized in that: For a given set of video data with category labels, perform the following operations in sequence: Step (1) uniformly sampling the original video to obtain RGB frame image sequence and optical flow image sequence, respectively constructing a teacher network and a student network with a spatiotemporal feature fusion module as the core for action recognition; Step (2) randomly samples the original video in units of segments to obtain a set of sampled video segments; specifically: Uniformly sample n video clips, that is, Among them, the four-dimensional tensor X i represents the i-th video clip, t is the number of video frames in the video clip; Step (3) constructs a video clip contribution score module, which takes as input a set of sampled video clips, and uses an adversarial perturbation sample generation module and a timing reconstruction module to output a contribution score for each video clip; specifically: (3-1). The video clip contribution score module consists of an adversarial perturbation sample generation module and a time series reconstruction calculation module; (3-2) Construct an adversarial perturbation sample generation module to measure the robustness of the student model to the video sample. The input is the video clip X i , for video segment X i Add perturbation η to get attack sample is a real number domain; where θ is the parameter of the action recognition model, y is the action label of the action recognition model, is the cross entropy loss, is the loss function for training the current student model, the subscript ce represents the cross entropy, ∈>0 is a hyperparameter, is the loss function for the original video clip X i The gradient of the original video clip X is , sign(·) is the sign function, when the input is positive, the result is 1, when the input is negative, the result is -1, and when the input is zero, the result is 0; i With attack sample X′ i Input the student action recognition model respectively, and output the adversarial robustness score of the video clip based on the difference in cross entropy loss between the two. The subscript adv indicates adversarial; (3-3) To measure the impact of the video clip on the student model, the input is the loss function of step (3-1) about the original video clip X i Gradient Perform l2 norm constraint and output gradient influence score The superscript gra indicates gradient; (3-4) Construct a time reconstruction calculation module to measure the strength of the temporal relationship of the video clip action and reflect the frequency of occurrence of an action in a certain temporal interval. The input is the video clip X i , from X i Randomly extract K frames for mask processing, that is, assign 1 to all elements in its feature matrix to obtain the masked video clip The sampled video frames are reconstructed using the multi-head attention autoencoder model pre-trained on the ImageNet-21K dataset. The multi-head attention autoencoder contains 2 multi-head self-attentions and 8 multi-layer perceptrons to obtain the reconstructed video clips. Then calculate the reconstruction score Characterizes the strength of the temporal relationship of actions in video clips, and the superscript tem indicates the score timing; (3-5) Calculate the final contribution of the video clip, specifically calculate the linear combination of the above three scores, and output the final contribution of the video clip G = αG adv +βG tem +(1-α-β)G gra , 0<α<0.5 and 0<β<0.5 are hyperparameters; Step (4) constructs a diversified few-sample selection module, with inputs of all video clips and their contribution scores, and uses the self-distillation module and the sample diversification module to output a small number of differentiated video samples; specifically: (4-1). The diversified few-shot selection module consists of a self-distillation module and a sample diversification module, where few-shot refers to a small number of labeled training samples and diversified refers to differentiated training samples; (4-2) Input is all video clips and their contribution scores G, and the clip with the highest contribution in each video Perform average pooling on its features in the time dimension to obtain the feature matrix of the i′th video in the k′th round The fragment with the highest impact factor The feature matrix of the jth frame in the k′th round is 0<j<t,0<i′<N,N is the total number of samples, and the output is the feature matrix of each video in the k′th round One round means the process of passing the entire training data set through the neural network once; (4-3) Construct a self-distillation module to alleviate the catastrophic forgetting problem in the process of few-sample distillation; the input is the k′th feature matrix of the i′th video and the characteristic matrix in k′-1 rounds The memory-enhanced feature matrix is ​​obtained by using the self-distillation strategy, namely 0<γ<1 is a hyperparameter; the output is the memory enhancement feature matrix F for each video i ″; (4-4). Construct a sample diversification module to select the videos required for training. The input is the final contribution score G of all training videos and the memory enhancement feature matrix F′ i′ , ,The method of solving the subset selection problem by using the determinant point process method ,selects a subset with diversity from the sample set. The more similar samples are less likely to be ,sampled at the same time; This module starts from a complete dataset with N videos In the process, we select Q differentiated video samples to form a set in, Represents the j′th original video, whose feature matrix set Represents the k′th original video, whose feature matrix set Will gather Each feature matrix F′ j′ , Flatten them into one-dimensional vectors respectively, The Q expanded one-dimensional vectors are concatenated into a comprehensive matrix Will The N expanded one-dimensional vectors are concatenated into a comprehensive matrix Utilizing parameters Measuring Collection The diversity of video samples, det(·) represents the determinant, is the identity matrix, and the superscript T indicates transpose; (4-5). Optimize the objective function through the greedy algorithm: Among them, G j′ For collection The final contribution score of the j′th sample in , 0<j′≤Q, argmax(×) represents the index used to find the maximum value, this objective function selects the optimal training video set 0<λ<1 is a hyperparameter, outputting the optimal training video set Step (5) inputting the differentiated video samples into the teacher network and the student network, calculating the cross entropy loss and focal loss between the real action category and the action category predicted by the student network, calculating the distance loss between the updated teacher action category probability and the student action category probability, and optimizing the student network using the stochastic gradient descent method; Step (6) Repeat steps (3) to (5) until the student model converges, uniformly sample the new video sample to obtain RGB frame image sequence and optical flow image sequence, input them into the student network to obtain the predicted action category probability, and select the category with the largest probability as the action category of the new video.

2. The action recognition method based on progressive few-sample knowledge distillation as claimed in claim 1, characterized in that: Step (1) specifically includes: (1-1). Sample the original video at a sampling rate of s frames per second Perform uniform sampling, t', w, h, c are the total duration, width, height, and number of channels of the video sample, 20≤s≤50, and obtain the frame stream image sequence Among them, the subscript I represents the image, I k represents the kth RGB three-channel image with width w and height h in the sequence, n' represents the number of images in the frame stream image sequence, and k represents the frame stream image index; (1-2) Based on the obtained frame flow image sequence, calculate the optical flow of 10 consecutive frames before and after to obtain the optical flow image sequence Among them, the subscript O represents the optical flow, O k represents the k-th optical flow image with width w, height h and channel dimension 10 corresponding to the frame flow image with index k; (1-3). Use two VGG-16 networks pre-trained on the ImageNet dataset to form a teacher encoder. The VGG-16 network is a VGG network with 16 layers. Use two VGG-16 networks to extract frame flow image and optical flow image features respectively; Use two VGG-11 networks with randomly initialized parameters to construct a student encoder to extract frame flow image and optical flow image features. The VGG-11 network is a VGG network with 11 layers. (1-4). Convert the frame stream image sequence Input into one of the VGG-16 networks of the teacher encoder to obtain the teacher spatial encoding features The superscript tea means teacher, and the subscript Represents Spatial, w1, h1 and c1 represent the width, height and number of channels of the encoded feature respectively; The optical flow image sequence Input the teacher encoder into another VGG-16 network to obtain the teacher temporal encoding features Subscript Indicates feature timing; Frame stream image sequence and optical flow image sequence Input the two VGG-11 networks of the student encoder at the same time to obtain the student spatial encoding features Temporal coding characteristics with students The superscript stu means student; (1-5). The spatial encoding features in the teacher and student models and the optical flow image sequence are concatenated along the channel dimension. The features of the concatenated teacher and student models are and The subscript con represents feature concatenation; the teacher model uses a three-layer converter to fuse the concatenated features, where each layer of converter contains a multi-head self-attention and a multi-layer perceptron, and the output is The subscript m indicates feature fusion. The student model uses three layers of three-dimensional large-core convolutional layers for feature extraction, where each large-core convolutional layer consists of n' large-core convolutional kernels. A layer normalization operation and two sets of 1×1 convolutions are used to map features to a higher dimension and map them back to the original channel dimension respectively. The output is (1-6). Finally, the teacher model and the fusion features of the student model The video actions are classified after a fully connected layer and a softmax(·) function.

3. The action recognition method based on progressive few-sample knowledge distillation as claimed in claim 2, characterized in that: Step (5) is specifically: (5-1). Collect the best training videos Input the teacher model and the student model. In order to alleviate the problem of balancing the number of samples of different classes, calculate the action category label of the video sample in the form of one-hot encoding. and the student action category probability vector The focal loss between The superscript C is the number of action categories. Wherein, the subscript fl represents the focal loss and p is p stu The probability value of the output of the correct action category in , α′>0 and γ′>0 are both hyper parameters, and the probability of the teacher action category The soft label of the teacher model is obtained by processing with the softmax(·) function The softened logical value of the "i" class is expressed as in, Represents the logical value of the i-th category of the teacher model 1≤i″≤C, the category index 1≤j″≤C, T represents the distillation temperature, exp(×) represents the exponential function, the distillation temperature of each video sample is positively correlated with the final contribution G of the sample T=μG, μ≥0 is a hyperparameter, and the distillation temperature is adaptively adjusted for different samples; calculate the probability of the teacher action category and the student action category probability vector p stu The relative entropy between As a distance loss, the subscript kl represents the divergence; (5-2). Calculate the final loss function ζ>0 is a hyperparameter, and the student network is optimized using stochastic gradient descent to perform adaptive progressive knowledge distillation using only a small number of labeled samples.

4. The action recognition method based on progressive few-sample knowledge distillation as claimed in claim 3, characterized in that: Step (6) is specifically: (6-1) Repeat steps (3) to (5) until the student model converges; (6-2). For new videos Uniform sampling to obtain RGB frame image sequence and optical flow image sequence The kth RGB three-channel image in the sequence with width w0 and height h0 The k-th optical flow image with width w0, height h0 and channel dimension 10 corresponding to the frame flow image with index k (6-3) The RGB frame image sequence obtained by sampling and optical flow image sequence Input the trained student model to obtain the new student action category probability vector Select the category with the highest probability As new video The action category of the new video is the largest category in the action category probability vector.

Images