Action recognition method based on multi-scale spatiotemporal feature distillation

Abstract

The present invention discloses an action recognition method based on multi-scale spatiotemporal feature distillation. The method of the present invention first samples a given original video to obtain a frame image sequence and an optical flow image sequence, and uses a convolutional neural network to obtain spatiotemporal coding features; then captures local features of different scales through convolution operations of different sizes to obtain multi-scale spatial features; at the same time, a self-attention mechanism is used to construct a long-term dependency relationship of temporal coding features to obtain long-term temporal features; then the multi-scale spatial features are fused with the long-term temporal features to obtain multi-scale spatiotemporal features for action classification; finally, the multi-scale spatial features are subjected to mask distillation to achieve the migration of task-related knowledge in the teacher model to the student model. The method of the present invention can not only enhance and fuse the spatiotemporal coding features of the video to obtain multi-scale spatiotemporal features, but also use mask feature distillation to achieve model lightweighting, improve the accuracy of action recognition and facilitate terminal deployment.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, in particular to the field of visual model compression and action recognition, and specifically relates to an action recognition method based on multi-scale spatiotemporal feature distillation. Background Art

[0002] Action recognition aims to accurately capture the action patterns in the video and give the category to which they belong. It is widely used in real-life scenarios in many fields such as motion analysis, safe driving, intelligent monitoring, human-computer interaction, film and television entertainment, etc. In recent years, due to the rapid popularization of deep learning technology in the field of computer vision, existing action recognition methods mainly use deep models represented by convolutional neural networks. However, the complex model structure and the huge number of parameters require more computing resources, which makes it difficult to deploy action recognition models on terminals with limited computing resources, such as embedded devices or edge devices. In response to this, researchers have proposed model compression technology, which aims to reduce the number of parameters of deep neural networks without losing accuracy as much as possible, and reduce storage and computing overhead. Therefore, it is of great practical need to use model compression technology for action recognition to achieve lightweight models for terminal deployment.

[0003] The key to action recognition is to accurately extract the spatiotemporal features of the original video to determine the category to which the action belongs. Existing action recognition methods are mainly divided into three categories: 1) Use a single three-dimensional convolutional neural network (3D Convolutional Neural Network, 3D CNN) to perform spatiotemporal modeling on the original video clips and extract spatiotemporal features; 2) Use two different convolutional neural networks (i.e., two-stream networks) to extract spatial features and temporal features respectively, and then perform feature fusion to obtain video spatiotemporal features; 3) First use a two-dimensional convolutional neural network to extract spatial features from the video frame by frame, and then use a recurrent neural network (Recurrent Neural Network, RNN) such as a long short-term memory network (Long Short-Term Memory, LSTM) to perform temporal modeling on the extracted single-frame spatial features to obtain video spatiotemporal features containing spatial information and temporal information.

[0004] As an important means to achieve model compression, knowledge distillation uses the knowledge of a large model to guide the training of a small model, so that the accuracy of the latter is close to that of the former, and eventually replaces the former for reasoning. Existing knowledge distillation methods can be divided into response distillation, feature distillation, and relationship distillation according to the type of knowledge. Response distillation uses the output of the teacher model as knowledge for the student model to learn; feature distillation uses the intermediate layer features of the teacher model as the learning target of the intermediate layer features of the student model to achieve the purpose of the student model learning from the teacher model. A common practice is to use each layer of the teacher model as the learning target of the corresponding layer features of the student model, and simply calculate the mean square error loss of the teacher features and the student features to optimize the student model; relationship distillation uses the relationship information between the layers of the teacher model as knowledge to guide the training of the student model.

[0005] The shortcomings of the above methods are mainly manifested in the following aspects: (1) Whether it is a two-dimensional or three-dimensional CNN, the fixed-size convolution kernel makes the receptive field of the model only capture local features of fixed scale, but the scale of the key spatial features of different videos is not fixed; (2) Using LSTM for temporal modeling can only establish short-term temporal dependencies. For actions that last longer, it is impossible to capture complete temporal information; (3) When using feature distillation to compress the action recognition model, the existing method assumes that all local regions of the teacher feature have the same importance to the recognition task, but it is not consistent with the actual situation. For example, human features and object features have a greater impact on recognition, while background features have a smaller impact; if the mean square error of each region is calculated indiscriminately, the student model is easily disturbed by relatively unimportant features. Based on the above considerations, there is an urgent need for an action recognition method that can fully explore and integrate the spatiotemporal features of the video and effectively learn the regional features related to the task. Summary of the invention

[0006] The purpose of the present invention is to address the shortcomings of the prior art and provide an action recognition method based on multi-scale spatiotemporal feature distillation, which can not only capture multi-scale local features to enhance the original spatial features and construct long-term temporal dependencies of short-term temporal features, but also achieve better feature distillation effects by allowing the student model to focus on feature areas that have a greater impact on the task, thereby realizing lightweight action recognition models.

[0007] The method of the present invention sequentially performs the following operations on a given video data set containing category labels:

[0008] Step (1) uniformly samples the original video to obtain a frame flow image sequence and an optical flow image sequence;

[0009] Step (2) constructing a teacher encoder and a student encoder, inputting a frame flow image sequence and an optical flow image sequence, and outputting teacher spatiotemporal coding features and student spatiotemporal coding features respectively;

[0010] Step (3) constructing a teacher and student multi-scale spatiotemporal feature fusion module, inputting the teacher's spatiotemporal coding features and the student's spatiotemporal coding features respectively, and outputting the corresponding multi-scale spatial features and multi-scale spatiotemporal features; the multi-scale spatiotemporal feature fusion module is composed of a temporal feature enhancement submodule, a spatial feature enhancement submodule and a three-dimensional convolutional layer;

[0011] Step (4) constructs a teacher classifier and a student classifier, with the corresponding multi-scale spatiotemporal features as input and the teacher and student action category probabilities as output; the teacher network consisting of a teacher encoder, a teacher multi-scale spatiotemporal feature fusion module and a teacher classifier is optimized using the stochastic gradient descent method; the teacher spatiotemporal encoding features of step (2) are sequentially input into the trained teacher multi-scale spatiotemporal feature fusion module and the teacher classifier to obtain updated teacher multi-scale spatial features and updated teacher action category probabilities;

[0012] Step (5) constructs a mask feature distillation module, the input of which is the updated teacher multi-scale spatial feature and the student multi-scale spatial feature of step (3), and the output is the mask spatial feature of the teacher and the student;

[0013] Step (6) calculates the cross entropy loss between the true label and the student action category probability, calculates the distance loss between the updated teacher action category probability and the student action category probability, calculates the distillation loss between the mask space features of the teacher and the student, and uses the stochastic gradient descent method to optimize the student network composed of the student encoder, the student multi-scale spatiotemporal feature fusion module, and the student classifier. The new video is passed through the student network of steps (1) to (4) in sequence to obtain the updated student action category probability, and the category with the largest probability is selected as the action category of the video.

[0014] Further, step (1) specifically comprises:

[0015] (1-1) Process the video V into a frame stream image sequence at a sampling rate of 10 to 20 frames per second The subscript I represents the frame stream, n represents the number of frame stream images, and I i represents an RGB three-channel image with index i, width w0, and height h0 in the sequence, i = 1, 2, ..., n, represents the field of real numbers;

[0016] The one-hot encoding action category corresponding to the video V is marked as y = [y1, y2, ..., y N ], such as the action category y where index l is l =1 means the video belongs to the action category, such as l =0 means the video does not belong to the action category, l = 1, 2, ..., N, where N represents the number of action categories.

[0017] (1-2) Use the Lucas-Kanad method to calculate the optical flow and obtain the optical flow image sequence Wherein, the subscript O represents the optical flow, n is the number of optical flow images which is the same as the number of frame flow images, and O i It represents the optical flow dual-channel image with index i, width w0, and height h0 in the sequence, and corresponds to the frame stream image according to the index i.

[0018] Furthermore, step (2) is specifically:

[0019] (2-1) Two ResNet34s form a teacher encoder, where the ResNet34 is a residual neural network (Residual Neural Network, ResNet) with 34 layers pre-trained with the ImageNet dataset; the two ResNet34s are used to process frame flow images and optical flow images respectively; two ResNet18s with randomly initialized parameters form a student encoder;

[0020] (2-2) Convert the frame stream image sequence and optical flow image sequence Input the two ResNet34 of the teacher encoder respectively to obtain the teacher spatial encoding features and teacher temporal coding features The superscript (T) represents the teacher network, the subscript s represents the spatial network, the subscript t represents the temporal network, w1, h1 and c1 represent the width, height and number of channels of the encoded features respectively;

[0021] Frame stream image sequence and optical flow image sequence Input the two ResNet18 of the student encoder respectively to obtain the student spatial encoding features and temporal encoding features The superscript (S) stands for Student Network.

[0022] Furthermore, step (3) is specifically:

[0023] (3-1) Use the self-attention mechanism to construct the teacher and student temporal feature enhancement submodules, and input the teacher and student temporal encoding features respectively and Obtain the reconstructed teacher and student temporal coding features and Reshape(·) is the dimension reconstruction function, While keeping the elements in the tensor unchanged, the tensor dimension is changed from Refactored to d s =w1·h1·c1 represents the spatial dimension;

[0024] The reconstructed teacher sequence encoding features are mapped using a fully connected layer and divided into corresponding query features, key features, and value features. The reconstructed student sequence encoding features are mapped using a fully connected layer and divided into corresponding query features, key features, and value features. Among them, FC(·) represents the fully connected layer, which changes the dimension from Mapping Subscript a represents attention, Split(·) is the partition function, and the dimension is The features are divided into three dimensions: Features;

[0025] Computing teacher attention features and student attention characteristics in The teacher and student attention features are mapped using the fully connected layer and dimensionally reconstructed to obtain the teacher's long-term time series features. and students' long-term temporal characteristics FC(·) transforms the dimension from Mapping Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to Output long-term time series features of teachers and students and

[0026] (3-2) Use two-dimensional convolution kernels of different sizes and nearest neighbor interpolation upsampling fusion operations to construct teacher and student spatial feature enhancement submodules, and input teacher and student spatial encoding features respectively and Compute local spatial features of teachers and students at different scales {h r (·) = ReLU(conv2d r (·))|r=3,5,7} means a two-dimensional convolution operation with a size of r×r and input and output channels of c1, and then activated by the ReLU function. and denote the corresponding local spatial features of the teacher and the student, respectively; w3 and h3, w5 and h5, w7 and h7 denote the width and height of the corresponding spatial features, respectively;

[0027] Calculate teacher and student upsampling fusion features of different scales in, Respectively represent the width and height as w r1 ×h r1 The spatial features are upsampled to w r2 ×h r2 The nearest neighbor interpolation upsampling operation of the spatial features, Respectively represent the teacher and student upsampling fusion features of different scales obtained by the corresponding upsampling operation and addition operation. and Output as teacher and student multi-scale spatial features; subscripts s3 and s5 represent two-dimensional convolutions of size 3×3 and 5×5;

[0028] (3-3) Use a three-dimensional convolutional neural network to construct a multi-scale spatiotemporal feature fusion module for teachers and students, with the input being the teacher's long-term temporal features output from steps (3-1) and (3-2) Multi-scale spatial features Long-term temporal characteristics of students Multi-scale spatial features The multi-scale spatiotemporal features of teachers and students are calculated using addition operations and 3D convolution operations respectively. and in, conv3d(·) means the convolution kernel size is n×w1×h1, the input channel is c1, and the output channel is d ts The 3D convolution operation, where the subscript ts stands for temporal-spatial, outputs the multi-scale temporal-spatial features of the teacher and the student. and

[0029] Furthermore, step (4) is specifically:

[0030] (4-1) Use the fully connected layer and softmax function to build the teacher classifier and student classifier, and input the multi-scale spatiotemporal features of the teacher and student and Calculate the probability distribution of teacher and student action categories and FC(·) transforms the dimension from Mapping Calculate the true label y and the probability distribution p of the teacher action category (T) The cross entropy loss between The subscript ce stands for CrossEntropy. represents the cross entropy between two probability distributions;

[0031] (4-2) Using the stochastic gradient descent method, according to the teacher cross entropy loss Optimize the teacher network consisting of the teacher encoder, the teacher multi-scale spatiotemporal feature fusion module, and the teacher classifier;

[0032] (4-3) The teacher's spatiotemporal coding features of step (2) and Input the trained teacher multi-scale spatiotemporal feature fusion module and teacher classifier in turn to obtain the updated teacher multi-scale spatial feature Updated teacher long-term temporal characteristics Updated teacher action category probabilities

[0033] Furthermore, step (5) is specifically:

[0034] (5-1) Construct a mask feature distillation module and randomly initialize the mask tensor m represents the number of masks that the mask tensor can generate; for the updated teacher multi-scale spatial features Perform dimensional reconstruction to obtain the multi-scale spatial features of the reconstructed teacher Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to

[0035] (5-2) Calculate teacher mask features * represents the Khatri-Rao product of the matrix (i.e., Kronecker in the column dimension); Perform dimension reconstruction to obtain the reconstructed teacher mask feature Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to exist The average value is calculated on the last dimension to obtain the teacher's average mask feature Will and Input the trained teacher multi-scale spatiotemporal feature fusion module to obtain the teacher multi-scale spatiotemporal mask feature

[0036] (5-3) The teacher's multi-scale spatiotemporal mask features Input the trained teacher classifier to obtain the probability distribution of the teacher mask action category Calculate the true label y and The cross entropy loss Calculate the diversity loss of the mask tensor E The subscript div stands for diversity. Using the stochastic gradient descent method, according to the mask loss Update the mask tensor E;

[0037] (5-4) Students’ multi-scale spatial features Reconstruct and obtain the multi-scale spatial features of the reconstructed students Reshape(·) changes the tensor dimension from Refactored to

[0038] (5-5) Use the updated mask tensor E' to construct a mask feature distillation module, with the input being the reconstructed teacher multi-scale spatial feature of step (5-1) And the reconstructed student multi-scale spatial features of step (5-4) Compute teacher and student mask features and as the output of this module.

[0039] Finally, step (6) is as follows:

[0040] (6-1) Calculate the probability distribution p of the student action category of the true label y and step (4-1) (S) The cross entropy loss between Calculate the updated teacher action category probability p of step (4-3) (T)' The probability distribution p of the student action category in step (4-1) (S) The KL (Kullback-Leibler) divergence between As the distance loss, where the subscript kl represents the KL divergence; calculate the teacher mask feature of step (5-5) and student mask features The L2 loss between As the distillation loss, the subscript fd represents feature distillation (FeatureDistillation);

[0041] (6-2) Construct a student network consisting of a student encoder, a student multi-scale spatiotemporal feature fusion module, and a student classifier, and its overall loss function is: Optimize the student network using stochastic gradient descent;

[0042] (6-3) Repeat steps (1-1) to (4-1) for the new video V'. The student network obtains a new student action category probability p (S)' , select the category with the largest probability l'=argmax(p (S)' ), that is, the category indexed as l' is used as the action category of the new video V'.

[0043] The method of the present invention has the following characteristics: 1) convolution operations of different sizes are used to capture spatial features of different scales, so that the video spatial features contain local spatial information of different sizes; 2) the long-term relationship of the original short-term temporal coding features is constructed through the self-attention mechanism, and the temporal dependency of actions lasting longer can be fully learned; 3) by performing mask distillation on task-related multi-scale spatial features, the student model can learn the teacher knowledge that has a greater impact on the task, thereby improving the accuracy of model compression. The method of the present invention is suitable for action recognition tasks with real-time requirements, and its beneficial effects include: 1) using convolution operations of different sizes to capture local spatial features of different scales, thereby enhancing the original spatial coding features, and being able to learn video spatial information with richer scales compared to using convolution operations of fixed sizes; 2) using the self-attention mechanism to globally model the short-term temporal features between video frames, and obtaining long-term temporal features, so that the video temporal features contain action information with a longer time span; 3) using mask feature distillation to transfer task-related knowledge in the teacher model to the student model, so that the student model with fewer parameters has an accuracy similar to that of the teacher model, and realizes the lightweight of the action recognition model. This lays the foundation for practical application scenarios such as safe driving, intelligent monitoring, and human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings.

[0046] like Figure 1 As shown in the figure, the action recognition method based on multi-scale spatiotemporal feature distillation first uniformly samples the original video to obtain RGB frame image sequences and optical flow image sequences, and uses the teacher and student encoders to extract the video spatiotemporal coding features; then uses the multi-scale spatiotemporal feature fusion module to capture the local spatial features of different scales in the spatial features through convolution kernels of different sizes and fuse them to enhance the original spatial coding features; at the same time, the self-attention mechanism is used to construct long-term dependencies for short-term temporal features to enhance the original temporal coding features; then three-dimensional convolution is used to obtain multi-scale spatiotemporal features for action category determination; in addition, the mask feature distillation module is used to generate corresponding mask features based on the multi-scale spatial features of the teacher and the student, so that the student model can focus on the task-related local areas in the teacher features, achieve better knowledge distillation effects, and improve the accuracy of model compression. The specific steps are as follows:

[0047] Step (1) uniformly samples the original video to obtain a frame flow image sequence and an optical flow image sequence; specifically:

[0048] (1-1) Process the video V into a frame stream image sequence at a sampling rate of 16 frames per second The subscript I represents the frame stream, n represents the number of frame stream images, and I i represents an RGB three-channel image with index i, width w0, and height h0 in the sequence, i = 1, 2, ..., n, represents the field of real numbers;

[0049] The action category corresponding to the video V in one-hot encoding is marked as y = [y1, y2, ..., y N ], such as the action category y where index l is l =1 means the video belongs to the action category, such as l =0 means the video does not belong to the action category, l = 1, 2, ..., N, where N represents the number of action categories.

[0050] (1-2) Use the Lucas-Kanad method to calculate the optical flow and obtain the optical flow image sequence The subscript O represents the optical flow, n is the number of optical flow images that is the same as the number of frame flow images, O i It represents the optical flow dual-channel image with index i, width w0, and height h0 in the sequence, and corresponds to the frame stream image according to the index i.

[0051] Step (2) constructs a teacher encoder and a student encoder, inputs a frame stream image sequence and an optical flow image sequence, and outputs the teacher spatiotemporal coding features and the student spatiotemporal coding features respectively; specifically:

[0052] (2-1) Two ResNet34s form a teacher encoder, where the ResNet34 is a residual neural network with 34 layers pre-trained with the ImageNet dataset; the two ResNet34s are used to process frame flow images and optical flow images respectively; two ResNet18s with randomly initialized parameters form a student encoder;

[0053] (2-2) Convert the frame stream image sequence and optical flow image sequence Input the two ResNet34 of the teacher encoder respectively to obtain the teacher spatial encoding features and teacher temporal coding features The superscript (T) denotes the teacher network, the subscript s denotes space, the subscript t denotes time, and w1, h1, and c1 denote the width, height, and number of channels of the encoded features, respectively;

[0054] Frame stream image sequence and optical flow image sequence Input the two ResNet18 of the student encoder respectively to obtain the student spatial encoding features and student temporal coding features The superscript (S) indicates the student network.

[0055] Step (3) constructs a teacher and student multi-scale spatiotemporal feature fusion module, inputs the teacher's spatiotemporal coding features and the student's spatiotemporal coding features respectively, and outputs the corresponding multi-scale spatial features and multi-scale spatiotemporal features; the multi-scale spatiotemporal feature fusion module is composed of a temporal feature enhancement submodule, a spatial feature enhancement submodule and a three-dimensional convolutional layer; specifically:

[0056] (3-1) Use the self-attention mechanism to construct the teacher temporal feature enhancement submodule and the student temporal feature enhancement submodule, and input the teacher temporal encoding feature respectively. and student temporal coding features Obtain the reconstructed teacher and student temporal coding features and Reshape(·) is the dimension reconstruction function, While keeping the elements in the tensor unchanged, the tensor dimension is changed from Refactored to Space dimension d s =w1·h1·c1;

[0057] The reconstructed teacher sequence encoding features are mapped using a fully connected layer and divided into corresponding query features, key features, and value features. The reconstructed student sequence encoding features are mapped using a fully connected layer and divided into corresponding query features, key features, and value features. Among them, FC(·) represents the fully connected layer, which changes the dimension from Mapping Subscript a represents attention, Split(·) is the partitioning function, and the dimension is The features are divided into three dimensions: Features;

[0058] Computing teacher attention features and student attention characteristics in The teacher and student attention features are mapped using the fully connected layer and dimensionally reconstructed to obtain the teacher's long-term time series features. and students' long-term temporal characteristics FC(·) transforms the dimension from Mapping Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to Output teacher long-term time series features and students' long-term temporal characteristics

[0059] (3-2) Using two-dimensional convolution kernels of different sizes and nearest neighbor interpolation upsampling fusion operations, we construct teacher and student spatial feature enhancement submodules, and input the teacher spatial encoding features respectively. and student spatial encoding features Compute local spatial features of teachers and students at different scales {h r (·) = ReLU(conv2d r (·))|r=3,5,7} means a two-dimensional convolution operation with a size of r×r and input and output channels of c1, and then activated by the ReLU function. and denote the corresponding local spatial features of the teacher and the student, respectively; w3 and h3, w5 and h5, w7 and h7 denote the width and height of the corresponding spatial features, respectively;

[0060] Calculate teacher and student upsampling fusion features of different scales Among them, {U r1→r2 (·)} (r1,r2)＝{(7,5),(5,3),(3,1)} Indicates that the width and height are w r1 ×h r1 The spatial features are upsampled to w r2 ×h r2 The nearest neighbor interpolation upsampling operation of the spatial features, Respectively represent the up-sampling fusion features of teachers and students of different scales obtained through the corresponding up-sampling operation and addition operation. and Output as teacher and student multi-scale spatial features; subscripts s3 and s5 represent two-dimensional convolutions of size 3×3 and 5×5;

[0061] (3-3) Use three-dimensional convolutional neural network to construct teacher multi-scale spatiotemporal feature fusion module and student multi-scale spatiotemporal feature fusion module, the input of which is teacher long-term time series feature Multi-scale spatial features Long-term temporal characteristics of students Multi-scale spatial features The teacher is calculated by adding and 3D convolution respectively. and students' multi-scale spatiotemporal characteristics and conv3d(·) means the convolution kernel size is n×w1×h1, the input channel is c1, and the output channel is d ts The three-dimensional convolution operation, subscript ts represents time and space, outputs the multi-scale spatiotemporal features of the teacher and students' multi-scale spatiotemporal characteristics

[0062] Step (4) constructs a teacher classifier and a student classifier, with the corresponding multi-scale spatiotemporal features as input and the teacher and student action category probabilities as output, respectively. The teacher network consisting of the teacher encoder, the teacher multi-scale spatiotemporal feature fusion module and the teacher classifier is optimized using the stochastic gradient descent method; the teacher spatiotemporal encoding features are sequentially input into the trained teacher multi-scale spatiotemporal feature fusion module and the teacher classifier to obtain updated teacher multi-scale spatial features and updated teacher action category probabilities; specifically:

[0063] (4-1) Use the fully connected layer and softmax function to construct the teacher classifier and the student classifier, and input the multi-scale spatiotemporal features of the teacher and the student respectively. and Calculate the probability distribution of teacher action categories and the probability distribution of student action categories FC(·) transforms the dimension from Mapping Calculate the true label y and the probability distribution p of the teacher action category (T) The cross entropy loss between The subscript ce represents cross entropy, represents the cross entropy between two probability distributions;

[0064] (4-2) Using the stochastic gradient descent method, according to the teacher cross entropy loss Optimize the teacher network consisting of the teacher encoder, the teacher multi-scale spatiotemporal feature fusion module, and the teacher classifier;

[0065] (4-3) Encoding the teacher space features and teacher temporal coding features Input the trained teacher multi-scale spatiotemporal feature fusion module and teacher classifier in turn to obtain the updated teacher multi-scale spatial feature Updated teacher long-term temporal characteristics Updated teacher action category probabilities

[0066] Step (5) constructs a mask feature distillation module, the input of which is the updated teacher multi-scale spatial features and student multi-scale spatial features, and the output is the teacher and student mask spatial features; specifically:

[0067] (5-1) Construct a mask feature distillation module and randomly initialize the mask tensor m represents the number of masks that the mask tensor can generate; for the updated teacher multi-scale spatial features Perform dimensional reconstruction to obtain the multi-scale spatial features of the reconstructed teacher Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to

[0068] (5-2) Calculate teacher mask features * represents the Khatri-Rao product of matrices; Perform dimension reconstruction to obtain the reconstructed teacher mask feature Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to exist The average value is calculated on the last dimension to obtain the teacher's average mask feature Will and Input the trained teacher multi-scale spatiotemporal feature fusion module to obtain the teacher multi-scale spatiotemporal mask feature

[0069] (5-3) The teacher's multi-scale spatiotemporal mask features Input the trained teacher classifier to obtain the probability distribution of the teacher mask action category Calculate the true label y and The cross entropy loss Calculate the diversity loss of the mask tensor E The subscript div represents diversity; using the stochastic gradient descent method, according to the mask loss Update the mask tensor E;

[0070] (5-4) Students’ multi-scale spatial features Reconstruct and obtain the multi-scale spatial features of the reconstructed students Reshape(·) reshapes the tensor from 1 to 2 while keeping the elements in the tensor unchanged. Refactored to

[0071] (5-5) Use the updated mask tensor E' to construct a mask feature distillation module, and the input is the reconstructed teacher multi-scale spatial feature and reconstructing students' multi-scale spatial features Calculate teacher mask features and student mask features as the output of this module.

[0072] Step (6) calculates the cross entropy loss between the true label and the student action category probability, calculates the distance loss between the updated teacher action category probability and the student action category probability, calculates the distillation loss between the mask space features of the teacher and the student, and uses the stochastic gradient descent method to optimize the student network composed of the student encoder, the student multi-scale spatiotemporal feature fusion module, and the student classifier. The new video is sequentially passed through the student network of steps (1) to (4) to obtain the updated student action category probability, and the category with the largest probability is selected as the action category of the video; specifically:

[0073] (6-1) Calculate the true label y and the probability distribution p of the student action category (S) The cross entropy loss between Calculate the updated teacher action category probability p (T)' and the probability distribution p of student action categories (S) The KL divergence between As the distance loss, the subscript kl represents the KL divergence; the teacher mask feature is calculated and student mask features The L2 loss between As the distillation loss, the subscript fd represents feature distillation;

[0074] (6-2) Construct a student network consisting of a student encoder, a student multi-scale spatiotemporal feature fusion module, and a student classifier, and its overall loss function is: Optimize the student network using stochastic gradient descent;

[0075] (6-3) Repeat steps (1-1) to (4-1) for the new video V'. The student network obtains a new student action category probability p (S)' , select the category with the largest probability l'=argmax(p (S)' ), that is, the category indexed as l' is used as the action category of the new video V'.

[0076] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An action recognition method based on multi-scale spatiotemporal feature distillation, characterized in that: For a given set of video data with category labels, perform the following operations in sequence: Step (1) uniformly samples the original video to obtain a frame flow image sequence and an optical flow image sequence; Step (2) constructing a teacher encoder and a student encoder, inputting a frame flow image sequence and an optical flow image sequence, and outputting teacher spatiotemporal coding features and student spatiotemporal coding features respectively; Step (3) constructing a teacher multi-scale spatiotemporal feature fusion module and a student multi-scale spatiotemporal feature fusion module; the multi-scale spatiotemporal feature fusion module is composed of a temporal feature enhancement submodule, a spatial feature enhancement submodule and a three-dimensional convolutional layer; The input of the temporal feature enhancement submodule is respectively the teacher's temporal coding feature and the student's temporal coding feature, and the output is respectively the teacher's long-term temporal feature and the student's long-term temporal feature; The spatial feature enhancement submodule inputs are teacher spatial coding features and student spatial coding features, and outputs are teacher multi-scale spatial features and student multi-scale spatial features; The three-dimensional convolutional layer inputs are respectively the teacher's long-term temporal features and the teacher's multi-scale spatial features, as well as the student's long-term temporal features and the student's multi-scale spatial features, and the outputs are respectively the teacher's multi-scale spatiotemporal features and the student's multi-scale spatiotemporal features; Step (4) constructing a teacher classifier and a student classifier, with the corresponding multi-scale spatiotemporal features as input and the teacher and student action category probabilities as output, respectively, and using the stochastic gradient descent method to optimize the teacher network consisting of the teacher encoder, the teacher multi-scale spatiotemporal feature fusion module, and the teacher classifier; inputting the teacher spatiotemporal encoding features into the trained teacher multi-scale spatiotemporal feature fusion module and the teacher classifier in turn to obtain updated teacher multi-scale spatial features and updated teacher action category probabilities; Step (5) constructs a mask feature distillation module, the input of which is the updated teacher multi-scale spatial features and student multi-scale spatial features, and the output is the teacher and student mask spatial features; Step (6) calculates the cross entropy loss between the true label and the student action category probability, calculates the distance loss between the updated teacher action category probability and the student action category probability, calculates the distillation loss between the mask space features of the teacher and the student, and uses the stochastic gradient descent method to optimize the student network composed of the student encoder, the student multi-scale spatiotemporal feature fusion module, and the student classifier. The new video is passed through the student network of steps (1) to (4) in sequence to obtain the updated student action category probability, and the category with the largest probability is selected as the action category of the video.

2. The action recognition method based on multi-scale spatiotemporal feature distillation according to claim 1, characterized in that: Step (1) specifically includes: (1-1) Process the video V into a frame stream image sequence at a sampling rate of 10 to 20 frames per second The subscript I represents the frame stream, n represents the number of frame stream images, and I i Represents an RGB three-channel image with index i, width w0, and height h0 in the sequence. The action category corresponding to the video V in one-hot encoding is marked as y = [y1, y2, ..., y N ], such as the action category y where index l is l =1 means the video belongs to the action category, such as l =0 means the video does not belong to the action category, l = 1, 2, ..., N, where N represents the number of action categories. (1-2) Use the Lucas-Kanad method to calculate the optical flow and obtain the optical flow image sequence The subscript O represents the optical flow, n is the number of optical flow images that is the same as the number of frame flow images, O i It represents the optical flow dual-channel image with index i, width w0, and height h0 in the sequence, and corresponds to the frame stream image according to the index i.

3. The action recognition method based on multi-scale spatiotemporal feature distillation according to claim 2, characterized in that: Step (2) is specifically: (2-1) Two ResNet34s form a teacher encoder, where the ResNet34 is a residual neural network with 34 layers pre-trained with the ImageNet dataset; the two ResNet34s are used to process frame flow images and optical flow images respectively; two ResNet18s with randomly initialized parameters form a student encoder; (2-2) Convert the frame stream image sequence and optical flow image sequence Input the two ResNet34 of the teacher encoder respectively to obtain the teacher spatial encoding features and teacher temporal coding features Frame stream image sequence and optical flow image sequence Input the two ResNet18 of the student encoder respectively to obtain the student spatial encoding features and student temporal coding features 4. The action recognition method based on multi-scale spatiotemporal feature distillation according to claim 3, characterized in that: Step (3) is specifically: (3-1) Use the self-attention mechanism to construct the teacher temporal feature enhancement submodule and the student temporal feature enhancement submodule, and input the teacher temporal encoding feature respectively. and student temporal coding features Obtain the reconstructed teacher and student temporal coding features and Reshape(×) is the dimension reconstruction function, While keeping the elements in the tensor unchanged, the tensor dimension is changed from Refactored to represents the real number field, with spatial dimension d s =w1·h1·c1; the superscript (T) represents the teacher network, the superscript (S) represents the student network, the subscript t represents the time series, w1, h1 and c1 represent the width, height and number of channels of the encoding feature respectively; The reconstructed teacher time series encoding features are mapped using a fully connected layer and divided into corresponding query features, key features, and value features. The reconstructed student sequence encoding features are mapped using a fully connected layer and divided into corresponding query features, key features, and value features. Among them, FC(×) represents the fully connected layer, which changes the dimension from Mapping Subscript a represents attention, Split(×) is the partitioning function, and the dimension is The features are divided into three dimensions: Features; Computing teacher attention features and student attention characteristics in The teacher and student attention features are mapped using the fully connected layer and dimensionally reconstructed to obtain the teacher's long-term time series features. and students' long-term temporal characteristics FC(×) transforms the dimension from Mapping Reshape(×) reshapes the tensor from x to y while keeping the elements in the tensor unchanged. Refactored to Output teacher long-term time series features and students' long-term temporal characteristics (3-2) Using two-dimensional convolution kernels of different sizes and nearest neighbor interpolation upsampling fusion operations, we construct teacher and student spatial feature enhancement submodules, and input the teacher spatial encoding features respectively. and student spatial encoding features Compute local spatial features of teachers and students at different scales Among them, {h r (·) = ReLU(conv2d r (·))|r=3,5,7} means a two-dimensional convolution operation with a size of r×r and input and output channels of c1, and then activated by the ReLU function. and denote the corresponding local spatial features of the teacher and the student, respectively; w3 and h3, w5 and h5, w7 and h7 denote the width and height of the corresponding spatial features, respectively; Calculate teacher and student upsampling fusion features of different scales Among them, {U r1→r2 (·)} (r1,r2)＝{(7,5),(5,3),(3,1)} Indicates that the width and height are w r1 ×h r1 The spatial features are upsampled to w r2 ×h r2 The nearest neighbor interpolation upsampling operation of the spatial features, Respectively represent the up-sampling fusion features of teachers and students of different scales obtained through the corresponding up-sampling operation and addition operation. and As the multi-scale spatial features of teachers and students, the subscript s represents space; subscripts s3 and s5 represent two-dimensional convolutions of size 3×3 and 5×5; (3-3) Use three-dimensional convolutional neural network to construct teacher multi-scale spatiotemporal feature fusion module and student multi-scale spatiotemporal feature fusion module, the input of which is teacher long-term time series feature Multi-scale spatial features Long-term temporal characteristics of students Multi-scale spatial features The multi-scale spatiotemporal features of the teacher are calculated using addition operations and three-dimensional convolution operations respectively. and students' multi-scale spatiotemporal characteristics and conv3d(·) means the convolution kernel size is n×w1×h1, the input channel is c1, and the output channel is d ts The three-dimensional convolution operation, subscript ts represents time and space, outputs the multi-scale spatiotemporal features of the teacher and students' multi-scale spatiotemporal characteristics 5. The action recognition method based on multi-scale spatiotemporal feature distillation according to claim 4, characterized in that: Step (4) is specifically: (4-1) Use the fully connected layer and softmax function to construct the teacher classifier and the student classifier, and input the multi-scale spatiotemporal features of the teacher and the student respectively. and Calculate the probability distribution of teacher action categories and the probability distribution of student action categories FC(×) transforms the dimension from Mapping Calculate the true label y and the probability distribution p of the teacher action category (T) The cross entropy loss between The subscript ce represents cross entropy, represents the cross entropy between two probability distributions; (4-2) Using the stochastic gradient descent method, according to the teacher cross entropy loss Optimize the teacher network consisting of the teacher encoder, the teacher multi-scale spatiotemporal feature fusion module, and the teacher classifier; (4-3) Encoding features of teacher space and teacher temporal coding features Input the trained teacher multi-scale spatiotemporal feature fusion module and teacher classifier in turn to obtain the updated teacher multi-scale spatial feature Updated teacher long-term time series features Updated teacher action category probabilities 6. The action recognition method based on multi-scale spatiotemporal feature distillation according to claim 5, characterized in that: Step (5) is specifically: (5-1) Construct a mask feature distillation module and randomly initialize the mask tensor m represents the number of masks that the mask tensor can generate; for the updated teacher multi-scale spatial features Perform dimensional reconstruction to obtain the multi-scale spatial features of the reconstructed teacher Reshape(×) reshapes the tensor from x to y while keeping the elements in the tensor unchanged. Refactored to (5-2) Calculate teacher mask features * represents the Khatri-Rao product of matrices; Perform dimension reconstruction to obtain the reconstructed teacher mask feature Reshape(×) reshapes the tensor from x to y while keeping the elements in the tensor unchanged. Refactored to exist The average value is calculated on the last dimension to obtain the teacher's average mask feature. Will and Input the trained teacher multi-scale spatiotemporal feature fusion module to obtain the teacher multi-scale spatiotemporal mask feature (5-3) The teacher's multi-scale spatiotemporal mask features Input the trained teacher classifier to obtain the probability distribution of the teacher mask action category Calculate the true label y and The cross entropy loss Calculate the diversity loss of the mask tensor E The subscript div indicates diversity; Using stochastic gradient descent, according to the mask loss Update the mask tensor E; (5-4) Students’ multi-scale spatial features Reconstruct and obtain the multi-scale spatial features of the reconstructed students Reshape(×) reshapes the tensor from x to y while keeping the elements in the tensor unchanged. Refactored to (5-5) Use the updated mask tensor E' to construct a mask feature distillation module, and the input is the reconstructed teacher multi-scale spatial feature and reconstructing students' multi-scale spatial features Calculate teacher mask features and student mask features as the output of this module.

7. The action recognition method based on multi-scale spatiotemporal feature distillation according to claim 6, characterized in that: Step (6) is specifically: (6-1) Calculate the true label y and the probability distribution p of the student action category (S) The cross entropy loss between Calculate the updated teacher action category probability p (T)' and the probability distribution p of student action categories (S) The KL divergence between As the distance loss, the subscript kl represents the KL divergence; the teacher mask feature is calculated and student mask features The L2 loss between As the distillation loss, the subscript fd represents feature distillation; (6-2) Construct a student network consisting of a student encoder, a student multi-scale spatiotemporal feature fusion module, and a student classifier, and its overall loss function is: Optimize the student network using stochastic gradient descent; (6-3) Repeat steps (1-1) to (4-1) for the new video V'. The student network obtains a new student action category probability p (S) ', select the category with the highest probability l' = argmax(p (S)' ), that is, the category indexed as l' is used as the action category of the new video V'.

Images