Method and system for action recognition based on video

Abstract

The present disclosure relates to a method for performing action recognition based on a video, comprising: using a kernel cross-correlator to calculate the position of a tracking target in each frame based on data of consecutive frames included in the video; and using a long short-term memory model LSTM to calculate motion features in the video based on the data of each frame and the position of the tracking target in each frame. The present disclosure also relates to a system and a storage medium for performing action recognition based on a video.

Description

Technical Field

[0001] The present invention relates to a method and system for performing action recognition based on video. Background Art

[0002] In recent years, human action recognition has attracted much attention due to its potential applications in automatic video analysis, video surveillance, sports event analysis, and virtual reality. The surge in multimedia devices on the Internet of Things (IoT) has generated unprecedented amounts of data, and the large amount of video data has put forward increasingly higher requirements for human action recognition. Although static image classification has achieved great success, human action recognition in real videos such as movies, sports videos, and daily life consumption videos is still a problem. The problem is caused by some inherent characteristics of action videos, such as intra-class variation, occlusion, viewpoint change, background noise, motion speed, and personnel differences.

[0003] With the remarkable success of deep neural networks in the field of image recognition, human action recognition has become the second largest application of deep networks in the field of computer vision. There are a large number of published studies using various deep neural networks to improve the accuracy of action recognition in videos. The classic two-stream deep action recognition network usually consists of a spatial stream and a temporal stream, such as Figure 1 As shown in the figure. The spatial stream extracts appearance features, and the temporal stream uses an optical flow estimator to extract motion information. Each stream uses a deep convolutional network connected to a softmax classifier, and finally the classification results of the two streams are fused. This dual-stream recognition model is based on biological inspiration, that is, there are two different visual memory modules in the human brain, which are responsible for analyzing static images and motion information respectively. However, this method ignores the fact that the brain areas involved in motion visual analysis are also involved in processing the dynamic information implicit in static images; in addition, these recognition methods have a common disadvantage: additional motion feature information needs to be manually extracted during the learning process. This additional information has high computational complexity and faces limitations when applied to real-time problems and large-scale video data. Summary of the invention

[0004] A brief overview of the disclosure is given below in order to provide a basic understanding of some aspects of the disclosure. However, it should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify the key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to give some concepts of the disclosure in a simplified form as a prelude to a more detailed description given later.

[0005] According to a first aspect of the present disclosure, a method for action recognition based on video is provided, comprising: based on data of consecutive frames included in the video, using a kernel cross-correlator to calculate the position of a tracked target in each frame; and based on the data of each frame and the position of the tracked target in each frame, using a long short-term memory model LSTM to calculate motion features in the video.

[0006] According to a second aspect of the present disclosure, a system for action recognition based on video is provided, comprising: a spatial feature extraction module, configured to process each frame included in the video to obtain a feature map of each frame; a motion flow estimation module, configured to use a kernel cross-correlator to calculate the position of a tracked target in each frame based on the feature maps of consecutive frames; a motion feature estimation module, configured to use a long short-term memory model LSTM to calculate motion features in the video based on the feature map of each frame and the position of the tracked target in each frame; and an action classification module, configured to use a softmax classifier to perform action classification on the calculated motion features.

[0007] According to a third aspect of the present disclosure, a system for performing action recognition based on video is provided, comprising: a memory having instructions stored thereon; and a processor configured to execute the instructions stored on the memory to execute the method according to the above aspects of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which includes computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors execute the method according to the above aspects of the present disclosure.

[0009] In view of the problems of independent extraction of spatial and temporal information and the need for additional manual feature extraction in the current dual-stream recognition model, this paper proposes a spatiotemporal video recognition network that integrates a motion-aware long short-term memory module. The long short-term memory module is mainly used for simultaneous motion estimation and spatiotemporal feature extraction. Motion estimation is based on the kernelized cross-correlation method and does not require pre-estimation of the optical flow network to understand the temporal behavior in the video. Since the network is designed in an end-to-end manner, no manual feature extraction module is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0011] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0012] Figure 1 Schematic diagram of the existing two-stream deep action recognition network.

[0013] Figure 2 It is a structural diagram of a system for performing action recognition based on video according to an embodiment of the present disclosure.

[0014] Figure 3 The present invention is a schematic diagram of the principles of a motion flow estimation module and a motion feature estimation module in a system for performing action recognition based on video according to an embodiment of the present disclosure.

[0015] Figure 4 A block diagram of an exemplary hardware configuration capable of implementing an embodiment according to the present disclosure. DETAILED DESCRIPTION

[0016] The following detailed description is made with reference to the accompanying drawings, and the following detailed description is provided to assist in a comprehensive understanding of the various exemplary embodiments of the present disclosure. The following description includes various details to assist in understanding, but these details are considered to be examples only and not to limit the present disclosure, which is defined by the appended claims and their equivalents. The words and phrases used in the following description are only used to enable a clear and consistent understanding of the present disclosure. In addition, for the sake of clarity and brevity, descriptions of well-known structures, functions, and configurations may be omitted. Those of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0017] In order to solve the problem that LSTM and ConvLSTM networks cannot extract rich temporal information, the present disclosure proposes to combine the motion flow estimation module and the motion feature estimation module (referred to as "motion-aware ConvLSTM spatiotemporal layer" in this paper). Figure 3 As shown in the figure, motion flow estimation is performed based on the kernel cross-correlator and the estimated motion information is used as the input of the ConvLSTM unit. This method only relies on the RGB image of the video, avoids the computational complexity of calculating the optical flow, and can effectively enhance the LSTM unit with motion information to better represent the video information.

[0018] Figure 2: is a structural diagram of a system 100 for performing action recognition based on video according to an embodiment of the present disclosure. The system 100 for performing action recognition based on video according to an embodiment of the present disclosure includes a spatial feature extraction module 110, a motion flow estimation module 120, a motion feature estimation module 130 and an action classification module 140. Among them, the spatial feature extraction module 110 uses one or more convolutional layers to process each frame included in the video. For example, the first three blocks of VGG16 can be used as convolutional layers to obtain a feature map of each frame, that is, to extract the basic spatial features of each frame. The spatial feature extraction module 110 only extracts basic spatial features from the video information, and these features have sufficient spatial and motion data to be passed to subsequent processing modules. The convolutional layer can be pre-trained on ImageNet and fine-tuned on the corresponding data set.

[0019] The motion flow estimation module 120 uses a kernel cross-correlator to calculate the position of the tracked target in each frame based on the feature maps of the continuous frames. Specifically, in the motion flow estimation module 120, one or more data blocks are determined in the data of the current frame with the calculated position of the tracked target in the previous frame as the center; the cross-correlation coefficient matrix calculated based on the data of the previous frame is used to perform cross-correlation filtering on the one or more data blocks to calculate the response matrix of the cross-correlation filtering; and the position of the tracked target in the current frame is determined according to the response matrix. In one embodiment, the kernel function of the kernel cross-correlator is a Gaussian kernel function, and the response matrix used for the training initialization of the kernel cross-correlator is a matrix in which the elements at the center position are 1 and the other elements are 0.

[0020] The motion flow estimation module 120 is based on the related action recognizer kernel cross correlator (KCC) by introducing kernel function, which extends the linear correlation to nonlinear space, has stronger robustness to signal noise and distortion, and the connection with the existing network shows that KCC provides a unified solution for the related filter. In addition, KCC is applicable to any kernel function, is not limited to the loop structure on the training data, and can predict affine transformations with custom properties. KCC eliminates the direct calculation of the kernel vector by using the fast Fourier transform (FFT), thereby obtaining better performance at a reasonable computational cost.

[0021] The implementation of the motion flow estimation module 120 in a specific embodiment is described below.

[0022] Define x as the input of the kernel cross-correlator, is the desired output of the core cross-correlator, is the kernel matrix. KCC is defined as:

[0023] g=κ z (x)*h (Formula 1)

[0024] Where g is the output of the correlator (i.e., the response matrix of the cross-correlation filter), h is the core cross-correlator (i.e., the cross-correlation coefficient matrix), and the kernel is set to the radial basis kernel. In order to reduce the computational complexity of the correlation operator, the equation is converted to the frequency domain, where the correlation is calculated by the conjugate multiplication of each element. Therefore, KCC can be rewritten as:

[0025]

[0026] Among them, Θ is element-by-element multiplication, h * Expressed as the conjugate of h, Symbolic to frequency domain transformation, kernel function κ z (x) uses a Gaussian kernel function, as shown in Formula 3.

[0027]

[0028] According to Formula 2, KCC can be trained by minimizing the sum of squared errors. The closed-form solution of is derived as:

[0029]

[0030] Among them, z is the training data, (n x ,n y ) is the size of the response matrix g. In this embodiment, center position tracking is adopted, so the response matrix g of the training target is set to a matrix in which the element at the center position is 1 and the other elements are 0, that is:

[0031]

[0032] By training the filter, we can calculate the transformation (t x ,t y ):

[0033]

[0034] in represents the inverse Fourier transform, is the center position of the image, (t x ,t y ) measures the amount of translation relative to the center, i.e., estimates the position translation of the tracked target.

[0035] Since the motion in different data blocks is usually different, this disclosure introduces a block-wise correlation flow matrix M, which estimates the transformation of each element in the input data for the data blocks around the position of the tracked target. t With x t-1 As input at time t and t-1 respectively, each element of the block-by-block correlation flow matrix M is defined as:

[0036]

[0037] in, and The calculations are as follows:

[0038]

[0039]

[0040] The motion feature estimation module 130 (also referred to herein as a “ConvLSTM unit”) is based on the feature map data X of each frame. t And the position M of the tracked target in each frame t To calculate the input gate function I of LSTM t , forget gate function F t , output gate function O t and candidate storage units Therefore, the long short-term memory model LSTM is used to calculate the motion features in the video.

[0041] The block-wise correlation flow matrix M is used as an additional information source inside the ConvLSTM unit. Figure 3 The internal structure of the proposed motion-aware ConvLSTM unit is shown. Compared with the ConvLSTM unit, there is an additional input, namely the input data of the previous time. The additional input can be used to learn the corresponding kernelized cross-correlation filter, and the filter is used for the current input to extract motion information.

[0042] The corresponding formula of the motion perception ConvLSTM unit is as follows:

[0043] I t =σ(W xi *X t +W hi *H t-1 +W mi *M t +b i )

[0044] F t =σ(W xf *X t +W hf *H t-1 +W mf *M t +b f )

[0045] O t =σ(W xo *X t +Who *H t-1 +W mo *M t +b o )

[0046]

[0047]

[0048] H t =O t Θtanh(C t ) (Formula 10)

[0049] Among them, the subscript t indicates that the parameter is about the current frame, the subscript t-1 indicates that the parameter is about the previous frame, the subscript i indicates that the parameter is about the input gate function, the subscript f indicates that the parameter is about the forget gate function, the subscript o indicates that the parameter is about the output gate function, and the subscript c indicates that the parameter is about the candidate storage unit C t , σ is the activation function, * is the convolution operation, H is the hidden state, W x , W h , W m is a two-dimensional convolution kernel, and b is a training parameter.

[0050] The action classification module 140 uses a softmax classifier to perform action classification on the calculated motion features. The present disclosure uses the proposed motion-aware ConvLSTM spatiotemporal layer to process the basic spatial features in the video frame and jointly extract space, motion, and long-term temporal dependencies. The convolution kernel size of the motion-aware ConvLSTM layer is 3×3. The size of the image block used to calculate the cross-correlation is also 7×7. Finally, the classifier (usually softmax) labels these features with the corresponding action class. The training loss uses categorical cross entropy. Since the Adam algorithm converges quickly, Adam is selected as the optimization method.

[0051] In a specific example, given a video V∈R t×w×h×ch , where t is the number of frames, w is the width, h is the height, and ch is the number of channels. The video segments are randomly sampled, and the starting point and step length are random. The extracted video slice is V clip ∈R t′×w′×h′×ch , where t'<t is the selected frame length, w' and h' are the sizes of the video slices, considering the sequence of the input video Video Slicing Select as:

[0052] V clip ={f i |i 1 ∈{1,2,…,tt′×s},ij =i j-1 +s} (Formula 11)

[0053] Among them, i 1 is the starting index of the slice frame, and s is the step size. Both parameters are randomly selected. The present invention extracts 30 frames of video clips from the video. In order to reduce overfitting, common spatial data amplification, random cropping and horizontal flipping are performed on each frame. In addition, enhancement is applied in the time dimension to cover multiple clips of the same video with different motion speeds and different action cycles.

[0054] exist Figure 2 In the network architecture of the video-based action recognition system shown, the present disclosure uses the first three blocks of VGG16 as convolutional layers to extract only basic spatial features from the video information, which have sufficient spatial and motion data to pass to the motion-aware ConvLSTM layer. The convolutional layers are pre-trained on ImageNet and fine-tuned on the corresponding datasets. The convolution kernel size of the motion-aware ConvLSTM layer is 3×3. The size of the image patch used to calculate the cross-correlation is also 7×7. Classification cross entropy is used as the training loss. Adam is selected as the optimization method because of its fast convergence speed.

[0055] The present disclosure is based on two evaluation metrics, per-slice accuracy and video accuracy. To measure the first metric, the class label with the largest softmax output is assigned to each slice, and the number of correctly assigned labels across all slices is measured. To calculate the second metric (i.e., video accuracy), the softmax scores of each video segment are averaged and the maximum of these scores is used as the class label. Then, the video accuracy is measured by the number of correctly assigned labels across all videos.

[0056] Comparing the motion-aware ConvLSTM two-stream convolutional network based on kernel cross-correlator proposed in this disclosure with the existing behavior recognition network, the LSTM and ConvLSTM networks can achieve accuracies of 80.5% and 85.2% respectively on the UCF101 dataset, and the network architecture proposed in this disclosure can achieve a video recognition accuracy of 94.5%.

[0057] Figure 4 2 is a block diagram of an exemplary hardware configuration 200 capable of implementing an embodiment according to the present disclosure. The hardware configuration 200 is an example of a hardware device to which the above aspects of the present disclosure can be applied. The hardware configuration 200 can be any machine configured to perform processing and / or computing. The hardware configuration 200 can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal data assistant (PDA), a smart phone, a car computer, or a combination thereof.

[0058] like Figure 4 As shown, hardware configuration 200 may include one or more elements that can be connected or communicated with bus 202 via one or more interfaces. Bus 202 may include, but is not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, etc. Hardware configuration 200 may include, for example, one or more processors 204, one or more input devices 206, and one or more output devices 208. One or more processors 204 may be any kind of processor, and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as special-purpose processing chips). Input device 206 may be any type of input device capable of inputting information to a computing device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. Output device 208 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer.

[0059] The hardware configuration 200 may also include or be connected to a non-transitory storage device 214, which may be any non-transitory storage device that can implement data storage, and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape, or any other magnetic medium, a compressed disk, or any other optical medium, a cache memory, and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions, and / or code. The hardware configuration 200 may also include a random access memory (RAM) 210 and a read-only memory (ROM) 212. The ROM 212 may store programs, utilities, or processes to be executed in a non-volatile manner. The RAM 210 may provide volatile data storage and store instructions related to the operation of the hardware configuration 200. The hardware configuration 200 may also include a network / bus interface 216 coupled to a data link 218. The network / bus interface 216 may be any type of device or system capable of enabling communication with an external device and / or network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset (such as a Bluetooth TM equipment, 802.11 equipment, WiFi equipment, WiMax equipment, cellular communication facilities, etc.).

[0060] The present disclosure may be implemented as any combination of an apparatus, a system, an integrated circuit, and a computer program on a non-transitory computer-readable medium. One or more processors may be implemented as an integrated circuit (IC), an application specific integrated circuit (ASIC), or a large scale integrated circuit (LSI), a system LSI, a super LSI, or a super LSI component that performs some or all of the functions described in the present disclosure.

[0061] The present disclosure includes the use of software, applications, computer programs, or algorithms. The software, applications, computer programs, or algorithms may be stored on a non-transitory computer-readable medium to enable a computer such as one or more processors to perform the above steps and the steps described in the accompanying drawings. For example, one or more memories store the software or algorithm in executable instructions, and one or more processors may associate a set of instructions to execute the software or algorithm to provide various functions according to the embodiments described in the present disclosure.

[0062] Software and computer programs (which may also be referred to as programs, software applications, applications, components or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, or an assembly language or a machine language. The term "computer-readable medium" refers to any computer program product, apparatus or device for providing machine instructions or data to a programmable data processor, such as magnetic disks, optical disks, solid-state storage devices, memories and programmable logic devices (PLDs), including computer-readable media that receive machine instructions as computer-readable signals.

[0063] For example, computer readable media may include dynamic random access memory (DRAM), random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired computer readable program code in the form of instructions or data structures and can be accessed by a general or special purpose computer or a general or special purpose processor. As used herein, disk or disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically by laser. Combinations of the above are also included in the scope of computer readable media.

[0064] The present invention designs a motion-aware ConvLSTM network for human action recognition in videos. Compared with the existing LSTM-like structure, it can effectively enhance the LSTM unit with motion information and better represent the video information. The motion-aware unit proposed in the present invention applies the kernel cross-correlator to the features of consecutive frames and uses the results in the recursive equation of convLSTM, which can simultaneously capture motion information, as well as spatial features and their long-term dependencies. The method only relies on the RGB image of the video, avoiding the computational complexity of calculating optical flow.

[0065] The subject matter of the present disclosure is provided as an example of devices, systems, methods and programs for performing the features described in the present disclosure. However, in addition to the above-mentioned features, other features or variations may also be expected. It is expected that the implementation of the components and functions of the present disclosure may be completed with any emerging technology that may replace any of the above-mentioned implementation technologies.

[0066] In addition, the above description provides examples without limiting the scope, applicability or configuration set forth in the claims. Without departing from the spirit and scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various embodiments may appropriately omit, replace or add various processes or components. For example, features described with respect to certain embodiments may be combined in other embodiments.

[0067] In addition, in the description of the present disclosure, the terms “first”, “second”, “third”, etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance and order.

[0068] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A method for action recognition based on video, comprising: Based on data of consecutive frames included in the video, using a kernel cross-correlator to calculate the position of the tracking target in each frame; as well as Based on the data of each frame and the position of the tracked target in each frame, the long short-term memory model LSTM is used to calculate the motion features in the video. Among them, calculating the position of the tracked target in each frame includes: Taking the calculated position of the tracked target in the previous frame as the center, determine one or more data blocks in the data of the current frame. Using a cross-correlation coefficient matrix calculated based on data of a previous frame, cross-correlation filtering is performed on the one or more data blocks to calculate a cross-correlation filtering response matrix, and Determine the position of the tracked target in the current frame according to the response matrix, The method of performing cross-correlation filtering on the one or more data blocks using the cross-correlation coefficient matrix calculated based on the data of the previous frame to calculate the response matrix of the cross-correlation filtering is based on the following formula: Where M is the block-by-block correlation flow matrix, which is used to estimate the transformation of each element in the input data for the data block around the position of the tracked target. represents the inverse Fourier transform, is the center position of the image, h is the cross-correlation coefficient matrix, g is the response matrix of the cross-correlation filter, Θ is the element-by-element multiplication, h * Expressed as the conjugate of h, Symbolic to frequency domain transformation, x t represents the input at time t, and They are calculated based on the following formulas:

2. The method according to claim 1, wherein: The kernel function of the kernel cross-correlator is a Gaussian kernel function, and the response matrix used for training initialization of the kernel cross-correlator is a matrix in which the element at the center position is 1 and the other elements are 0.

3. The method according to claim 1, wherein: Using LSTM to calculate the motion information in the video includes: based on the data X of each frame t And the position M of the tracked target in each frame t To calculate the gate function and candidate storage units of LSTM, LSTM is used to calculate the motion information in the video.

4. The method according to claim 3, wherein: Calculate the input gate function I of LSTM according to the following formula t , forget gate function F t , output gate function O t and candidate storage units I t =σ(W xi *X t +W hi *H t-1 +W mi *M t +b i ) F t =σ(W xf *X t +W hf *H t-1 +W mf *M t +b f ) O t =σ(W xo *X t +W ho *H t-1 +W mo *M t +b o ) Among them, the subscript t indicates that the parameter is about the current frame, the subscript t-1 indicates that the parameter is about the previous frame, the subscript i indicates that the parameter is about the input gate function, the subscript f indicates that the parameter is about the forget gate function, and the subscript o represents the parameter about the output gate function, the subscript c represents the parameter about the candidate storage unit, σ is the activation function, * is the convolution operation, H is the hidden state, W x , W h , W m is a two-dimensional convolution kernel, and b is a training parameter.

5. The method according to claim 1, further comprising: Before using a kernel cross-correlator to calculate the position of the tracked target in each frame, a convolutional layer is used to extract a feature map of each frame of the video, wherein the data is a feature map.

6. The method according to claim 1, further comprising: The calculated motion features are used to perform action classification using a softmax classifier.

7. A system for performing action recognition based on video, comprising: A spatial feature extraction module is configured to process each frame included in the video using a convolutional layer to obtain a feature map of each frame; A motion flow estimation module is configured to calculate the position of the tracked target in each frame using a kernel cross-correlator based on feature maps of consecutive frames; The motion feature estimation module is configured to calculate the motion features in the video using a long short-term memory model LSTM based on the feature map of each frame and the position of the tracked target in each frame; as well as The action classification module is configured to use a softmax classifier to perform action classification on the calculated motion features. Among them, calculating the position of the tracked target in each frame includes: Taking the calculated position of the tracked target in the previous frame as the center, determine one or more data blocks in the data of the current frame. Using a cross-correlation coefficient matrix calculated based on data of a previous frame, cross-correlation filtering is performed on the one or more data blocks to calculate a cross-correlation filtering response matrix, and Determine the position of the tracked target in the current frame according to the response matrix, The method of performing cross-correlation filtering on the one or more data blocks using the cross-correlation coefficient matrix calculated based on the data of the previous frame to calculate the response matrix of the cross-correlation filtering is based on the following formula: Where M is the block-by-block correlation flow matrix, which is used to estimate the transformation of each element in the input data for the data block around the position of the tracked target. represents the inverse Fourier transform, is the center position of the image, h is the cross-correlation coefficient matrix, g is the response matrix of the cross-correlation filter, Θ is the element-by-element multiplication, h * Expressed as the conjugate of h, Symbolic to frequency domain transformation, x t represents the input at time t, and They are calculated based on the following formulas:

8. A system for performing action recognition based on video, comprising: a memory having instructions stored therein; as well as A processor configured to execute instructions stored on the memory to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium comprising computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 6.

Images