Human body action recognition method and system based on key frame

Abstract

The invention discloses a human body action recognition method and system based on key frames, and the method comprises the steps: obtaining a target video, and extracting the key frames of actions from the target video; constructing motion features of the global features and the local features based on the key frame, wherein the motion features comprise a global motion image and a quantitative motion feature; constructing a multi-scale motion matrix through quantitative motion features calculated by different original points; and performing action recognition on the multi-scale motion matrix and the global features through a two-stage cascade neural network. The input parameters can be remarkably reduced by using the key frame. In the key frame extraction process, the time scales of different actions can be unified at the same time. Secondly, constructing a motion matrix of which the scale is much larger than that of only one original point by adopting a plurality of original points; therefore, higher precision can be achieved with the CNN having a deeper layer currently. Thirdly, a cascade neural network is provided, and efficient and accurate action prediction is carried out through global features and local features.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion recognition, and in particular to a human motion recognition method and system based on key frames. Background Art

[0002] Human action recognition (HAR) has a wide range of industrial applications, such as video retrieval, video caption generation, virtual reality, and human-computer interaction. In recent years, with the help of deep learning technology, the detection accuracy of HAR has been continuously improved. Many global feature-based methods have emerged together with CNNs because of the excellent feature extraction ability of CNNs. Motion history images (MHI), static history images (SHI), and optical flow (OF) images are typical global features. Using these features and their variants with various CNNs is the mainstream solution for HAR in the early days of deep learning. Although global feature-based methods have achieved remarkable performance in action recognition in a short time, the overlap of motion trajectories will inevitably lead to the loss of spatiotemporal information of the action.

[0003] To overcome this challenge, some researchers tried to extract contextual information between frames. Therefore, methods based on recurrent neural networks (RNNs) became popular for a time. Then, due to the low ability of RNNs to control long videos, long short-term memory (LSTM) networks quickly replaced RNNs as the mainstream of methods based on temporal information. With the continuous improvement of computing power, other researchers used 3D-CNN to obtain temporal and spatial features to recognize actions. However, methods based on 3D-CNN are not mainstream methods. This is because methods based on 3D-CNN require a lot of computing resources, which inevitably makes the entire model fall into difficulties in training and fine-tuning. In addition, due to its computing power requirements, it is difficult to apply to mobile terminals. Summary of the invention

[0004] The purpose of the present invention is to provide a keyframe-based human motion recognition method and system that can simultaneously meet the requirements of high precision, high efficiency, easy fine-tuning, few-sample learning and multi-scale learning, so as to solve the shortcomings of the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a human action recognition method based on key frames, the recognition method comprising the following steps: Obtain a target video, and extract key frames of the action from the target video; Construct motion features of global features and local features based on key frames, including global motion images and quantized motion features; Construct a multi-scale motion matrix through quantized motion features calculated from different original points; The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network.

[0006] In a preferred embodiment, extracting key frames of an action from a target video comprises the following steps: Enter the number of keyframes to extract , define the following array of quantitative indicators for storing key frame judgment: joint displacement array ADR , joint angle array , fusion weight array Wl ; Calculate the joint displacement between two frames, the expression is: , in( , , ) is the three-dimensional coordinate of a joint in the i-th frame, ( , , ) is the three-dimensional coordinate of a joint in the i-1th frame, represents the displacement of a joint in the i-th frame, the value range of i is (0, x], x represents the total number of frames of an action; Calculate the displacement of the nth joint in the i-th frame as , calculate the total displacement of all joints in the i-th frame as , calculate the total displacement of all joints frame by frame and store it in the joint displacement array ADR middle; Calculate the angle between adjacent joints in a coordinate system. The expression is: , in Representative i The angle of a joint in a frame, J m , J n and J l There are three adjacent joints. Indicates from J m arrive J n The vector of Indicates from J l arrive J n The vector of i-1 The angle of a joint in the frame is , No. i Frame and i-1 The change in a joint angle between frames is expressed as ; Calculate the change in the zth joint angle of the i-th frame as , calculate the total change of all joint angles in the i-th frame as , calculate the total change of all joint angles frame by frame and store it in the joint angle array middle; The joint displacement array ADR and the joint angle array The elements in are sorted in frame order; Calculate the joint displacement weight of the i-th frame, the expression is: , Calculate the joint angle weight of the i-th frame, the expression is: , Calculate the fusion weight of the i-th frame, the expression is: , Calculate the fusion weight of each frame and store it in the fusion weight array Wl middle; The weight array will be fused Wl Sort the elements in descending order and take the fusion weight array Wl The front The frame corresponding to the element is output as the final key frame.

[0007] In a preferred embodiment, the quantified motion features include joint displacement, joint motion direction, joint motion speed, and joint angle; The joint displacement is the spatial change of the joint, D is the joint displacement in 3D coordinates, and D is projected on the x, y, and z axes; The joint motion direction is the change in direction of the joint within a specific frame duration, which contains both spatial and temporal information. Based on the translation change of the vector, the joint motion direction is calculated using the cosine law as follows: , where {x, y, z} is the value of a joint in i The position in the frame, and { , , } is the displacement of the joint on the x, y and z axes between frame i and frame j, , , are respectively the projection values ​​of the motion direction within the (ji) frame duration on the planes xy, xz and zy, is the motion direction value within the duration of (ji) frame; The joint movement speed indicates the movement intensity of a joint. Based on D, the joint movement speed A is calculated as follows: , Where t is the duration of a frame, n is the number of interval frames; Joint angles describe the posture of the human body.

[0008] In a preferred embodiment, a multi-scale motion matrix is ​​constructed by quantized motion features calculated from different original points, including: Multiple original points are used to calculate the joint displacement, joint motion direction, joint motion speed and joint angle frame by frame to construct a multi-scale motion matrix.

[0009] In a preferred embodiment, the multi-scale motion matrix and the global features are used for action recognition through a two-stage cascade neural network, including the following steps: The multi-scale motion matrix is ​​input into the first-stage neural network to obtain the prediction result of the first-stage neural network. If the confidence of the prediction result of the first-stage neural network exceeds the confidence threshold, the prediction result of the first-stage neural network is the final result; if the confidence of the prediction result of the first-stage neural network is lower than the confidence threshold, the second-stage neural network starts to work, and the global features are input into the second-stage neural network. Another prediction result of the action is received from a global and semantic perspective, and the average of the two-stage prediction results is regarded as the final action of the prediction result confidence, and the prediction result with the highest value is selected.

[0010] A human action recognition system based on key frames, including a key frame extraction module, a feature construction module, a multi-scale motion matrix construction module, and an action recognition module; Key frame extraction module: obtain the target video and extract the key frames of the action from the target video; Feature construction module: constructs motion features of global features and local features based on key frames, including global motion images and quantized motion features; Multi-scale motion matrix construction module: constructs a multi-scale motion matrix through quantized motion features calculated from different original points; Action recognition module: The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network.

[0011] In the above technical solution, the technical effects and advantages provided by the present invention are: In the present invention, the key frames of the action video are automatically selected by a specific strategy without considering all frames. Therefore, the use of key frames can significantly reduce the input parameters. In addition, the number of key frames can be controlled. Therefore, in the key frame extraction process, the time scales of different actions can be unified at the same time. Secondly, multiple original points are used to construct a motion matrix with a much larger scale than only one original point. Therefore, the current CNN with deeper layers can be used to achieve higher accuracy. Thirdly, a cascaded neural network is proposed to perform efficient and accurate action prediction through global features and local features. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 is an overall block diagram of the method of the present invention; Figure 2 Schematic diagram of 25 joints of the human skeleton used in Example 1 of the present invention; Figure 3 This is a key frame extraction flow chart in Embodiment 1 of the present invention; Figure 4 Schematic diagram of joint angles used in Example 1 of the present invention; Figure 5 Schematic diagram of frame-by-frame calculation of joint displacement, joint movement direction, joint movement speed and joint angle in Example 1 of the present invention; Figure 6 Schematic diagram of constructing a multi-scale motion matrix using 8 joints as candidate values ​​of relative original points in Example 1 of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Example 1: Please refer to Figure 1 As shown, the human action recognition method based on key frames described in this embodiment includes the following steps: Obtain a target video, and extract key frames of the action from the target video; Construct motion features of global features and local features based on key frames, including global motion images and quantized motion features; Construct a multi-scale motion matrix (MSMM) by quantizing motion features calculated from different original points; The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network.

[0016] Depth images provide 3D coordinates of each pixel, which makes it easier for researchers to perform spatial calculations between pixels. In addition, the human body joints provided by depth sensors are highly relevant to the participants. Therefore, researchers can only focus on joints and hard bones and ignore other pixels in the depth image. Therefore, skeleton-based HAR methods always have an advantage in efficiency.

[0017] Taking Kinect v2.0 as an example, it provides information about 25 joints in total, each of which can be tracked or estimated by the embedded program. Figure 2 The 25 joints and the joints used as origin points for computing relative motion features are shown.

[0018] Extracting key frames of actions from the target video includes the following steps: First, if a keyframe of an action can be extracted from a video, the entire action can be accurately represented. Second, keyframes are always those with intense physical activity. Therefore, displacement and changes in body geometry can be quantified as two criteria for keyframe extraction.

[0019] In this embodiment, joint displacement and joint angle change are calculated frame by frame to quantify the intensity of the action in a certain frame.

[0020] The key frame extraction process is as follows Figure 3 As shown, the following steps are included: Enter the number of keyframes to extract , define the following array of quantitative indicators for storing key frame judgment: joint displacement array ADR , joint angle array , fusion weight array Wl .

[0021] Calculate the joint displacement between two frames, the expression is: , in( , , ) is the three-dimensional coordinate of a joint in the i-th frame, ( , , ) is the three-dimensional coordinate of a joint in the i-1th frame, represents the displacement of a joint in the i-th frame, where i ranges from (0, x], and x represents the total number of frames of an action. Therefore, the motion intensity of a frame can be quantified as the total displacement of the entire 25 joints, expressed as: , in Indicates i The total displacement of all 25 joints in the frame, Represents the displacement of the nth joint in the i-th frame.

[0022] Calculate the total displacement of all 25 joints frame by frame and store it in the joint displacement array ADR middle.

[0023] To further improve the accuracy of keyframe extraction, it is not enough to only consider joint displacement. Special attention should also be paid to changes in joint angles. Because some actions only contain a few joints that move violently, while other joints remain stationary. Such as boxing, clapping, waving, etc. These actions may change slightly in displacement, but may have a large change in geometry. In this embodiment, 19 joint angles are calculated frame by frame, such as Figure 4 shown.

[0024] Calculate the angle between adjacent joints in a coordinate system. The expression is: , in Representative i The angle of a joint in a frame, J m , J n and J l There are three adjacent joints. Indicates from J m arrive J n The vector of Indicates from J l arrive J n The vector of i-1 The angle of a joint in the frame is , No. i Frame and i-1 The change in a joint angle between frames is expressed as .

[0025] Calculate the total change of 19 joint angles in the i-th frame. The expression is: , in Indicates the change of the zth joint angle in the i-th frame, where z is the order of the joint angles, i is the order of the frames, and the value range of i is (0, x], where x represents the total number of frames of an action.

[0026] Calculate the total change of all 19 joint angles frame by frame and store them in the joint angle array middle.

[0027] The joint displacement array ADR and the joint angle array The elements in are sorted in frame order.

[0028] Calculate the joint displacement weight of the i-th frame, the expression is: , Calculate the joint angle weight of the i-th frame, the expression is: , Calculate the fusion weight of the i-th frame, the expression is: , Calculate the fusion weight of each frame and store it in the fusion weight array Wl middle.

[0029] The weight array will be fused Wl Sort the elements in descending order and take the fusion weight array Wl The front The frame corresponding to the element is output as the final key frame.

[0030] The quantified motion features include joint displacement, joint motion direction, joint motion speed and joint angle.

[0031] The joint displacement is the spatial change of the joint, D is the joint displacement in 3D coordinates, and D is projected on the x, y, and z axes.

[0032] The joint motion direction is the change in direction of the joint within a specific frame duration, which contains both spatial and temporal information. Based on the translation change of the vector, the joint motion direction is calculated using the cosine law as follows: , where {x, y, z} is the value of a joint in i The position in the frame, and { , , } is the displacement of the joint on the x, y and z axes between frame i and frame j, , , are respectively the projection values ​​of the motion direction within the (ji) frame duration on the planes xy, xz and zy, is the motion direction value within the (ji) frame duration.

[0033] Joint motion velocity indicates the intensity of a joint's motion over a period of time. From a physics perspective, it can be used to distinguish similar movements. Based on D, the joint motion velocity A is calculated as follows: , Where 0.3333 is the duration of one frame (Kinect v2 records video at 30 fps, meaning there are 30 frames in one second), and n is the number of frames per second.

[0034] The joint angle is a description of the human body posture. Figure 4 As shown, there are two types of joint angles, one consisting of three adjacent joints and their related bones, and the other consisting of the involved joints and dotted lines.

[0035] The coordinates of the joints, or even the original coordinates, are the only data used to describe an action. The three rules for action quantification and representation are as follows: (1) Relative coordinates are more suitable for representing human body movements than raw coordinates. This is because raw coordinates vary greatly and may have side effects, which are caused by the camera installation position. Therefore, methods based on raw coordinates are always dataset-oriented and have lower performance when the dataset changes or is applied to the real world.

[0036] (2) Performing necessary preprocessing on the coordinates helps improve accuracy. Although CNN has excellent feature extraction capabilities, calculating necessary motion features such as speed, direction, joint angles, etc. can provide more useful information to CNN and will definitely produce high detection accuracy.

[0037] (3) Several time scales need to be defined for action samples. Since the duration of different actions may vary greatly (for example, a "boxing" action may be completed in 10-15 frames, while a "sitting" action may take 30-60 frames), having only a unified scale may lead to loss of action information or action variation.

[0038] In this embodiment, the above three rules are followed and a small-scale problem is solved to further improve the flexibility and accuracy of the method of the present invention. In most research works, the action representation is an image or a floating-point matrix. Since the existing algorithms or depth cameras can only provide up to 25 joints, the motion image or motion matrix is ​​small in size. With the continuous improvement of hardware computing power, the small-size representation model that was once friendly to efficiency has lost its advantage due to the limitation of accuracy.

[0039] In order to meet all the requirements of the above analysis, a multi-origin idea is proposed to generate motion matrices with flexible scales.

[0040] The multi-scale motion matrix (MSMM) is constructed by quantizing motion features calculated from different raw points, including: Multiple original points are used to calculate the joint displacement, joint movement direction, joint movement speed and joint angle frame by frame to construct a multi-scale motion matrix.

[0041] The schematic diagram of frame-by-frame calculation of joint displacement, joint movement direction, joint movement speed and joint angle is as follows Figure 5 shown.

[0042] When the action occurs, the joints that move slightly relative to the origin point should be selected. Therefore, the eight joints of neck, shoulder joint-middle, shoulder joint-left, shoulder joint-right, hip middle, spine midpoint, left hip, and right hip are used as candidate values ​​for the relative origin point. Figure 6 shown.

[0043] There are two advantages of using multiple relative origin points. First, a motion feature can be calculated multiple times. Therefore, the scale of the feature dimension can also be enlarged. Second, a motion feature based on different relative origin points can partially reflect the geometric information of the human body in a certain frame, which is also useful for CNN to learn deep features.

[0044] In most methods based on "skeleton + CNN", the scale of motion representation is "joint × feature × frame". Taking 4 features as an example, the scale of an action with a duration of 45 frames is "25 × 4 × 45". If only 10 frames are extracted as key frames, the scaling ratio is "25 × 4 × 10", which cannot withstand deep conventional calculations. However, if all relative original points are calculated and their changes are used, the scale can be enlarged. For example, the joint displacement of a joint in 3D coordinates can be reflected on three planes. Therefore, a displacement feature can be divided into 4 features. Furthermore, if all 8 relative original points are used, the final scale of the action representation matrix can be enlarged to (25 × (4 × 4) × (10 × 8)), which can be resized to (200 × 160 × 1).

[0045] The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network, including the following steps: The multi-scale motion matrix (MSMM) is used as the input of the first-stage neural network due to its inherent advantages of small size and local features, and the prediction result of the first-stage neural network is obtained. If the confidence of the prediction result of the first-stage neural network exceeds the confidence threshold, the prediction result of the first-stage neural network is the final result; if the confidence of the prediction result of the first-stage neural network is lower than the confidence threshold, the second-stage neural network starts to work, and the global features are input to the second-stage neural network, and another prediction result of the action is received from the global and semantic perspective. The average of the two-stage prediction results is then regarded as the final action of the prediction result confidence, so the prediction result with the highest value is selected.

[0046] The advantages of this two-stage cascade neural network structure can be summarized in two aspects. First, the network uses both local and global features. The use of local features in the first-stage neural network can effectively avoid the self-intersection loss of global features, and in the second-stage neural network, when the confidence of the prediction results of the first-stage neural network is lower than the confidence threshold, the prediction results based on global features can provide a useful supplement to improve the prediction accuracy from a semantic perspective. Second, by changing the confidence threshold of the prediction results of the first-stage neural network, it is easy to draw a clear line between effectiveness and efficiency.

[0047] The present invention proposes an effective HAR method based on the use of key frames. First, the scheme regards frames containing significant displacement and geometric transformation as key frames, and selects the first N frames to describe an action. It is the core step of the whole method. Two main problems are solved. 1) The time scales of different actions are normalized to the same size. 2) The action video is greatly simplified. Then, the sparse motion history image (SMHI) and four motion features such as displacement, velocity, motion direction and joint angle of each joint are calculated and used as the input of the cascade neural network. The efficiency of the network is further improved by the mechanism of cascade neural network. Finally, in order to use a deeper network to achieve higher accuracy, the present invention adopts multiple raw points to obtain a motion matrix of larger size. The number of raw points can be flexibly defined to achieve high accuracy or high efficiency.

[0048] Embodiment 2: The keyframe-based human action recognition system described in this embodiment includes a keyframe extraction module, a feature construction module, a multi-scale motion matrix construction module, and an action recognition module; Key frame extraction module: obtain the target video and extract the key frames of the action from the target video; Feature construction module: constructs motion features of global features and local features based on key frames, including global motion images and quantized motion features; Multi-scale motion matrix construction module: constructs a multi-scale motion matrix through quantized motion features calculated from different original points; Action recognition module: The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network.

[0049] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0050] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A human action recognition method based on key frames, characterized in that: The identification method comprises the following steps: Obtain a target video, and extract key frames of the action from the target video; Construct motion features of global features and local features based on key frames, including global motion images and quantized motion features; Construct a multi-scale motion matrix through the quantized motion features calculated from different original points; The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network.

2. The human action recognition method based on key frames according to claim 1, characterized in that: Extracting key frames of actions from the target video includes the following steps: Enter the number of keyframes to extract , define the following array of quantitative indicators for storing key frame judgment: joint displacement array ADR , joint angle array , fusion weight array Wl ; Calculate the joint displacement between two frames, the expression is: ， in( , , ) is the three-dimensional coordinate of a joint in the i-th frame, ( , , ) is the three-dimensional coordinate of a joint in the i-1th frame, represents the displacement of a joint in the i-th frame, the value range of i is (0, x], x represents the total number of frames of an action; Calculate the displacement of the nth joint in the i-th frame as , calculate the total displacement of all joints in the i-th frame as , calculate the total displacement of all joints frame by frame and store it in the joint displacement array ADR middle; Calculate the angle between adjacent joints in a coordinate system. The expression is: ， in Representative i The angle of a joint in a frame, J m , J n and J l There are three adjacent joints. Indicates from J m arrive J n The vector of Indicates from J l arrive J n The vector of i-1 The angle of a joint in the frame is , No. i Frame and i-1 The change in a joint angle between frames is expressed as ; Calculate the change in the zth joint angle of the i-th frame as , calculate the total change of all joint angles in the i-th frame as , calculate the total change of all joint angles frame by frame and store it in the joint angle array middle; The joint displacement array ADR and an array of joint angles The elements in are sorted in frame order; Calculate the joint displacement weight of the i-th frame, the expression is: ， Calculate the joint angle weight of the i-th frame, the expression is: ， Calculate the fusion weight of the i-th frame, the expression is: ， Calculate the fusion weight of each frame and store it in the fusion weight array Wl middle; The weight array will be fused Wl Sort the elements in descending order and take the fusion weight array Wl The front The frame corresponding to the element is output as the final key frame.

3. The human action recognition method based on key frames according to claim 2, characterized in that: Quantified motion features include joint displacement, joint motion direction, joint motion speed, and joint angle; The joint displacement is the spatial change of the joint, D is the joint displacement in 3D coordinates, and D is projected on the x, y, and z axes; The joint motion direction is the change in direction of the joint within a specific frame duration, which contains both spatial and temporal information. Based on the translation change of the vector, the joint motion direction is calculated using the cosine law as follows: ， where {x, y, z} is the value of a joint in i The position in the frame, and { , , } is the displacement of the joint on the x, y and z axes between frame i and frame j, , , are respectively the projection values ​​of the motion direction within the (ji) frame duration on the planes xy, xz and zy, is the motion direction value within the duration of (ji) frame; The joint movement speed indicates the movement intensity of a joint. Based on D, the joint movement speed A is calculated as follows: ， Where t is the duration of a frame, n is the number of interval frames; Joint angles describe the posture of the human body.

4. The method for human motion recognition based on key frames according to claim 3, characterized in that: The multi-scale motion matrix is ​​constructed by quantizing motion features calculated from different original points, including: Multiple original points are used to calculate the joint displacement, joint motion direction, joint motion speed and joint angle frame by frame to construct a multi-scale motion matrix.

5. The key-frame-based human action recognition method according to claim 4, characterized in that: The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network, including the following steps: The multi-scale motion matrix is ​​input into the first-stage neural network to obtain the prediction result of the first-stage neural network. If the confidence of the prediction result of the first-stage neural network exceeds the confidence threshold, the prediction result of the first-stage neural network is the final result; if the confidence of the prediction result of the first-stage neural network is lower than the confidence threshold, the second-stage neural network starts to work, and the global features are input into the second-stage neural network. Another prediction result of the action is received from a global and semantic perspective, and the average of the two-stage prediction results is regarded as the final action of the prediction result confidence, and the prediction result with the highest value is selected.

6. A human action recognition system based on key frames, used to implement the recognition method according to any one of claims 1 to 5, characterized in that: It includes key frame extraction module, feature construction module, multi-scale motion matrix construction module and action recognition module; Key frame extraction module: obtain the target video and extract the key frames of the action from the target video; Feature construction module: constructs motion features of global features and local features based on key frames, including global motion images and quantized motion features; Multi-scale motion matrix construction module: constructs a multi-scale motion matrix through quantized motion features calculated from different original points; Action recognition module: The multi-scale motion matrix and global features are used for action recognition through a two-stage cascade neural network.

Images