Video action classification method based on hierarchical dynamic modeling of sandglass convolution and application
By constructing a hierarchical dynamic modeling network through hourglass convolution, the problems of heavy computational burden and target loss caused by visual displacement in video action recognition are solved, and higher accuracy of video action recognition is achieved.
Patent Information
- Application Number
- CN202211053069.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-31
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2042-08-31
AI Technical Summary
Existing video classification technologies suffer from problems such as heavy computational burden, limited temporal modeling capabilities, and target loss due to visual displacement when processing video action recognition, resulting in low recognition accuracy.
A hierarchical dynamic modeling network is constructed using hourglass convolution, including frame-level and segment-level dynamic information capture networks. The network captures video dynamic information hierarchically through hourglass convolution, utilizes the hourglass-shaped receptive field of hourglass convolution to capture large visual displacements, and performs feature extraction and classification through ResNet50 network and multi-layer convolutional blocks.
It improves the accuracy of human motion video recognition, effectively capturing visual displacement between different moments in the video and enhancing the accuracy of video motion classification.
Smart Images
Figure CN115424175B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision, and particularly relates to a video action classification method based on hierarchical dynamic modeling of hourglass convolution and application. BACKGROUND
[0002] The scale, position and perspective pattern of visual cues (such as semantics, objects) in a video evolve along the time axis. The discriminative motion pattern obtained by aggregating these dynamic changes is crucial for the classification of video content. In order to capture these features, the following methods currently exist:
[0003] Using optical flow information as external information can enhance the dynamic modeling of actions in a video. The most representative work is the two-stream network, which represents motion in the form of optical flow and sends static (RGB) and dynamic (optical flow) information into two independent convolutional neural networks, and then fuses the classification results of the two streams to obtain the final video classification result. Although the two-stream network is effective in learning dynamic features, the acquisition of optical flow information and the addition of an extra convolutional neural network branch make the two-stream network computationally heavy.
[0004] Subsequently, it was found that using one-dimensional temporal convolution, temporal aggregation can be well modeled by aggregating the same spatial position at adjacent times. Specifically, one-dimensional temporal convolution and two-dimensional spatial convolution are combined in a cascaded or parallel manner in the network, so that the two-dimensional convolutional neural network has a time perception ability, and therefore this paradigm is widely used in network design for video classification tasks. However, if the time dimension is not specially considered, the network designed based on this paradigm has limited time modeling capability. At the same time, the possible large visual displacement between adjacent time frames makes the rigid one-dimensional temporal convolution unable to capture the motion pattern well. For example, the action of "picking up a ping pong ball and putting the ball on the table" includes the interaction of core objects such as "hand" and "ping pong ball". As time goes on, the spatial semantics of a single frame gradually changes from "picking up the ball" to "holding the ball in the air" and "putting the ball on the table". In this process, the scale, position and pattern of "hand" and "ping pong ball" have all changed. The rigid one-dimensional temporal convolution only considers the dynamic changes of the same spatial position at different times, without considering large changes, so when the target object moves out of the receptive field in adjacent frames, it is easy to lose the core visual cues that the object has.
[0005] Attention strategy, a method of representing motion patterns using similarity between spatio-temporal changes, can also effectively model dynamic changes. However, due to the low efficiency of pair-wise similarity computation, it has the same heavy computational burden as the optical flow-based method.
[0006] In summary, current technologies applied to video classification have many shortcomings and drawbacks, resulting in poor classification performance and low accuracy. Summary of the Invention
[0007] This invention aims to address the shortcomings of existing technologies by proposing a hierarchical dynamic modeling method for video action classification based on hourglass convolution and its application. The goal is to achieve better modeling of video dynamics using hourglass convolution, and to utilize frame-level and segment-level dynamic information capture networks based on hourglass convolution to hierarchically model video dynamic information from multiple levels, thereby improving the accuracy of human action video recognition.
[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:
[0009] The video action classification method based on hierarchical dynamic modeling using hourglass convolution, as described in this invention, is characterized by the following steps:
[0010] Step 1: Video data extraction and preprocessing:
[0011] T keyframe images are uniformly sampled from the video of a person's movement V at a fixed frame rate, denoted as F = [F1, F2, ..., F]. t ,…,F T ], F t This represents the t-th keyframe, where T represents the keyframe number.
[0012] Sample the t-th keyframe F t In the character's motion video V, two consecutive frames before and after, and F... t The t-th segment is represented by the two consecutive frames before and after it. F represents t The first two frames, F represents t The previous frame, F represents t The next frame, F represents t The last two frames;
[0013] The t-th segment C t The resolution of each frame in the input video is scaled, and then an image block with a resolution of H×W is extracted from each frame and normalized before preprocessing to obtain the t-th input video data tensor. Thus, the input video data tensor C' = [C'1, C'2, ..., C''] of the character's motion video V is obtained. t ,…,C' T ], where H and W represent C' t Height and width, D represents C' t The number of channels;
[0014] Step 2, constructing a hierarchical hourglass convolution network, comprising: a frame-level dynamic information capturing network, a segment-level dynamic information capturing network and a classification network;
[0015] Step 2.1, constructing an hourglass convolution:
[0016] The hourglass convolution is composed of a group of spatial convolutions with a kernel size of (p·|i|+1, p·|i|+1) and a temporal convolution with a kernel size of K, wherein p is a parameter and i is a time offset;
[0017] The hourglass convolution processes any tensor X with a dimension of to obtain an output feature HgC(X), wherein T' represents the size of the time dimension, H' represents the height, W' represents the width, and D' represents the number of channels, and the t-th feature HgC(X) t of the output feature HgC(X) is obtained by using formula (1):
[0018]
[0019] In formula (1), X t+i is the t+i-th input feature of the tensor X in the T' time dimension, a i is the i-th parameter of the temporal convolution layer, f is a spatial convolution function, W p·|i|+1,p·|i|+1 is the parameter of the spatial convolution layer; t∈[0,T'-1];
[0020] Step 2.2, the frame-level dynamic information capturing network is composed of the first convolution block of the ResNet50 network and a frame-level dynamic information capturing module:
[0021] The first convolution block of the ResNet50 network is a spatial convolution with a convolution kernel of a×a;
[0022] The frame-level dynamic information capturing module is composed of a down-sampling layer, an hourglass convolution layer, a spatial convolution layer and an up-sampling layer:
[0023] The down-sampling layer is a spatial average pooling layer with a kernel size of b×b; the hourglass convolution layer is composed of two serially connected hourglass convolutions; the spatial convolution layer is a spatial convolution with a convolution kernel of a×a; and the up-sampling layer is an up-sampling operation for copying one pixel into four adjacent pixels;
[0024] The key frame image F=[F1, F2, …, F t ,…,F T ] of the human action video V is input into the first convolution block of the ResNet50 network for processing, and an output feature F S is obtained.
[0025] The input video data tensor C' = [C'1, C'2, ..., C''] of the character's action video V is... t ,…,C' T The input is fed into the frame-level dynamic information capture module, and after passing through a downsampling layer, an hourglass convolutional layer, a spatial convolutional layer, and an upsampling layer in sequence, the output feature F is obtained. fm ;
[0026] F S With F fm The summation yields the output M of the frame-level motion capture network. fm ;
[0027] Step 2.3: The segment-level dynamic information capture network consists of four convolutional blocks connected in series. Each convolutional block is composed of repeated units connected in series, and the number of repeated units contained in each convolutional block is different.
[0028] The repeating unit consists of a residual block and a fragment-level dynamic information capture module; the residual block contains two convolutional layers with 1×1 kernels and one convolutional layer with 3×3 kernels; the fragment-level dynamic information capture module contains two 1×1×1 convolutional layers, an hourglass convolution, a global average pooling layer, and a sigmoid activation function layer.
[0029] M fm After being input into the first 1×1 convolutional layer of the first repeating unit in the first convolutional block of the fragment-level dynamic information capture network, feature Y is obtained. Y is then input into the fragment-level dynamic information capture module and processed sequentially through the first 1×1×1 convolutional layer, an hourglass convolutional layer, a global average pooling layer, the second 1×1×1 convolutional layer, and a Sigmoid activation function layer to obtain feature A. A is multiplied by Y and then input into the residual block of the first repeating unit in the first convolutional block. After being processed sequentially through a 3×3 convolutional layer and the second 1×1 convolutional layer, the output Z' of the first repeating unit of the first convolutional block is obtained.
[0030] Z' is then input into the second repeated unit in the first convolutional block, and the result after the same processing is input into the next repeated unit. Thus, the result after processing by all repeated units in the first convolutional block is input into the next convolutional block for processing. Finally, the output Z of the hierarchical hourglass convolutional network is obtained from the last complex unit of the fourth convolutional block.
[0031] Step 3: The classification network consists of a global average pooling layer and a fully connected layer connected in series; Z is input into the classification network for processing to obtain the final action category;
[0032] Step 4, constructing a cross-entropy loss function as a loss function L of the hierarchical hourglass convolutional network, and training the hierarchical hourglass convolutional network by using an SGD optimizer, while calculating the loss function L to adjust network parameters, finally obtaining a trained hierarchical hourglass convolutional network as a video action classifier, for realizing video action classification.
[0033] The electronic device comprises a memory and a processor, wherein the memory is configured to store a program supporting the processor to execute the video action classification method, and the processor is configured to execute the program stored in the memory.
[0034] The computer readable storage medium stores a computer program, wherein the computer program is configured to execute the steps of the video action classification method when executed by a processor.
[0035] Compared with the prior art, the video action classification method has the following beneficial effects:
[0036] 1. The present application proposes a new type of time convolution, namely hourglass convolution (HgC), and constructs a video action recognition network based on the hierarchical hourglass convolution, which can effectively cope with the target loss caused by visual displacement between different time points of the video, thereby improving the accuracy of the human action video recognition.
[0037] 2. The hourglass convolution proposed in the present application has a sandglass-shaped receptive field, specifically: the spatial receptive field is enlarged in the front and rear time points, so that large visual displacement can be captured, and the spatio-temporal dynamic information modeling capability of the hourglass convolution is improved; and finally the recognition accuracy of the human action video is improved.
[0038] 3. The human video action classification network (Hierarchical Hourglass Convolutional Network, H 2 CN) constructed based on the hierarchical hourglass convolution in the present application simultaneously mines spatio-temporal dynamic information from two levels between adjacent frames and between adjacent segments, provides rich spatio-temporal dynamic information for the network, and further improves the recognition accuracy of the human action video of the network. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The flowchart of the video classification method in the embodiment of the present application is shown in the figure;
[0040] Figure 2 The schematic diagram of the hourglass convolution in the embodiment of the present application is shown in the figure;
[0041] Figure 3A video action classification network based on hourglass convolution hierarchical dynamic modeling in an embodiment of the present application is shown in the figure;
[0042] Figure 4 A frame-level dynamic information capture network in an embodiment of the present application is shown in the figure;
[0043] Figure 5 A segment-level dynamic information capture network in an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0044] In this embodiment, as shown in the figure, Figure 1 a video action classification method based on hourglass convolution hierarchical dynamic modeling is performed in the following steps:
[0045] Step 1, video data extraction and preprocessing:
[0046] T frame key frame images are uniformly sampled from the person action video V according to a fixed frame number, denoted as F = [F1, F2, …, FT], FT represents the tth key frame, T represents the number of key frames, and the number of key frames can generally be taken as 8, 16, 32, etc. t T t
[0047] The tth key frame F t is sampled in the person action video V, and F t and its two consecutive frames before and after are denoted as the tth segment t represents the two frames before F t represents the frame before F t represents the frame after F t represents the two frames after F t ;
[0048] After scaling the resolution of each frame in the tth segment C t , an image block with a resolution of HxW is taken out from each frame, and then normalized preprocessing is performed, to obtain the tth input video data tensor Thus, the input video data tensor C' = [C'1, C'2, …, C' t , …, C' T ] of the person action video V is obtained, wherein H and W represent the height and width of C' t , respectively, and the values of H and W can be set as 224 in the case of balancing recognition accuracy and computational efficiency, and D represents the number of channels of C' t , which is 3 in widely used RGB images;
[0049] Step 2. As shown in Figure 3 , a hierarchical hourglass convolutional network is constructed, including a frame-level dynamic information capturing network, a segment-level dynamic information capturing network and a classification network.
[0050] Step 2.1. Constructing an hourglass convolution:
[0051] The hourglass convolution is composed of a group of spatial convolutions with kernel size (p·|i|+1, p·|i|+1) and a temporal convolution with kernel size K, where i is the time offset, and p is the slope of the receptive field as the time offset increases. For example, when K = 3 and p = 2, the spatial convolution kernel sizes corresponding to the {t-1, t, t+1} frames are {(3, 3), (1, 1), (3, 3)} respectively.
[0052] The hourglass convolution processes any tensor X with dimension , where T' represents the size of the time dimension, H' represents the height, W' represents the width, and D' represents the number of channels, to obtain the output feature HgC(X). The process is as follows: first, use the above spatial convolution kernel to aggregate the spatial dimension information of the frames at the corresponding time offset. Then use the above temporal convolution to aggregate the time information along the time axis. The process of obtaining the t-th feature HgC(X) t of the output feature HgC(X) is as follows:
[0053]
[0054] In formula (1), X t+i is the t+i-th input feature of tensor X in the T' time dimension, a i is the i-th parameter of the temporal convolution layer, f is a spatial convolution function, W p·|i|+1,p·|i|+1 is the parameter of the spatial convolution layer; t ∈ [0, T'-1]; compared with the traditional method of video action recognition, the hourglass convolution additionally uses spatial convolution to first aggregate the spatial information of the frames at different time offsets, so that the hourglass convolution has a sandglass-shaped receptive field (as shown in Figure 2 ), thereby helping the hourglass convolution to aggregate spatio-temporal information that is difficult to aggregate due to visual offset at other time offsets. Therefore, compared with the temporal convolution widely used in the traditional method, the hourglass convolution can capture spatio-temporal information that the temporal convolution cannot capture, and better fit the spatio-temporal dynamic characteristics of video data.
[0055] Step 2.2. The frame-level dynamic information capturing network is composed of the first convolution block of the ResNet50 network and the frame-level dynamic information capturing module:
[0056] The first convolution block of the ResNet50 network is a spatial convolution with a convolution kernel of a x a. In general, the value of a is taken as 7.
[0057] The frame-level dynamic information capturing module is composed of a down-sampling layer, an hourglass convolution layer, a spatial convolution layer, and an up-sampling layer.
[0058] The down-sampling layer is a spatial average pooling layer with a kernel size of b x b, and the classical value of b in the present application is 2 in order to balance the recognition accuracy and the calculation efficiency; the hourglass convolution layer is composed of two serially connected hourglass convolutions; the spatial convolution layer is a spatial convolution with a convolution kernel of a x a; and the up-sampling layer is used for the up-sampling operation of copying one pixel into four adjacent pixels.
[0059] The key frame image F = [F1, F2, …, F t ,…,F T ] of the action video V of the person is input into the first convolution block of the ResNet50 network for processing, and the output feature F S is obtained.
[0060] The process of obtaining the frame-level dynamic information is shown in Figure 4 : the input video data tensor C' = [C'1, C'2, …, C' t ,…,C' T ] of the action video V of the person is input into the frame-level dynamic information capturing module, and is sequentially processed by the down-sampling layer, the hourglass convolution layer, the spatial convolution layer, and the up-sampling layer, and the output feature F fm is obtained; the main calculation of the frame-level dynamic information capturing module is concentrated in the hourglass convolution layer, and the present application first reduces the resolution of the input video data by using the down-sampling layer before using the hourglass convolution layer for calculation, thereby reducing the calculation consumption, and restores the resolution of the input video data by using the up-sampling layer after using the hourglass convolution layer for calculation, thereby not affecting the subsequent calculation.
[0061] Then, F S is added to F fm , and the output M fm of the frame-level dynamic information capturing network is obtained; in the traditional method, only the feature F S is obtained in this stage, and compared with the method proposed in the present application, the frame-level dynamic information is lacking, so the present application has higher recognition accuracy.
[0062] Step 2.3, the segment-level dynamic information capturing network is composed of four serially connected convolution blocks, and each convolution block is composed of serially connected repeating units, and the number of repeating units contained in each convolution block is different.
[0063] The repeating unit consists of a residual block and a fragment-level dynamic information capture module; the residual block contains two convolutional layers with 1×1 kernels and one convolutional layer with 3×3 kernels; the fragment-level dynamic information capture module contains two 1×1×1 convolutional layers, an hourglass convolution, a global average pooling layer, and a sigmoid activation function layer.
[0064] The process of obtaining fragment-level dynamic information is as follows Figure 5 As shown: M fm After the first 1×1 convolutional layer of the first repeating unit in the first convolutional block of the fragment-level dynamic information capture network is input, feature Y is obtained. Y is input into the fragment-level dynamic information capture module and passes through the first 1×1×1 convolutional layer, an hourglass convolutional layer, a global average pooling layer, a second 1×1×1 convolutional layer, and a Sigmoid activation function layer in sequence to obtain feature A. A is multiplied by Y and then input into the residual block of the first repeating unit in the first convolutional block. After passing through the 3×3 convolutional layer and the second 1×1 convolutional layer in sequence, the output Z' of the first repeating unit of the first convolutional block is obtained. In the above process, the present invention places the hourglass convolutional layer, which requires additional computation, between two 1×1×1 convolutional layers. By using the first 1×1×1 convolutional layer for channel dimensionality reduction, the present invention reduces the consumption of computational resources. Then, the second 1×1×1 convolutional layer restores the channel dimension. Traditional networks use temporal convolution to model segment-level dynamic information. This invention utilizes hourglass convolution to capture spatiotemporal information that temporal convolution cannot capture. Furthermore, by modeling dynamic information at both the frame and segment levels, this invention hierarchically models the spatiotemporal dynamic information in video data, resulting in higher recognition accuracy compared to traditional methods.
[0065] Z' is then input into the second repeated unit in the first convolutional block, and the result after the same processing is input into the next repeated unit. Thus, the result after processing by all repeated units in the first convolutional block is input into the next convolutional block for processing. Finally, the output Z of the hierarchical hourglass convolutional network is obtained from the last complex unit of the fourth convolutional block.
[0066] Step 3: The classification network consists of a global average pooling layer and a fully connected layer connected in series; Z is then input into the classification network for processing to obtain the final action category;
[0067] Step 4: Construct the cross-entropy loss function as the loss function L of the hierarchical hourglass convolutional network, and use the SGD optimizer to train the hierarchical hourglass convolutional network. At the same time, calculate the loss function L to adjust the network parameters, and finally obtain the trained hierarchical hourglass convolutional network as a video action classifier to achieve video action classification.
[0068] In this embodiment, an electronic device includes a memory and a processor, the memory is used to store a program supporting the processor to execute the video action classification method, and the processor is configured to execute the program stored in the memory.
[0069] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program performs the steps of the video action classification method when executed by a processor.
[0070] In order to illustrate the effectiveness of the present application, the following experiments are carried out for verification.
[0071] 1) The sandglass convolution is inserted into the ResNet network, named HgC-ResNet, and compared with TSN without using time convolution and R(2+1)D using ordinary time convolution on Something-SomethingV1, and the results are shown in Table 1.
[0072] Table 1 Performance comparison of sandglass convolution and R(2+1)D, TSN
[0073] Method Top-1 #P FLOPS TSN 19.7 23.9M 32.9G R(2+1)D 46.0 23.9M 32.9G HgC-ResNet 47.0 23.9M 33.1G
[0074] As observed in Table 1, both time convolution (R(2+1)D) and sandglass convolution (HgC-ResNet) can significantly improve the performance of two-dimensional convolutional neural network backbone network (TSN). At the same time, HgC-ResNet exceeds R(2+1)D by a significant margin (1%), but the computational cost is almost the same, and this comparison mainly shows the good ability of sandglass convolution in video motion modeling.
[0075] 2) On Something-SomethingV1&V2, compare the video action classification method (H 2 CN) based on the hierarchical dynamic modeling of sandglass convolution proposed in the present application with other most advanced action recognition models, and the results are shown in Table 2.
[0076] Table 2 Performance comparison of H 2 CN and other models on SomethingV1&V2
[0077] Method BackBone #Pretrain Something V1 Something V2 GST ResNet-50 ImageNet 47.0 61.6 TSM+TPN ResNet-50 ImageNet 49.0 62.0 TEINeT ResNet-50 ImageNet 47.4 61.3 TAM ResNet-50 ImageNet 46.5 60.5 STM ResNet-50 ImageNet 49.2 62.3 TDN ResNet-50 ImageNet 52.3 64.0 SELFYNeT ResNet-50 ImageNet 52.5 64.5 SmallBig ResNet-50 ImageNet 48.3 61.6 TimeSformer-HR Transformer Kinetics -- 62.5 ECO ResNet-18 Kinetics 39.6 -- I3D 3DResNet-50 ImageNet 41.6 -- H 2 CN ResNet-50 ImageNet 53.6 65.2
[0078] As shown in Table 2, compare H 2 CN with convolutional neural network-based architectures, including classic methods such as I3D, GST, TSM, and the latest methods such as TDN and SELFYNet. H 2CN achieved Top-1 accuracy of 53.6% and 65.2% on Something V1 & V2, respectively. Compared with other convolutional neural network-based methods, H2CN outperforms them with a clear advantage. These results demonstrate the H 2 CN's ability to capture various dynamic information. Compared with more complex Transformer-based methods like TimeSformer-HR], H 2 CN's performance is still competitive.
[0079] 3) On Diving48, compare the action recognition accuracy of the invention with other state-of-the-art action recognition models, and the results are shown in Table 3.
[0080] Table 3 H 2 CN's performance on Diving48 compared with other state-of-the-art models
[0081]
[0082]
[0083] From Table 3, it can be seen that compared with the convolutional neural network baseline, H 2 CN achieved the best performance of 87.0%. More importantly, H 2 CN's performance is better than the best Transformer-based method VIMPAC (85.5%).
Claims
1. A video action classification method based on hourglass convolution hierarchical dynamic modeling, characterized in that The steps are as follows: Step 1, video data extraction and preprocessing: T keyframe images are uniformly sampled from the video of a person's movement V at a fixed frame rate, denoted as F = [F1, F2, ..., F]. t ,…,F T ], F t This represents the t-th keyframe, where T represents the keyframe number. sampling the t-th key frame F t two consecutive frames before and after the person action video V, and F t and its two consecutive frames before and after are expressed as the t-th segment the first frame before F t , the first frame before F t , the first frame after F t , the second frame after F t ; The resolution of each frame in the t-th segment C t is scaled, and then a normalized preprocessing is performed on an image block with a resolution of HxW extracted from each frame, to obtain the t-th input video data tensor Thus, the input video data tensor C' = [C'1, C'2, …, C' t , …, C' T ] of the character action video V is obtained, wherein H and W represent the height and width of C' t , respectively, and D represents the channel number of C' t . Step 2, constructing a hierarchical hourglass convolution network, including: a frame-level dynamic information capture network, a segment-level dynamic information capture network, and a classification network; Step 2.1, constructing an hourglass convolution: The hourglass convolution is composed of a group of spatial convolutions with kernel size (p·|i|+1, p·|i|+1) and a temporal convolution with kernel size K, where p is a parameter and i is a time offset; The hourglass convolution processes any tensor X with dimensions to obtain an output feature HgC(X), where T' represents a time dimension size, H' represents a height, W' represents a width, and D' represents a number of channels, and the t-th feature HgC(X) t of the output feature HgC(X) is obtained using formula (1): In formula (1), X t+i is the t+i-th input feature of tensor X in the T' time dimension, a i is the i-th parameter of the time convolution layer, f is a spatial convolution function, W p·|i|+1,p·|i|+1 is a parameter of the spatial convolution layer; t ∈ [0, T'-1] Step 2.2, the frame-level dynamic information capture network is composed of the first convolution block of the ResNet50 network and the frame-level dynamic information capture module: The first convolution block of the ResNet50 network is a spatial convolution with a convolution kernel of a×a; The frame-level dynamic information capture module is composed of a down-sampling layer, an hourglass convolution layer, a spatial convolution layer, and an up-sampling layer: The down-sampling layer is a spatial average pooling layer with kernel size b×b; the hourglass convolution layer is composed of two serially connected hourglass convolutions; the spatial convolution layer is a spatial convolution with a convolution kernel of a×a; the up-sampling layer is an up-sampling operation that copies one pixel into four adjacent pixels; The key frame image F = [F1, F2, …, F t ,…,F T ] of the character action video V is input into the first convolutional block of the ResNet50 network for processing, and the output feature F S is obtained. The input video data tensor C' = [C'1, C'2, ..., C''] of the character's action video V is... t ,…,C' T The input is fed into the frame-level dynamic information capture module, and after passing through a downsampling layer, an hourglass convolutional layer, a spatial convolutional layer, and an upsampling layer in sequence, the output feature F is obtained. fm ; F S is added to F fm to obtain the output M of the frame-level dynamic information capture network fm ; Step 2.3, the segment-level dynamic information capture network is composed of four convolution blocks connected in series, and each convolution block contains a different number of repeating units; The repeating unit is composed of a residual block and a segment-level dynamic information capture module; the residual block contains two convolution layers with a convolution kernel of 1×1 and a convolution layer with a convolution kernel of 3×3; the segment-level dynamic information capture module contains two 1×1×1 convolution layers, an hourglass convolution, a global average pooling layer, and a Sigmoid activation function layer; M fm After the first 1x1 convolutional layer in the first repeating unit of the first convolutional block of the segment-level dynamic information capture network is input, a feature Y is obtained. Y is input into the segment-level dynamic information capture module and sequentially processed by the first 1x1x1 convolutional layer, an hourglass convolutional layer, a global average pooling layer, the second 1x1x1 convolutional layer, and the Sigmoid activation function layer to obtain a feature A. A is multiplied by Y, and the result is input into the residual block of the first repeating unit of the first convolutional block and sequentially processed by the 3x3 convolutional layer and the second 1x1 convolutional layer to obtain an output Z' of the first repeating unit of the first convolutional block. Z' is input into the second repeating unit in the first convolution block and the result after the same processing is input into the next repeating unit, so that the result after processing all repeating units in the first convolution block is input into the next convolution block for processing, and finally the output Z of the hierarchical hourglass convolution network is obtained from the last repeating unit of the fourth convolution block; Step 3, the classification network is composed of a global average pooling layer and a fully connected layer connected in series; and after Z is input into the classification network for processing, the final action category is obtained; Step 4, constructing a cross-entropy loss function as the loss function L of the hierarchical hourglass convolution network, and using the SGD optimizer to train the hierarchical hourglass convolution network, while calculating the loss function L to adjust the network parameters, finally obtaining the trained hierarchical hourglass convolution network as a video action classifier for realizing video action classification.
2. An electronic device comprising a memory and a processor, characterized in that The memory is used to store programs supporting the processor to execute the video action classification method of claim 1, and the processor is configured to execute the programs stored in the memory.
3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the video action classification method of claim 1.