A video action recognition method and device based on spatiotemporal channel excitation

Abstract

The present invention provides a method and device for video action recognition based on spatiotemporal channel excitation. The method comprises: constructing a video action dataset, the dataset including video clips and video action categories; constructing a video action recognition model, the video action recognition model including a spatiotemporal channel excitation module for extracting features from the video to be recognized using a channel excitation method in the spatial and temporal dimensions to obtain a feature map containing spatial and temporal features; iteratively training the video action recognition model based on the video action dataset using a loss function; and inputting the video to be recognized into the trained video action recognition model to obtain an action recognition result. The present invention solves the problem that existing video action recognition methods ignore the interaction between the spatial and temporal dimensions, resulting in poor recognition effect and low efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a video action recognition method and device based on spatiotemporal channel excitation. Background Art

[0002] With the rapid development of safe cities and artificial intelligence, video recording equipment is becoming increasingly prevalent, and video software is also proliferating. Surveillance cameras in hospitals, shopping malls, and public transportation record massive amounts of content daily to ensure stable public order and travel safety. All of these scenarios require efficient and accurate video understanding.

[0003] Temporal modeling is crucial for video understanding. For example, models without temporal modeling capabilities can lead to recognition confusion in parking and driving scenarios. Since the introduction of two-stream networks, deep learning methods have gradually become the standard for video understanding. TSM, one of the most effective models, breaks away from traditional two-dimensional convolutional methods by independently processing the temporal dimension by shifting feature channels forward and backward along the temporal dimension. This allows for modeling temporal and spatial information with fewer network parameters, enabling both offline and online video understanding. However, while the TMS model simulates temporal information through partial channel shifting, it ignores the interaction between spatial and temporal dimensions, and its feature extraction capabilities still need to be improved. Summary of the Invention

[0004] In view of the above analysis, the present invention aims to provide a video action recognition method based on spatiotemporal channel excitation, which is used to solve the problem that the feature extraction method for video in the prior art ignores the interaction between the spatial and temporal dimensions, and the extracted features ignore the time dimension information, resulting in poor recognition effect and low efficiency using the feature.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] In one aspect, the present invention provides a method for video action recognition based on spatiotemporal channel excitation, the method comprising:

[0007] Constructing a video action dataset, wherein the dataset includes video clips and video action categories;

[0008] Constructing a video action recognition model, the video action recognition model includes a spatiotemporal channel excitation module for extracting features of the video to be recognized using a channel excitation method in the spatial dimension and the temporal dimension to obtain a feature map that integrates the spatial features and the temporal features;

[0009] Based on the video action dataset, iteratively training the video action recognition model through a loss function to obtain a trained video action recognition model;

[0010] The video to be recognized is input into the trained video action recognition model to obtain an action category recognition result.

[0011] Furthermore, the video action recognition model is constructed based on the Resnet50 model; at least one spatiotemporal channel excitation module is set after each Resblock module from the 2nd stage to the 5th stage of the Resnet50 model, and spatial and temporal features are extracted and fused on the output of each Resblock module.

[0012] Furthermore, the spatiotemporal channel excitation module includes an input module, an SGAP module, a TGAP module and a feature fusion module;

[0013] The input module is used to divide the feature map output by the corresponding Resblock module into two groups of feature maps X according to the channel dimension s and X te , the feature map output by the Resblock module is a feature map composed of multiple consecutive frames of images;

[0014] The SGAP module and the TGAP module are used to respectively s and X te The channel excitation method is used to extract features in the spatial dimension and the temporal dimension respectively to obtain the spatial feature map and the temporal feature map;

[0015] The feature fusion module is used to perform feature fusion on the spatial feature map and the temporal feature map, and extract a feature map of the fused spatial features and temporal features of the video.

[0016] Furthermore, the SGAP module uses a channel excitation method to extract features in the spatial dimension to obtain a spatial feature map, including: using the SGAP module to extract a set of feature maps X s Perform average pooling in the spatial dimension, while keeping the temporal dimension unchanged, to obtain a primary spatial feature map; use one-dimensional convolution to model the spatial features of the primary spatial feature map to obtain a spatial feature map corresponding to the video;

[0017] The TGAP module uses a channel excitation method to extract features in the time series dimension to obtain a time series feature map, including: using the TGAP module to extract features of another set of feature maps X te Average pooling is performed on the temporal dimension, keeping the spatial dimension unchanged, to obtain a primary temporal feature map; two-dimensional convolution is used to model the temporal features of the primary temporal feature map to obtain a temporal feature map corresponding to the video.

[0018] Furthermore, the adopting one-dimensional convolution to model the spatial features of the primary spatial feature map includes:

[0019] Perform 1D convolution on the primary spatial feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer and G is the grouping hyperparameter;

[0020] Perform 1D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G;

[0021] The feature map after channel recovery is batch normalized and activated by the activation function;

[0022] The feature weight obtained after activation is superimposed on the primary spatial feature map to obtain the spatial feature map corresponding to the video.

[0023] Furthermore, a set of feature maps X is given by the following formula s Perform average pooling to obtain the primary spatial feature map:

[0024]

[0025] in, is the primary spatial feature map obtained after global average pooling, F SGAP (X s ) represents a set of feature maps X s Perform spatial global average pooling, H×W is the height and width of the input feature map, x s (:,:,h,w) means that the temporal dimension remains unchanged and a set of feature maps Xs are average pooled in the spatial dimension.

[0026] Furthermore, the primary spatial feature map is modeled by the following formula:

[0027]

[0028] in, Indicates modeling of the primary spatial feature map, Conv1D C / 2G→C / 2Gr Indicates dimensionality reduction of the primary spatial feature map from C / 2G→C / 2Gr, Conv1D C / 2Gr→C / 2G Indicates the dimensionality recovery of the feature map from C / 2Gr→C / 2G, where r is the dimensionality reduction ratio of the convolutional layer; Indicates an overlay operation.

[0029] Furthermore, another set of feature maps X is given by the following formula te Perform average pooling to obtain the primary temporal feature map:

[0030]

[0031] Among them, is the primary temporal feature map obtained after global temporal average pooling, F TGAP (X te ) represents the temporal global average pooling of the primary temporal feature map, T is the number of frames of the input feature map, x te (:,t,:,:) indicates average pooling of the time series dimension, while other dimensions remain unchanged.

[0032] Furthermore, the feature fusion module is used to perform feature fusion on the spatial feature map and the temporal feature map, including:

[0033] The feature fusion module receives the temporal feature map and the spatial feature map output by the SGAP module and the TGAP module, merges the temporal feature map and the spatial feature map by staggering the channels, and shuffles the merged feature map;

[0034] The feature map after channel shuffling is extracted through the average pooling operation, and then a residual connection is performed with the merged feature map, and after activation through the activation function, a feature map corresponding to the video containing spatial features and temporal features is obtained.

[0035] On the other hand, a computer device is also disclosed, which includes at least one processor and at least one memory communicatively connected to the processor; the memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the aforementioned video action recognition method based on spatiotemporal channel excitation.

[0036] Beneficial effects of this technical solution:

[0037] The present invention is based on the Resnet50 model and combines the spatiotemporal channel excitation module. Through the fusion method of separable temporal attention features and spatial attention features, the temporal features and spatial features are extracted respectively through SGAP and TGAP branches, and the association of spatiotemporal features is ensured through channel excitation and hybrid recombination operations. The method improves the performance of temporal feature extraction and spatial feature extraction while maintaining high efficiency. A large number of experiments conducted on two benchmark datasets show that the video action recognition method of the present invention outperforms existing action recognition networks in both accuracy and visual effects.

[0038] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description or be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0040] Figure 1 is a flow chart of a method for video action recognition based on spatiotemporal channel excitation according to an embodiment of the present invention;

[0041] Figure 2 is a flow chart of temporal feature extraction and spatial feature extraction according to an embodiment of the present invention;

[0042] Figure 3 This is a flow chart of the feature fusion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the implementation cases of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0044] An embodiment of the present invention provides a video action recognition method based on spatiotemporal channel excitation, such as Figure 1 As shown, the method includes:

[0045] Step S1: constructing a video action dataset, wherein the dataset includes video clips and video action categories;

[0046] Step S2: constructing a video action recognition model, wherein the video action recognition model includes a spatiotemporal channel excitation module, which is used to extract features of the video to be recognized using a channel excitation method in the spatial dimension and the temporal dimension to obtain a feature map that integrates spatial features and temporal features;

[0047] Step S3: Based on the video action dataset, iteratively training the video action recognition model using a loss function to obtain a trained video action recognition model;

[0048] Step S4: inputting the video to be recognized into the trained video action recognition model to obtain an action category recognition result.

[0049] Specifically, in step S1, this embodiment selects UCF101 and HMDB51 as training datasets. UCF101 contains 13,320 video clips and 101 action categories, while HMDB51 contains 7,000 video clips and 51 action categories. Based on these two datasets, the video clips are further augmented with random horizontal flips and 90°, 180°, and 270° rotations to expand the training data. To use image data types when training the model, RGB video frames are extracted from the datasets and saved as images. These files are stored in folders and named according to their action categories. This embodiment divides all video frame data into training and test sets in an 8:2 ratio.

[0050] In step S2, the video action recognition model of this embodiment is constructed based on the Resnet50 model; and at least one spatiotemporal channel excitation module is set after each Resblock module from the second stage to the fifth stage of the Resnet50 model to extract and fuse spatial and temporal features of the output of each Resblock module;

[0051] Among them, each spatiotemporal channel excitation module includes an input module, an SGAP module, a TGAP module and a feature fusion module;

[0052] The input module is used to divide the feature map output by the corresponding Resblock module into two groups of feature maps X according to the channel dimension s and X te ,The feature map output by the Resblock module is a feature map composed of multiple consecutive frames of images;

[0053] The SGAP module and the TGAP module are used to analyze the two sets of feature maps X s and X te The channel excitation method is used to extract features in the spatial dimension and the temporal dimension respectively to obtain the spatial feature map and the temporal feature map;

[0054] The feature fusion module is used to fuse the spatial feature map and the temporal feature map to extract the feature map of the fused spatial features and temporal features of the video.

[0055] Specifically, such as Figure 2 As shown, the proposed spatiotemporal channel excitation module can be effectively used in videos to extract separable temporal and spatial attention features. Based on the TSM model, this method first groups feature maps by channel dimension, then calculates the dependencies between the two feature groups in the spatial and temporal channel dimensions. Finally, all features are cross-fused using a channel shuffle operation. This approach improves the feature representation after temporal and spatial channel transfer with less computation and enhances the performance of the TSM model.

[0056] As a specific embodiment, a video composed of multiple frames of images is given as the input of the video action recognition model. After feature extraction by the Resblock module, a feature map composed of multiple frames of images is obtained. The obtained feature map is input to the input module of the spatiotemporal channel excitation module. Each group of feature maps is divided into two groups X by the input module. s and X te , which are used for spatial feature extraction and temporal feature extraction respectively, and are expressed as:

[0057]

[0058]

[0059] Where T is the number of image frames, which is 8 in this embodiment, and C is the number of image channels x s n and x te n Represents the feature map X s and X te The nth frame in .

[0060] Furthermore, the two sets of feature maps are respectively extracted using the channel excitation method in the spatial dimension and the temporal dimension, that is, the SGAP module and the TGAP module are respectively used for spatial feature extraction and temporal feature extraction, wherein the SGAP module contains a time compression submodule and a pooling submodule for time compression and pooling operations, and the TGAP module contains a time compression submodule and a pooling submodule for spatial compression and pooling operations;

[0061] More specifically, the input feature map is split into two groups X s and X te , each group has c channels, satisfying 2cG=CT, G is a grouping hyperparameter, preferably, G can be 8, 10 or 12 groups, etc., according to actual needs. In this embodiment, G is 8; the average pooling method is used by the SGAP module to perform the first group of feature maps X s Feature extraction is performed on the spatial dimension to obtain a spatial feature map; the second set of feature maps X is pooled using the average pooling method through the TGAP module te Feature extraction is performed on the time dimension to obtain a time series feature graph.

[0062] Among them, the channel excitation method is used to extract features in the spatial dimension to obtain a spatial feature map, including: a set of feature maps X s Average pooling is performed in the spatial dimension, while the temporal dimension remains unchanged, to obtain a primary spatial feature map; one-dimensional convolution is used to model the spatial features of the primary spatial feature map to obtain the spatial feature map corresponding to the video.

[0063] One-dimensional convolution is used to model the spatial features of the primary spatial feature map, including: performing 1D convolution on the primary spatial feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer; performing 1D convolution on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G; batch normalizing the feature map after channel restoration and activating it through an activation function; superimposing the feature weights obtained after activation on the primary spatial feature map to obtain the spatial feature map corresponding to the video.

[0064] Preferably, a set of feature maps X is given by the following formula s Perform average pooling to obtain the primary spatial feature map:

[0065]

[0066] in, is the primary spatial feature map obtained after global average pooling, F SGAP (X s ) represents a set of feature maps X s Perform spatial global average pooling, H×W is the height and width of the input feature map, x s (:,:,h,w) means that the temporal dimension remains unchanged and a set of feature maps Xs are average pooled in the spatial dimension.

[0067] The primary spatial feature map is modeled by the following formula:

[0068]

[0069] in, Indicates modeling of the primary spatial feature map, Conv1D C / 2G→C / 2Gr Indicates that the primary spatial feature map is from C / 2G→C / 2G r Perform dimensionality reduction, Conv1D C / 2Gr→C / 2G Indicates the dimensionality recovery of the feature map from C / 2Gr→C / 2G, where r is the dimensionality reduction ratio of the convolutional layer; Indicates an overlay operation.

[0070] Furthermore, the channel excitation method is used to extract features in the time series dimension to obtain a time series feature map, including: te Perform average pooling in the temporal dimension, keeping the spatial dimension unchanged, to obtain a primary temporal feature map; use two-dimensional convolution to model the temporal features of the primary temporal feature map to obtain a temporal feature map corresponding to the video;

[0071] Among them, two-dimensional convolution is used to model the spatial features of the primary temporal feature map, including: performing 2D convolution on the primary temporal feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer; performing a 2D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G; the feature map after channel restoration is batch normalized and activated by an activation function; the feature weights obtained after activation are superimposed on the primary temporal feature map to obtain the temporal feature map corresponding to the video.

[0072] Preferably, another set of feature maps X is calculated by the following formula: te Perform average pooling to obtain the primary temporal feature map:

[0073]

[0074] in, is the primary temporal feature map obtained after global temporal average pooling, F TGAP (X te ) represents the temporal global average pooling of the primary temporal feature map, T is the number of frames of the input feature map, x te (:,t,:,:) indicates average pooling of the time series dimension, while other dimensions remain unchanged.

[0075] The primary time series characteristic graph is modeled by the following formula:

[0076]

[0077] in, Indicates modeling of primary temporal feature maps, Conv2D C / 2G→C / 2Gr Indicates that the feature is reduced from C / 2G→C / 2Gr by 2D convolution, Conv2D C / 2Gr→C / 2G Indicates that the feature is restored from C / 2Gr→C / 2G through 2D convolution. Indicates superimposing the extracted feature weights with the primary temporal feature map.

[0078] In this embodiment, spatial features and temporal features are modeled respectively through 1D convolution and 2D convolution, and superimposed with the initial feature vector, so as to realize linear combination (information integration) of each pixel on different channels, and retain the original planar structure of the original image. Through channel adjustment, the functions of dimensionality increase and dimensionality reduction are completed, the parameters of model training can be controlled, and the effective features of the video can be obtained more specifically, and useless features can be filtered out. This incentive mechanism focuses on improving the ability to integrate the spatial information and temporal information of the model, and adopts channel grouping to learn spatial and temporal information in parallel, and filters out effective spatiotemporal semantics through an adaptive fusion mechanism. After passing through the convolution layer and being activated, the expression ability of the model is effectively improved, and the effectiveness of feature extraction is improved.

[0079] Furthermore, the spatial feature map and the temporal feature map are subjected to feature fusion through a feature fusion module to extract a feature map of the fused spatial features and temporal features of the video.

[0080] Specifically, this embodiment uses a feature fusion module to merge the temporal feature map and the spatial feature map by staggering the channels; Figure 3 As shown, the merged feature map is shuffled by channels; the feature map after channel shuffling is extracted by average pooling operation and then residually connected with the merged feature map, and activated by an activation function to obtain the feature map corresponding to the video containing spatial features and temporal features.

[0081] More specifically, feature fusion is performed using the following formula:

[0082] I out =ReLU(X ST +Pool(Shuffle(X ST )));

[0083] Among them, I out Represents the feature map output after feature fusion, X ST represents the sampled feature map after the attention excitation and merging of time-space separation, Shuffle(*) represents the ST Perform channel shuffling, Pool(*) means to perform average pooling and resampling on the feature map after channel shuffling, and compare it with X in a residual way. ST After connection, Relu activation function is used.

[0084] In step S3, the video action recognition method was implemented on the PyTorch platform. Experiments were conducted using Nvidia Tesla V100s graphics cards. The model was trained using a mini-batch stochastic gradient descent algorithm and the Adam optimization algorithm. The batch size was set to 12, the initial learning rate was 0.0001, the learning rate decay period was 10, the decay factor was 0.1, the dropout was set to 0.8, and the number of model iterations was 25. Research has shown that using pre-training is an effective method for initializing convolutional neural networks when the target dataset lacks sufficient training samples. To avoid overfitting, a pre-trained model from the Kinetics dataset was used to initialize the network weights.

[0085] It should be noted that, in this embodiment, the spatiotemporal channel excitation module is added to the four backbones of the Resnet50 model to improve the network's action recognition ability; when evaluating the performance of the action recognition network, the evaluation indicator used is precision; precision refers to the proportion of events correctly predicted by the network in all prediction samples, reflecting the accuracy of the action recognition network, TP represents the number of correct predictions, FP represents the number of incorrect predictions, TP+FP represents the total number of predictions, and precision (accuracy) represents the proportion of the number of correct predictions of the model to the total number of predictions.

[0086] This implementation also verifies the effectiveness of the video action recognition method based on spatiotemporal channel excitation proposed in this invention through ablation experiments;

[0087] First, we analyzed the impact of the location of the spatiotemporal channel excitation module in the backbone network on network performance. Furthermore, to verify the module's effectiveness, we compared the impact of varying the number of spatiotemporal channel excitation modules on network performance in the stage 3 branch.

[0088] Table 1 shows the results of the first ablation experiment, in which three spatiotemporal channel excitation (SSA) modules were added to stage 1, stage 2, stage 3, and stage 4, respectively. The experimental results show that the effect of the spatiotemporal channel excitation module improves significantly as the network deepens. While maintaining the same basic settings, adding the spatiotemporal channel excitation module to stage 4 can effectively improve the network's temporal extraction capabilities. As shown in Table 1, using the spatiotemporal channel excitation module improves network performance by 0.5% on the UCF101 dataset and by 1.4% on the HMDB51 dataset. This demonstrates that the module helps fuse spatial and temporal features, demonstrating its effectiveness.

[0089] Table 1. The impact of spatiotemporal attention block position on network performance

[0090] Structure SSA Pre-train UCF101 HMDB51 Baseline ×0 Kinetics 94.6％ 73.5％ Stage 1 ×3 Kinetics 94.6％ 73.4％ Stage 2 ×3 Kinetics 94.7％ 73.6％ Stage 3 ×3 Kinetics 94.8％ 74.1％ Stage 4 ×3 Kinetics 95.1％ 74.9％

[0091] Table 2 shows the results of the second ablation experiment, in which the number of spatiotemporal channel excitation modules was set to 0, 1, 3, and 6, respectively. As the number of modules increased, performance on the UCF101 and HMDB51 datasets gradually improved, but the number of parameters and computational complexity also increased. This shows that the spatiotemporal channel excitation module proposed in this example can effectively improve network performance while adding only a small number of parameters and computational complexity.

[0092] Table 2 Effect of the number of spatiotemporal attention excitation blocks on network performance

[0093] Structure SSA UCF101 HMDB51 #Parameter FLOPs Baseline ×0 94.6％ 73.5％ 24.33M 58.42G Stage 3 ×1 94.7％ 73.8％ 24.33M 58.42G Stage 3 ×3 95.0％ 74.3％ 24.53M 58.42G Stage 3 ×6 95.2％ 74.9％ 24.83M 58.43G

[0094] The network with the spatiotemporal channel excitation module is compared with seven state-of-the-art lightweight SISR methods at *2, *3, and *4 scales, including IDT, DRTAP, TWO-STREAM, C3D, P3D, R(2+1)D, TSN, and TSM. As shown in Table 3, SSA outperforms other state-of-the-art lightweight methods on all benchmarks in terms of both 2DCNN and 3DCNN metrics.

[0095] Table 3 Comparison of the effects of various methods on two datasets

[0096]

[0097] Another embodiment of the present invention also discloses a computer device, including at least one processor and at least one memory communicatively connected to the processor; the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the aforementioned video action recognition method based on spatiotemporal channel excitation.

[0098] In summary, the video action recognition method based on spatiotemporal channel excitation provided by the embodiments of the present invention is based on the Resnet model and adopts the spatiotemporal channel excitation module to realize the separable spatiotemporal attention feature fusion method, which improves the temporal feature extraction performance while maintaining high efficiency. The SGAP and TGAP branches of the present invention can extract temporal and spatial features respectively, and ensure the correlation of spatiotemporal features through channel excitation and hybrid recombination operations; the method focuses on improving the fusion ability of spatial information and temporal information of the model, adopts channel grouping to learn spatial and temporal information in parallel, and filters out effective spatiotemporal semantics through an adaptive fusion mechanism; a large number of experiments on two benchmark data sets show that the present invention is superior to other advanced action recognition networks in terms of accuracy and visual effects.

[0099] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0100] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A video action recognition method based on spatiotemporal channel excitation, characterized in that: include: Constructing a video action dataset, wherein the dataset includes video clips and video action categories; Constructing a video action recognition model, the video action recognition model includes a spatiotemporal channel excitation module, which is used to extract features of the video to be recognized using a channel excitation method in the spatial dimension and the temporal dimension to obtain a feature map that integrates spatial features and temporal features; the spatiotemporal channel excitation module includes an input module, a SGAP module, a TGAP module and a feature fusion module; The input module is used to divide the feature map output by the corresponding Resblock module into two groups of feature maps X according to the channel dimension s and X te , the feature map output by the Resblock module is a feature map composed of multiple consecutive frames of images; The SGAP module and the TGAP module are used to respectively s and X te The channel excitation method is used to extract features in the spatial dimension and the temporal dimension respectively to obtain spatial feature maps and temporal feature maps, including: using the SGAP module to extract a set of feature maps X s Perform average pooling in the spatial dimension, and keep the temporal dimension unchanged to obtain a primary spatial feature map; use one-dimensional convolution to model the spatial features of the primary spatial feature map to obtain the spatial feature map corresponding to the video; use the TGAP module to model another set of feature maps X te Perform average pooling in the temporal dimension, keeping the spatial dimension unchanged, to obtain a primary temporal feature map; use two-dimensional convolution to model the temporal features of the primary temporal feature map to obtain a temporal feature map corresponding to the video; Among them, a set of feature maps X is calculated by the following formula s Perform average pooling to obtain the primary spatial feature map: in, is the primary spatial feature map obtained after global average pooling, F SGAP (X s ) represents a set of feature maps X s Perform spatial global average pooling, H×W is the height and width of the input feature map, x s (:,:,h,w) means that the temporal dimension remains unchanged, and a set of feature maps X is s Perform average pooling; The primary spatial feature map is modeled by the following formula: in, Indicates modeling of the primary spatial feature map, Conv1D C / 2G→C / 2Gr Indicates dimensionality reduction of the primary spatial feature map from C / 2G→C / 2Gr, Conv1D C / 2Gr→C / 2G Indicates the dimensionality recovery of the feature map from C / 2Gr→C / 2G, where C is the number of image channels, G is the grouping hyperparameter, and r is the dimensionality reduction ratio of the convolutional layer; Indicates superposition operation; Another set of feature maps X is calculated by the following formula te Perform average pooling to obtain the primary temporal feature map: Among them, is the primary temporal feature map obtained after global temporal average pooling, F TGAP (X te ) represents the temporal global average pooling of the primary temporal feature map, T is the number of frames of the input feature map, x te (:,t,:,:) means average pooling is performed on the time series dimension, while other dimensions remain unchanged; The primary time series characteristic graph is modeled by the following formula: in, Indicates modeling of primary temporal feature maps, Conv2D C / 2G→C / 2Gr Indicates that the feature is reduced from C / 2G→C / 2Gr by 2D convolution, Conv2D C / 2Gr→C / 2G Indicates that the feature is restored from C / 2Gr to C / 2G through 2D convolution, where C is the number of image channels, G is the grouping hyperparameter, and r is the dimensionality reduction ratio of the convolution layer. Indicates superimposing the extracted feature weights with the primary temporal feature map; The feature fusion module is used to perform feature fusion on the spatial feature map and the temporal feature map to extract a feature map of the fused spatial features and temporal features of the video, including: the feature fusion module receives the temporal feature map and the spatial feature map output by the SGAP module and the TGAP module, merges the temporal feature map and the spatial feature map by staggering the channels, and performs channel shuffling on the merged feature map; extracts features from the channel shuffled feature map through an average pooling operation, performs a residual connection with the merged feature map, and activates the feature map corresponding to the video by an activation function to obtain a feature map containing spatial features and temporal features; The video action recognition model is constructed based on the Resnet50 model; the spatiotemporal channel excitation module is set in stage 4 of the Resnet50 backbone network to improve the network's time series extraction capability; Based on the video action dataset, iteratively training the video action recognition model through a loss function to obtain a trained video action recognition model; The video to be recognized is input into the trained video action recognition model to obtain an action category recognition result.

2. The video action recognition method based on spatiotemporal channel excitation according to claim 1 is characterized in that: The adopting one-dimensional convolution to model the spatial features of the primary spatial feature map includes: Perform 1D convolution on the primary spatial feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer and G is the grouping hyperparameter; Perform 1D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G; The feature map after channel recovery is batch normalized and activated by the activation function; The feature weight obtained after activation is superimposed on the primary spatial feature map to obtain the spatial feature map corresponding to the video.

3. A computer device, characterized in that: It includes at least one processor and at least one memory communicatively connected to the processor; the memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the video action recognition method based on spatiotemporal channel excitation according to any one of claims 1-2.

Images